Method and system for assisting traffic incident deployment and control detection based on visual lane line extraction

Abstract

The invention belongs to the technical field of traffic event detection, and discloses a method for assisting traffic event deployment and control detection based on visual lane line extraction, and the method comprises the following steps: S1, training a lane line detection model which supports the real-time online reasoning of a lane line; s2, scene change detection is judged based on the video monitoring image; s3, performing lane line extraction on the image with scene change in the step S2 by adopting a lane line detection model; s4, fitting a traffic incident detection rule; s5, a rule based on a preset value; and S6, traffic incident detection and alarm. According to the invention, scene structure information such as lane lines and the like can be comprehensively utilized, vehicle and road conditions are analyzed, whether potential accidents exist or not is judged through parking, and the accuracy and robustness of traffic incident detection can be improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of traffic incident detection, and in particular relates to a method and system for assisting traffic incident control detection based on visual lane line extraction. Background Art

[0002] my country's highway network is developing rapidly, the density of roadside surveillance is increasing, and visual-based traffic incident detection products have been gradually deployed. However, currently, roadside video surveillance is mainly based on pan-tilt cameras and zoom cameras. Due to manual operation or vibration, there are changes in monitoring scenes, which leads to abnormal operation of traffic incident detection algorithms.

[0003] The existing vision-based traffic incident detection methods are not robust enough in traffic monitoring environments. Specifically, they have the following defects or deficiencies:

[0004] 1) Unstable manual control: PTZ cameras require manual or automatic adjustment of the camera angle to cover different road areas. However, frequent manual operation, angle adjustment, repositioning, etc. may cause the camera's shooting direction to shift or even produce perspective errors. Such misoperation may cause the monitoring area to deviate from the target area, affect the accuracy of traffic event detection, and even invalidate the control rules.

[0005] 2) Impact of vibration: Under the influence of external factors such as wind, traffic vibration, and construction, the roadside surveillance cameras will produce certain vibrations, which will cause the camera image to be blurred and the lane line extraction to be incorrect, thus affecting the accuracy and reliability of the event detection results.

[0006] 3) Zoom camera focal length adjustment: During the zoom camera focal length adjustment process, the image quality may be degraded due to incorrect focus adjustment, lens shaking, etc., which affects the accuracy of lane line extraction and the performance of traffic event detection. The change of monitoring scene leads to the mismatch of traffic event detection and control rules.

[0007] Therefore, there is an urgent need for a traffic event detection method that can comprehensively utilize scene structure information such as lane lines to improve detection accuracy and robustness. Summary of the invention

[0008] In view of this, the purpose of the present invention is to provide a method and system for assisting traffic incident control detection based on visual lane line extraction. The present invention aims to solve the problems of low accuracy and poor robustness of existing traffic incident detection methods.

[0009] The present invention provides a method for assisting traffic event control detection based on visual lane line extraction, comprising the following steps:

[0010] S1. Train a lane detection model that supports real-time online lane reasoning;

[0011] S2. Scene change detection based on video surveillance images;

[0012] Based on the traffic monitoring video images collected by highway cameras, the changes between consecutive frames in the monitoring image sequence are analyzed, and the image difference between adjacent video frames is calculated to determine whether the scene change occurs;

[0013] If a scene change occurs, proceed to step S3; if no scene change occurs, continue to obtain the video stream to determine the scene change;

[0014] S3. Use the lane line detection model to extract the lane line of the image where the scene change occurs in step S2;

[0015] S4. Fitting traffic event detection rules;

[0016] Based on the lane line information extracted in step S3, dynamically fitting and adapting to new traffic event detection rules;

[0017] S5. Rules based on preset values;

[0018] The lane fitting rule length, length-width ratio, and the ratio between the vehicle bounding box and the rule area are used to determine whether the fitting is successful. If the fitting is successful, the original manual control rules are invalidated and the detection and control rules after fitting are updated. If the fitting fails, the lane extraction information is re-acquired for fitting.

[0019] S6. Traffic incident detection and alarm;

[0020] Based on the initial manual control rules or the updated traffic event detection and control rules after the scene changes, perform vehicle detection and tracking on each input image frame, and output the vehicle position and tracking ID in each frame image;

[0021] Based on the vehicle position and tracking ID in each frame image, the traffic events that occur are judged in real time, and event detection is performed based on the fitted rules. Traffic events include abnormal parking in the lane, traffic congestion, pedestrians or non-motor vehicles entering the lane;

[0022] After a traffic incident is detected, an alarm is triggered and the information of the traffic incident is recorded: incident type, time, and location.

[0023] Further, the step S1 includes the following sub-steps:

[0024] S1.1 collects lane data and labels it in an offline manner;

[0025] S1.2 Perform lane extraction training based on U-Net or LaneNet end-to-end detection model;

[0026] After the S1.3 model is trained, it is published and deployed to the inference framework TensorRT to obtain a lane line detection model that supports real-time online inference of lane lines.

[0027] Further, the step S3 includes the following sub-steps:

[0028] S3.1 obtain the monitoring panoramic group picture after the scene change;

[0029] S3.2 performs edge detection and parking area ROI extraction on the monitoring panoramic group image after the scene change;

[0030] S3.3 uses a pre-trained lane line detection model to learn lane line features from the image output in step S3.2 and perform end-to-end lane line detection.

[0031] Further, the content of step S4 is: based on the lane line information extracted in step S3, the edge points or contours of the lane line are fitted to generate the actual lane line:

[0032] There are several ways to fit the edge points or contours of lane lines:

[0033] Polynomial fitting: Use quadratic or cubic polynomials to fit the geometric features of lane lines, including lane curvature and bending degree;

[0034] The parameters of the lane line are fitted by the least square method to generate a smooth curve representing the lane line;

[0035] For the edge image after lane line extraction, the points on the edge are mapped to the parameter space through Hough transform, and the straight lines in the image are found and automatically fitted into the traffic event detection and control rules.

[0036] Furthermore, in step S6, vehicle detection adopts deep learning detection network yolo series or Faster-rcnn series.

[0037] Furthermore, in step S6, the vehicle tracking adopts a sort / deep sort multi-target tracking method.

[0038] The present invention also discloses a system for assisting traffic event control detection based on visual lane line extraction, including a video stream acquisition module, a scene change detection module, a lane line extraction module, a traffic event detection rule fitting module, and a traffic event detection and alarm module;

[0039] The video stream acquisition module is used to collect and store traffic monitoring video images and cache video frames;

[0040] The scene change detection module analyzes the changes between consecutive frames in the monitoring image sequence based on the traffic monitoring video images collected by the video stream acquisition module, calculates the image difference between adjacent video frames, and thus determines whether a scene change occurs;

[0041] The lane line extraction module is used to extract lane lines from images with scene changes;

[0042] The traffic incident detection rule fitting module is used to generate actual lane lines and dynamically fit new traffic incident detection rules;

[0043] The traffic event detection and alarm module is used to determine the occurrence of traffic events in real time, perform event detection based on fitting rules, and alarm when a traffic event is detected.

[0044] Beneficial effects:

[0045] 1. The present invention can comprehensively utilize scene structure information such as lane lines to analyze vehicle and road conditions, and then determine whether there is a potential accident by stopping, thereby improving the accuracy and robustness of traffic event detection.

[0046] 2. Automatically adjust the monitoring rules: When the scene changes, the system can automatically adjust the monitoring rules to ensure that traffic event detection can be carried out continuously and accurately in the new monitoring environment. By dynamically adjusting the lane line extraction and event detection rules, the system can continuously adapt to changes in the monitoring scene and avoid detection failures caused by manual operation or vibration.

[0047] 3. System feedback and operation and maintenance: The system can monitor the status of the camera in real time, and promptly provide feedback on the camera's viewing angle, vibration information, etc., prompting the operation and maintenance personnel to make necessary corrections. The system can also automatically adjust the camera's focal length or angle according to scene changes and lane line extraction accuracy to ensure the best monitoring effect.

[0048] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 The present invention is a flow chart of a method for assisting traffic event control and detection based on visual lane line extraction. DETAILED DESCRIPTION

[0050] In order to make the technical solutions, advantages and purposes of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the described embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the protection scope of this application.

[0051] like Figure 1 As shown, the present invention provides a method for assisting traffic event control detection based on visual lane line extraction, comprising the following steps:

[0052] S1. Train a lane detection model that supports real-time online lane reasoning;

[0053] S1.1 collects lane data and labels it in an offline manner;

[0054] S1.2 Perform lane extraction training based on the U-Net or LaneNet end-to-end detection model. The number of training samples n should be greater than 3000.

[0055] After the S1.3 model is trained, it is published and deployed to the inference framework TensorRT, which means that a lane detection model that supports real-time online lane inference is obtained. It can automatically learn lane features from images and perform end-to-end lane detection.

[0056] S2. Scene change detection based on video surveillance images;

[0057] Based on the traffic monitoring video images collected by highway cameras, the changes between consecutive frames in the monitoring image sequence are analyzed, and the image difference between adjacent video frames is calculated (such as based on image difference, optical flow method, feature point matching, etc.) to automatically determine whether the monitoring screen has a scene change;

[0058] If a scene change occurs (for example, the camera angle of view is adjusted or vibration causes the angle of view to shift), then proceed to step S3; if no scene change occurs, then continue to obtain the video stream to determine the scene change;

[0059] S3. Use the lane line detection model to extract the lane line of the image where the scene change occurs in step S2;

[0060] S3.1 obtain the monitoring panoramic group picture after the scene change;

[0061] S3.2 performs edge detection and parking area ROI extraction on the monitoring panoramic group image after the scene change;

[0062] S3.3 uses the pre-trained lane line detection model to learn the lane line features from the image output by step S3.2, perform end-to-end lane line detection, automatically extract lane lines from new video frames, identify lane lines in the picture, and analyze the geometric features of lanes; the results of lane line extraction will be used for subsequent event detection to ensure that lane line information remains accurate even when the viewing angle is adjusted or vibrated.

[0063] S4. Fitting traffic event detection rules;

[0064] Based on the lane line information extracted in step S3, dynamically fitting and adapting to new traffic event detection rules;

[0065] Based on the lane line information extracted in step S3, the edge points or contours of the lane lines are fitted to generate actual lane lines. There are several ways to fit the edge points or contours of the lane lines:

[0066] Polynomial fitting: Use quadratic or cubic polynomials to fit the geometric features of lane lines, including lane curvature and bending degree;

[0067] The parameters of the lane line are fitted by the least square method to generate a smooth curve representing the lane line;

[0068] For the edge image after lane line extraction, the points on the edge are mapped to the parameter space through Hough transform, and the straight lines in the image are found and automatically fitted into the traffic event detection and control rules.

[0069] For example, lane line deviation, lateral deviation of vehicles and other behaviors can trigger traffic event detection. The rule adaptively adjusts the monitoring range and control strategy to cope with problems such as camera angle adjustment, vibration or scene changes.

[0070] S5. Rules based on preset values;

[0071] The lane fitting rule length, length-width ratio, and the ratio between the vehicle bounding box and the rule area are used to determine whether the fitting is successful. If the fitting is successful, the original manual control rules are invalidated and the detection and control rules after fitting are updated. If the fitting fails, the lane extraction information is re-acquired for fitting.

[0072] S6. Traffic incident detection and alarm;

[0073] Based on the initial manual control rules or the updated traffic event detection control rules after the scene changes, perform vehicle detection (common deep learning detection network yolo series or Faster-rcnn series can be used) and vehicle tracking (common sort / deep sort multi-target tracking methods can be used) for each input frame image, and output the vehicle position and tracking ID in each frame image;

[0074] Based on the vehicle position and tracking ID in each frame image, the traffic events that occur are judged in real time, and event detection is performed based on the fitted rules. Traffic events include abnormal parking in the lane, traffic congestion, pedestrians or non-motor vehicles entering the lane;

[0075] When a traffic incident is detected, an alarm is triggered, and the traffic management department is notified of the incident information (such as incident type, time, location, etc.), or the information is fed back to the driver or other traffic participants.

[0076] The present invention also provides a system for assisting traffic event control detection based on visual lane line extraction, including a video stream acquisition module, a scene change detection module, a lane line extraction module, a traffic event detection rule fitting module, and a traffic event detection and alarm module;

[0077] The video stream acquisition module is used to collect and store traffic monitoring video images and cache video frames;

[0078] The scene change detection module is based on the traffic monitoring video images collected by the video stream acquisition module, analyzes the changes between consecutive frames in the monitoring image sequence, calculates the image difference between adjacent video frames, and thus determines whether a scene change occurs;

[0079] The lane line extraction module is used to extract lane lines from images with scene changes;

[0080] The traffic incident detection rule fitting module is used to generate actual lane lines and dynamically fit new traffic incident detection rules;

[0081] The traffic incident detection and alarm module is used to determine the occurrence of traffic incidents in real time, perform event detection based on fitting rules, and alarm when a traffic incident is detected.

[0082] The present invention relates to the field of traffic event detection, and in particular to a method and system that can support visual lane line extraction based on roadside monitoring cameras to assist in traffic event control detection. The method of the present invention solves the problem of failure of traffic event detection control rules due to manual operation or vibration of a pan-tilt camera or a zoom camera, thereby improving the robustness and accuracy of traffic event detection.

[0083] It is hereby declared that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention is described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should be included in the scope of the claims of the present invention.

Claims

1. A method for assisting traffic incident control detection based on visual lane line extraction, characterized in that: The following steps are involved: S1. Train a lane detection model that supports real-time online lane reasoning; S2. Scene change detection based on video surveillance images; Based on the traffic monitoring video images collected by highway cameras, the changes between consecutive frames in the monitoring image sequence are analyzed, and the image difference between adjacent video frames is calculated to determine whether the scene change occurs; If a scene change occurs, proceed to step S3; if no scene change occurs, continue to obtain the video stream to determine the scene change; S3. Use the lane line detection model to extract the lane line of the image where the scene change occurs in step S2; S4. Fitting traffic event detection rules; Based on the lane line information extracted in step S3, dynamically fitting and adapting to new traffic event detection rules; S5. Rules based on preset values; The lane fitting rule length, length-width ratio, and the ratio between the vehicle bounding box and the rule area are used to determine whether the fitting is successful. If the fitting is successful, the original manual control rules are invalidated and the detection and control rules after fitting are updated. If the fitting fails, the lane extraction information is re-acquired for fitting. S6. Traffic incident detection and alarm; Based on the initial manual control rules or the updated traffic event detection and control rules after the scene changes, perform vehicle detection and tracking on each input image frame, and output the vehicle position and tracking ID in each frame image; Based on the vehicle position and tracking ID in each frame image, the traffic events that occur are judged in real time, and event detection is performed based on the fitted rules. Traffic events include abnormal parking in the lane, traffic congestion, pedestrians or non-motor vehicles entering the lane; After detecting a traffic incident, an alarm will be triggered and the information of the traffic incident will be recorded: incident type, time, and location.

2. According to claim 1, a method for assisting traffic incident control and detection based on visual lane line extraction is characterized in that: The step S1 comprises the following sub-steps: S1.1 collects lane data and labels it in an offline manner; S1.2 Perform lane extraction training based on U-Net or LaneNet end-to-end detection model; After the S1.3 model is trained, it is published and deployed to the inference framework TensorRT to obtain a lane line detection model that supports real-time online inference of lane lines.

3. According to claim 2, a method for assisting traffic incident control and detection based on visual lane line extraction is characterized in that: The step S3 comprises the following sub-steps: S3.1 obtain the monitoring panoramic group picture after the scene change; S3.2 performs edge detection and parking area ROI extraction on the monitoring panoramic group image after the scene change; S3.3 uses a pre-trained lane line detection model to learn lane line features from the image output in step S3.2 and perform end-to-end lane line detection.

4. According to claim 3, a method for assisting traffic incident control and detection based on visual lane line extraction is characterized in that: The content of step S4 is: based on the lane line information extracted in step S3, the edge points or contours of the lane line are fitted to generate the actual lane line; There are several ways to fit the edge points or contours of lane lines: Polynomial fitting: Use quadratic or cubic polynomials to fit the geometric features of lane lines, including lane curvature and bending degree; The parameters of the lane line are fitted by the least square method to generate a smooth curve representing the lane line; For the edge image after lane line extraction, the points on the edge are mapped to the parameter space through Hough transform, the straight lines in the image are found, and they are automatically fitted into the traffic event detection and control rules.

5. The method for assisting traffic incident control and detection based on visual lane line extraction according to claim 4 is characterized in that: In step S6, vehicle detection uses a deep learning detection network YOLO series or Faster-RCNN series.

6. The method for assisting traffic incident control and detection based on visual lane line extraction according to claim 5 is characterized in that: In step S6, the vehicle tracking adopts the sort / deep sort multi-target tracking method.

7. A system for assisting traffic incident control and detection based on visual lane line extraction, characterized in that: It includes video stream acquisition module, scene change detection module, lane line extraction module, traffic event detection rule fitting module, and traffic event detection and alarm module; The video stream acquisition module is used to collect and store traffic monitoring video images and cache video frames; The scene change detection module analyzes the changes between consecutive frames in the monitoring image sequence based on the traffic monitoring video images collected by the video stream acquisition module, calculates the image difference between adjacent video frames, and thus determines whether a scene change occurs; The lane line extraction module is used to extract lane lines from images with scene changes; The traffic incident detection rule fitting module is used to generate actual lane lines and dynamically fit new traffic incident detection rules; The traffic event detection and alarm module is used to determine the occurrence of traffic events in real time, perform event detection based on fitting rules, and alarm when a traffic event is detected.

Images