A system and method for detecting illegal lane changing based on vehicle-road cooperation

Abstract

The application belongs to the field of intelligent vehicle-road cooperation, and particularly relates to a system and method for detecting illegal lane changing based on vehicle-road cooperation, comprising the following steps: constructing a neural network model for detecting illegal lane changing, the model input being continuous multiple frames of images, including two-stage reasoning, in the first stage, the model performs target detection on each frame of image, in the second stage, the model performs correlation and tracking of vehicle targets, and outputs tracking ID and correlated trajectory information of each vehicle target; training the neural network model used for detecting illegal lane changing; sending image data to be detected collected by road testing into the trained neural network model, obtaining detection illegal lane changing related information in continuous multiple frames of images and trajectory information of each vehicle, and judging whether the vehicle has illegal behavior according to the information, and uploading illegal vehicle data to the cloud after identification. The sensing capability and idle resources of intelligent networked vehicles are fully utilized, and the traffic safety of the road is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of intelligent vehicle-road cooperation, and particularly relates to a system and method for detecting illegal lane changing based on vehicle-road cooperation. BACKGROUND

[0002] In modern society, the increase of road traffic violations and the frequent occurrence of traffic accidents have become a serious social problem. Among them, illegal lane changing behavior is one of the main reasons for traffic accidents. The traditional illegal lane changing detection method and process mainly rely on traffic cameras and monitoring equipment, but it has some limitations, for example, there is no complete system for comprehensive detection of illegal lane changing of vehicles, and in addition, the camera at the roadside sometimes cannot obtain clear illegal vehicle license plate information due to the problem of visual angle, and cannot fully cover the illegal lane changing behavior on the road.

[0003] On the other hand, there are many idle computing resources in intelligent connected vehicles on the road that cannot be used, and vehicle-road cooperation technology also provides a basis for utilizing these resources. Vehicle-road cooperation refers to the information exchange and cooperation between vehicles and road infrastructure. Through the sensors mounted on the vehicle and the communication with the road infrastructure, the vehicle can obtain information about the road environment, and the road infrastructure can also provide real-time road conditions and navigation information to the vehicle.

[0004] Therefore, there is an urgent need for a method and system that utilizes the idle computing resources of intelligent connected vehicles to accurately detect and effectively punish illegal lane changing behavior. SUMMARY

[0005] The purpose of the present application is to utilize intelligent connected vehicles and roadside intelligent devices, as well as vehicle-road cooperation fusion perception, to realize a complete illegal lane changing detection mechanism, fully utilize the perception ability and idle resources of intelligent connected vehicles, and thus improve the traffic safety of the road and save a part of human resource expenditure.

[0006] To achieve the above purpose, the present application provides the following technical solutions:

[0007] A method for detecting illegal lane changing based on vehicle-road cooperation, comprising the following steps:

[0008] S1. Construct a neural network model for detecting illegal lane changing according to the following strategy:

[0009] The model input is a continuous multi-frame image, including two-stage inference. In the first stage, the model performs target detection on each frame of image, and the targets include vehicles, road centerlines and vehicle license plates.

[0010] And obtain relevant information on detected traffic violations, including the center coordinates of the vehicle detection frame, the vehicle's orientation, the set of points on the lane lines, the type of lane lines, and whether the vehicle has a license plate.

[0011] In the second stage, the model takes the vehicle detection boxes and feature vectors of the continuous frames detected in the first stage as input, performs vehicle target association and tracking, and outputs the tracking ID and associated trajectory information of each vehicle target.

[0012] S2 uses image data collected during road tests to initialize and train the neural network model used for detecting traffic violations;

[0013] S3 sends the image data to be detected collected by the road test into the trained neural network model to obtain the detection violation information in the continuous multi-frame images, as well as the trajectory information of each vehicle, and judges whether the vehicle has violated the rules based on the information, including the vehicle crossing the center solid line or the center double solid line, driving in the wrong direction, speeding, and illegal overtaking.

[0014] S4: When a vehicle is determined to be in violation and contains a license plate, the license plate number is identified from the image of the license plate of the detected violating vehicle.

[0015] When a vehicle is determined to be in violation and does not have a license plate, the detected image frame of the vehicle and the annotation information given by the neural network model are transmitted to other nearby vehicles, notifying them to re-identify the vehicle. The nearby vehicles use the vehicle re-identification algorithm to find the violating vehicle, perform target detection, detect the license plate image of the violating vehicle and send it back, and then identify the license plate number on the returned license plate image.

[0016] After identification, the data of vehicles violating traffic rules is uploaded to the cloud.

[0017] Furthermore, the types of lane lines include solid center lines, double solid center lines, solid lines on both sides of the lanes, and lane dividers.

[0018] Furthermore, in step 3, the lane line is set as a straight line, and its coordinate system is a graphical coordinate system. The set of points of the lane line is fitted into a straight line using the least squares method. This yields the slope of the lane line and a coordinate that is both in the set of points of the lane line and on the lane line itself, called the lane line coordinate. Determining whether a vehicle has committed a traffic violation includes the following steps:

[0019] Step 31 uses the following algorithm to determine whether a vehicle is crossing the solid center line or the double solid center line:

[0020] Let the center coordinates of the vehicle detection bounding box be (VX1, VY1), (VX2, VY2), ..., (VX... n VY nThe coordinates and slopes of the center solid line and the center double solid line are (LX1, LY1), ..., (LX1, LY1), respectively. m ,LY m ) and K1,K2…,K o ;

[0021] Based on the coordinates (LX) of the center solid line and the center double solid line i ,LY i ) and slope K i The equation for the solid lane line can be obtained as follows:

[0022] y = K i *(x-LX i )+LY i

[0023] The center coordinates of the detection frame for each vehicle (VX) i VY i Given a solid center line or a double solid center line, calculate the shortest distance from the vehicle's center point to the solid line, corresponding to each other in pairs:

[0024] First, set the coordinates of the vehicle's center point (VX) i VY i Substituting into the equation of the solid line, we obtain the projected coordinates (PX) of the point on the solid line. i ,PY i ):

[0025] PX i =(K i *(K i *VX i -K i *LX i +VY i -LY i )) / (1+K i 2 )

[0026] PY i =K i *(PX i -LX i )+LY i

[0027] Then, calculate the distance d from the vehicle's center point to the solid line:

[0028] d = sqrt((PX) i -VX i ) 2 +(PY i -VY i ) 2 ),

[0029] If the shortest distance d is less than the set threshold, the vehicle is judged to have crossed the solid line;

[0030] Step 32 uses the following algorithm to determine whether the vehicle is traveling in the wrong direction:

[0031] Let the center coordinates of the vehicle detection bounding box be (VX1, VY1), (VX2, VY2), ..., (VX... n VY n The vehicle's travel directions are VF1,...,VF n The coordinates and slope of the solid center line between the forward and reverse lanes are (LX) c ,LY c ) and K c Vehicle direction of travel VF1,...,VF n ∈[-1,1], meaning there are only two possible vehicle directions. Even diagonal directions are considered as the closest straight-ahead direction. Vehicles on either side of the solid center line travel in opposite directions. Assuming the correct direction of travel for the upper lane on the solid center line is 1 and the correct direction of travel for the lower lane is -1, the following calculation can be used to determine if a vehicle is traveling in the wrong direction:

[0032] For each vehicle coordinate (VX) i VY i ), Calculate the value of the equation of the central solid line:

[0033] LValue i =(VY-LY c )-K c *(VX-LX c ),

[0034] If, LValue i >0, then the vehicle (VX) i VY i Above the center solid line, LValue i <0, then the vehicle (VX) i VY i Below the center solid line, LValue i =0, then the vehicle (VX) i VY i On the center solid line;

[0035] If LValue i *VF i A value greater than 0 indicates that the vehicle is not traveling in the wrong direction. i *VF i <0 indicates the vehicle is traveling in the wrong direction;

[0036] Step 32 uses the following algorithm to determine whether the vehicle is speeding:

[0037] First, a series of vehicle numbers are obtained from target tracking, namely V1, V2, ..., V n Let the center coordinates of the vehicle detection bounding box be (VX1, VY1), (VX2, VY2), ..., (VX... n VY n Vehicle V i The coordinates of consecutive frames are represented as Let the speed limit on the road be v. max Then, the following formula can be used to calculate the vehicle speed:

[0038] Calculate vehicle V i Displacement between two frames:

[0039]

[0040] Vehicle V i The time interval between two frames is Δt, from which the vehicle V can be calculated. i speed v i for:

[0041]

[0042] Finally, determine vehicle V. i speed v i Is it greater than the speed limit v? max If it is greater than that, it is judged as speeding;

[0043] Step 34 uses the following algorithm to determine whether the vehicle illegally overtook:

[0044] First, a series of vehicle numbers are obtained from target tracking, namely V1, V2, ..., V n Let the center coordinates of the vehicle detection bounding box be (VX1, VY1), (VX2, VY2), ..., (VX... n VY n This example uses a two-lane roadway in the same direction for ease of understanding; it does not mean this technology cannot be applied to other types of lanes. The coordinates and slopes of the solid lines on both sides of the roadway are (LX1,LY1), (LX2,LY2) and K1,K2, respectively. The coordinates and slopes of the dividing line between the two lanes in the same direction are (LX3,LY3) and K3, respectively.

[0045] The following method can be used to determine the number V. i The lane where the vehicle is located and the order of the vehicles:

[0046] First, calculate the distances from the vehicle to the solid line of the left lane, the solid line of the right lane, and the lane divider line respectively:

[0047] d1=|(VY i -LY1)-K1*(VX i -LX1)|

[0048] d2=|(VY i -LY2)-K2*(VX i -LX2)|

[0049] d3=|(VY i -LY3)-K3*(VX i -LX3)|

[0050] Then determine the vehicle's lane using the following formula:

[0051]

[0052] At this point, based on the corresponding direction of the lane in the coordinate system, the front and rear sequence of vehicles in the left and right lanes can be obtained. Each frame saves a sequence, and a detection is performed every certain number of frames to obtain the vehicle position information in these image frames. The system finds all vehicles in these image frames that are in the left high-speed lane in a certain frame, then appear in the right lane in a subsequent frame, and then return to the left lane. The system then determines whether the distance between these vehicles and the following vehicle after overtaking is less than the safe distance. If it is less than the safe distance, it is judged as overtaking.

[0053] Furthermore, the first stage of inference uses the Yolov5 model.

[0054] Furthermore, the first stage of inference employs the DeepSORT model.

[0055] Furthermore, in step S4, the transmission of the image frame and annotation information of the vehicle violating the traffic rules, as well as the transmission of license plate images to other nearby vehicles, all adopt the PC5 protocol.

[0056] Furthermore, in step S4, the Yolov5+DeepSORT algorithm is used to re-identify vehicles in the vicinity to find the vehicle violating the traffic rules.

[0057] The present invention also discloses a system for detecting illegal lane changes based on vehicle-road cooperation, including a roadside smart camera pole, on which a camera unit, an RSU, and a roadside communication unit are electrically connected in sequence. The RSU can communicate with the OBU on the vehicle through the roadside communication unit via V2I. The system detects illegal lane changes of vehicles by performing the aforementioned method steps.

[0058] Furthermore, the camera unit is a bullet camera.

[0059] This invention fully utilizes the sensing capabilities and idle resources of intelligent connected vehicles to detect illegal lane changes, forming a complete and effective detection mechanism. This improves road traffic safety and saves on human resource expenditures. It also avoids situations where a single perspective fails to obtain clear license plate information of violating vehicles, thus improving detection efficiency and effectiveness. Attached Figure Description

[0060] Figure 1 This is a schematic diagram of a vehicle-road cooperative system for detecting illegal lane changes and its application scenarios, as described in an embodiment of the present invention.

[0061] Figure 2 This is a flowchart of a method for detecting illegal lane changes based on vehicle-road cooperation in an embodiment of the present invention. Detailed Implementation

[0062] Through specific examples, we will describe in detail the embodiments of the present invention, enabling those skilled in the art to easily understand other advantages and effects of the present invention. Furthermore, the present invention can be implemented or applied through different specific embodiments, and various details in this specification can be modified or changed according to different viewpoints and applications, provided that the core idea of ​​the present invention is not departed. It should be noted that the schematic diagrams provided in the following embodiments are only intended to outline the basic concept of the present invention, and features can be combined with each other without conflict.

[0063] The illustrations are for illustrative purposes only and are not actual photographs, nor should they be considered as limiting the scope of the invention. To better illustrate embodiments of the invention, some components in the drawings may be omitted, enlarged, or reduced in size, but this does not represent the dimensions of the actual product. Those skilled in the art will understand that certain known structures and their descriptions may be omitted from the drawings.

[0064] The vehicle-road cooperative system for detecting illegal lane changes in this example is basically as follows: Figure 1 As shown, the system includes a roadside smart camera pole, on which a bullet camera, an RSU, and a roadside communication unit are electrically connected in sequence. The RSU can communicate with the OBU on the vehicle via the roadside communication unit through V2I. The system detects illegal lane changes of vehicles by means of vehicle-road cooperative detection, and then uploads the relevant information of the violation to a remote location via 5G communication.

[0065] In this example, the basic process of detecting illegal lane changes based on vehicle-road cooperation is as follows: Figure 2As shown, in the early model building stage, image data needs to be collected through roadside smart camera poles to establish a dataset. The collected data is divided into training set, test set and validation set to initialize the model used to detect violations, hereinafter referred to as the detection model. The training set, test set and validation set are divided according to a certain ratio. For example, if the total dataset has 1000 images, then 700 images can be allocated to the training set, 100 images to the test set and 200 images to the validation set.

[0066] The roadside smart camera pole in this example includes, but is not limited to, a bullet camera, a Roadside Unit (RSU), a GPS device, a roadside communication unit, and a storage unit. The bullet camera in this example is used for target capture and video detection, with a frame rate of 30fps / s and a resolution of 1920*1080. The RSU provides computing power on the roadside, executing algorithms such as target detection and tracking. The roadside communication unit communicates with the vehicle, the GPS device determines the location of the roadside equipment, and the storage unit stores deep learning model weights, algorithm code, and some temporary information.

[0067] The input to the detection model in this example is a series of image frames. These image frames need to be preprocessed and manually labeled. The image preprocessing used in this example is to scale the resolution of the input model's image to 640*640 proportionally and fill the empty areas with gray.

[0068] If the length and width of the input image are set to L and W respectively, the scaling ratio is calculated using the following formula:

[0069]

[0070] Here, it is assumed that the length of the input image is always greater than its width. The selected scaling factor Scale3 is the smaller of the two factors, i.e., Scale1, as shown in the formula below:

[0071] Scale3=Min(Scale1,Scale2)=Scale1

[0072] The scaled image is 640 pixels long and 100 pixels wide. Then we can obtain the pixel height that needs to be filled at both ends of the image. The preprocessing part is then completed by filling the top and bottom sides of the image with gray pixels of height H1. The final image size input to the model is 3*640*640.

[0073] In this example, the software used to annotate the image is Labelimg, an image annotation tool that generates XML annotation files.

[0074] The detection model in this example employs a two-stage inference process using YOLOv5 and DeepSort. After preprocessing, consecutive image frames first enter the Backbone network of the YOLOv5 model for feature extraction, resulting in feature maps M1 (20x20), M2 (40x40), and M3 (80x80). These three feature maps are then fed into the Neck network of the YOLOv5 model for feature fusion. The fused feature map is then fed into the Head network to predict a series of bounding box locations and object categories. The Non-Maximum Suppression (NMS) algorithm is used to select the most representative bounding boxes. NMS selects the optimal bounding boxes based on confidence scores and overlap. In this part of the Yolov5 model inference, we can obtain information such as the center coordinates of the vehicle detection box, the vehicle's orientation, the set of points of the lane lines (including the center solid line, the double center solid line, the solid lines of the two side lanes, and the lane divider), and whether the vehicle has a license plate. Since most lanes are straight lines, and in order to reduce the computational burden, we set the lane lines as straight lines and the coordinate system as a graph coordinate system. Here, we use the least squares method to fit the set of points of the lane lines into a straight line, which can give us the slope of the lane lines, as well as a coordinate that is both in the set of points of the lane lines and on the lane lines. This coordinate is called the lane line coordinate.

[0075] Next, the vehicle detection boxes and their corresponding feature vectors from multiple consecutive frames detected by YOLOv5 are used as input, and the DeepSort algorithm is applied for target association and tracking. The DeepSort algorithm uses a Kalman filter to estimate and predict the target's state, and a Hungarian algorithm to associate the target with its trajectory. By calculating the similarity between feature vectors and combining the prediction results from the Kalman filter, DeepSort can establish a coordinate trajectory for each vehicle target and update the trajectory's state. The tracking ID and associated trajectory information for each target can then be output. This allows us to obtain the vehicle ID and coordinates at each time point in consecutive frames.

[0076] Successive image frames are first processed by the YOLOv5 model. After preprocessing, the images are fed into the Backbone network for feature extraction, resulting in three feature maps: M1 (20x20), M2 (40x40), and M3 (80x80). These three feature maps are then fused into the Neck network. Finally, the three feature maps are fed into the Head network to predict bounding box locations and object categories. The Non-Maximum Suppression (NMS) algorithm is used to select the most representative bounding boxes. NMS selects the best bounding boxes based on confidence scores and overlap. In this part of the Yolov5 model inference, we can obtain information such as the center coordinates of the vehicle detection box, the vehicle's orientation, the set of points of the lane line, the different types of lane lines (including solid center line, double solid center line, solid lines on both sides of the lane, and lane divider), and whether the vehicle has a license plate. Since most lanes are straight lines, and in order to reduce the computational burden, we set the lane lines as straight lines and the coordinate system as a graph coordinate system. Here, we use the least squares method to fit the set of points of the lane line into a straight line, which can give us the slope of the lane line and a coordinate that is both in the set of points of the lane line and on the lane line. This coordinate is called the lane line coordinate.

[0077] Using the vehicle detection bounding boxes and their feature vectors from consecutive frames detected by YOLOv5 as input, the DeepSort algorithm is applied for target association and tracking. DeepSort uses a Kalman filter for target state estimation and prediction, and a Hungarian algorithm for target-trajectory association. By calculating the similarity between feature vectors and combining the Kalman filter prediction results, DeepSort can establish a coordinate trajectory for each vehicle target and update the trajectory state. It then outputs the tracking ID and associated trajectory information for each target. This allows obtaining the vehicle ID and coordinates at each time point in consecutive frames.

[0078] After obtaining the trained detection model, the smart camera pole inputs the captured images into the detection model using the optimal weights, resulting in object detection and tracking results. These results include the center coordinates of the vehicle detection box, the vehicle's orientation, the vehicle ID at each time point in consecutive frames, the center coordinates of the vehicle detection box, and whether the vehicle contains license plate information. From the first frame image, the slope of the lane lines (including the center solid line, double center solid lines, solid lines on both sides of the lanes, and the lane divider) and a coordinate that is both in the lane line point set and on the lane line itself are obtained; this is called the lane line coordinate.

[0079] After obtaining the detection results, it is necessary to determine whether the vehicle has violated lane change regulations, including crossing the solid center line, crossing double solid center lines, driving in the wrong direction, speeding, and illegal overtaking. After determining that the vehicle has violated regulations, the system will output the type of violation, the coordinates of the detection box of the violating vehicle, and the length and width of the detection box, thereby cutting out the image of the violating vehicle.

[0080] The complete system workflow is as follows: First, the intelligent camera pole acquires continuous image frames and inputs them into the model. Using different detection heads of the Yolov5 model and the DeepSORT model, information such as the center coordinates of the vehicle detection box, the vehicle's orientation, the set of points for lane lines (including the center solid line, double center solid lines, solid lines on both sides of the lane, and lane dividers), whether the vehicle has a license plate, and the vehicle ID at each time point in the continuous frames are extracted. This information is then input into algorithms for detecting illegal overtaking, double solid line crossing, speeding, and wrong-way driving, respectively, to obtain relevant information about the violating vehicle, including the image frame in which the violating vehicle appears, the coordinates of the detection box, and its length and width.

[0081] Then, these image frames are input into the YOLOv5 model to determine whether the smart camera pole has captured the license plate of the violating vehicle. If the license plate is captured, it is cropped out and the license plate number is extracted using OpenCV on the smart camera pole. If the license plate is not captured, the smart camera pole transmits the detected image information frame of the vehicle and the YOLOv5+DeepSORT annotation information to other nearby vehicles via V2I, notifying them to re-identify the vehicle. Nearby vehicles use the YOLOv5+DeepSORT vehicle re-identification algorithm to find the violating vehicle, and use YOLOv5 to determine whether it contains a license plate. They then transmit the image of the violating vehicle with the license plate to the smart camera pole via V2I. The smart camera pole inputs the image into the YOLOv5 model, locates the license plate portion, and uses OpenCV to recognize the license plate number. After recognition, the violating vehicle data is uploaded to the cloud service center for processing and penalty. In the following algorithm, the coordinate system is a graphical coordinate system, and all lane lines are straight lines.

[0082] Based on the obtained vehicle detection frame coordinates and the coordinates and slope of the lane lines (solid center line, double solid center line), it is determined whether the vehicle has crossed the solid center line or the double solid center line.

[0083] The following algorithm is used to determine whether a vehicle is crossing the solid center line or the double solid center line:

[0084] Let the center coordinates of the vehicle detection bounding box be (VX1, VY1), (VX2, VY2), ..., (VX... n VY nThe coordinates and slopes of the center solid line and the center double solid line are (LX1, LY1), ..., (LX1, LY1), respectively. m ,LY m ) and K1,K2…,K o ;

[0085] Based on the coordinates (LX) of the center solid line and the center double solid line i ,LY i ) and slope K i The equation for the solid lane line can be obtained as follows:

[0086] y = K i *(x-LX i )+LY i

[0087] For the center coordinates of the detection frame of each vehicle (VX) i VY i );

[0088] For each detected vehicle and its corresponding solid center line or double solid center line, calculate the shortest distance from the vehicle's center point to the solid line:

[0089] First, set the coordinates of the vehicle's center point (VX) i VY i Substituting into the equation of the solid line, we obtain the projected coordinates (PX) of the point on the solid line. i ,PY i ):

[0090] PX i =(K i *(K i *VX i -K i *LX i +VY i -LY i )) / (1+K i 2 )

[0091] PY i =K i *(PX i -LX i )+LY i

[0092] Then, calculate the distance d from the vehicle's center point to the solid line:

[0093] d = Sqrt((PX) i -VX i ) 2 +(PY i -VY i ) 2 )

[0094] The threshold is set to 0.7 meters. If the shortest distance d is less than the set threshold, the vehicle is judged to have crossed the solid line.

[0095] Based on the obtained vehicle direction and coordinates, as well as the coordinates and slope of the center solid line, the lane in which the vehicle is located is calculated to determine whether the vehicle is traveling in the wrong direction. The following algorithm is used for this determination:

[0096] Let the center coordinates of the vehicle detection bounding box be (VX1, VY1), (VX2, VY2), ..., (VX... n VY n The vehicle's travel directions are VF1,...,VF n The coordinates and slope of the solid center line between the forward and reverse lanes are (LX) c ,LY c ) and K c .

[0097] Assume the vehicle's direction of travel is VF1,...,VF n ∈[-1,1], meaning there are only two possible directions of vehicle travel, and even diagonal directions will be judged as the closer to the straight direction. Vehicles on either side of the solid center line travel in opposite directions. Assuming the correct travel direction of the upper lane on the solid center line is 1 and the correct travel direction of the lower lane is -1, the following calculation can be used to determine if a vehicle is traveling in the wrong direction:

[0098] For each vehicle coordinate (VX) i VY i Perform the following operations:

[0099] Calculate the value of the equation of the central solid line, LValue. i :

[0100] LValue i =(VY-LY c )-K c *(VX-LX c )

[0101] LValue i >0, then the vehicle (VX) i VY i Above the center solid line;

[0102] LValue i <0, then the vehicle (VX) i VY i ) Below the center solid line;

[0103] LValue i =0, then the vehicle (VX)i VY i ) on the center solid line.

[0104] If LValue i *VF i A value greater than 0 indicates that the vehicle is not traveling in the wrong direction; LValue i *VF i <0 indicates the vehicle is traveling in the wrong direction.

[0105] By using the vehicle ID and coordinates at each time point in consecutive frames obtained from target tracking, the following algorithm can be used to determine whether speeding has occurred:

[0106] First, a series of vehicle numbers are obtained from target tracking, namely V1, V2, ..., V n Let the center coordinates of the vehicle detection bounding box be (VX1, VY1), (VX2, VY2), ..., (VX... n V Y n), vehicle V i The coordinates of consecutive frames are represented as Let the speed limit on the road be v. max Then, the following formula can be used to calculate the vehicle speed:

[0107] Calculate vehicle V i Displacement between two frames:

[0108]

[0109] Vehicle V i The time interval between two frames is Δt, from which the vehicle V can be calculated. i speed v i for:

[0110]

[0111] Finally, determine vehicle V. i speed v i Is it greater than the speed limit v? max If it is greater than that, it is judged as speeding.

[0112] By using the vehicle ID and coordinates at each time point in consecutive frames obtained from target tracking, as well as the coordinates and slopes of lane lines (including solid center line, double solid center line, solid lines on both sides of the lane, and lane divider), the following algorithm can be used to determine whether an overtaking violation has occurred:

[0113] First, a series of vehicle numbers are obtained from target tracking, namely V1, V2, ..., V n Let the center coordinates of the vehicle detection bounding box be (VX1, VY1), (VX2, VY2), ..., (VY1, VY2).n VY n This example uses a two-lane roadway in the same direction for ease of understanding, and does not mean that this technology cannot be applied to other types of lanes. The coordinates and slopes of the solid lines on both sides of the lanes are (LX1,LY1), (LX2,LY2) and K1,K2, respectively, and the coordinates and slopes of the dividing line between the two lanes in the same direction are (LX3,LY3) and K3, respectively.

[0114] The following method can be used to determine the number V. i The lane where the vehicle is located and the order of the vehicles:

[0115] First, calculate the distances from the vehicle to the solid line of the left lane, the solid line of the right lane, and the lane divider line respectively:

[0116] d1=|(VY i -LY1)-K1*(VX i -LX1)|

[0117] d2=|(VY i -LY2)-K2*(VX i -LX2)|

[0118] d3=|(VY i -LY3)-K3*(VX i -LX3)|

[0119] Then determine the vehicle's lane using the following formula:

[0120]

[0121] At this point, based on the lane direction and its corresponding direction in the coordinate system, the front-to-back sequence of vehicles in the left and right lanes can be obtained. One sequence is saved for each frame, and a detection is performed every 300 frames to obtain the vehicle position information of the previous 300 frames. The system identifies all vehicles that were in the left high-speed lane in one frame, then appeared in the right lane in subsequent frames, and then returned to the left lane. For these vehicles, it is then determined whether the distance between them and the vehicle behind them after overtaking is less than the safe distance. If it is less than the safe distance, it is considered an overtaking maneuver.

[0122] When the intelligent camera device determines that a vehicle has violated traffic regulations, and the image captured by the intelligent camera pole fails to capture the license plate of the violating vehicle, it will transmit the detected image information frame of the vehicle and the YOLOv5+DeepSORT annotation information to other nearby vehicles via V2I. These nearby vehicles will then be notified to re-identify the vehicle. The nearby vehicles will use the YOLOv5+DeepSORT vehicle re-identification algorithm to locate the violating vehicle and transmit the image of the violating vehicle with its license plate to the intelligent camera pole via V2I. The intelligent camera pole will input the image into the YOLOv5 model, locate the license plate portion, and use OpenCV to recognize the license plate number. After recognition, the data of the violating vehicle will be uploaded to the cloud service center for penalty processing.

[0123] The vehicle and the roadside smart camera pole communicate via the PC5 protocol, using the OBU on the vehicle and the RSU on the smart camera pole.

[0124] The smart camera pole acquires the vehicle's violation category, the coordinates of the detection box, and the length and width of the detection box obtained in step S4. Then, it transmits the image frame of the violating vehicle and the Yolov5+DeepSort annotation information to other nearby vehicles via V2I, notifying them to execute the Yolov5+DeepSORT algorithm to re-identify the vehicle and find the violating vehicle.

[0125] After the nearby vehicles find the image of the violating vehicle using the YOLOv5+DeepSORT re-identification model, the YOLOv5 model is used to detect whether the image contains a license plate. If not, the YOLOv5+DeepSORT re-identification model will continue to be executed until an image of the violating vehicle with a license plate is detected or the smart camera pole notifies that an image of the violating vehicle with a license plate has been obtained.

[0126] On the OBU, license plate recognition is implemented using image processing with Python and OpenCV. After extracting the portion of the image from the license plate location detected by Yolov5, thresholding, edge detection, and morphological operations are performed on the image. Based on the obtained contour features, the specific location of the license plate is identified, and the license plate is segmented. Then, image denoising and grayscale processing are performed on the image to segment the license plate characters. Finally, template matching is used to recognize and output the characters.

[0127] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A method for detecting illegal lane changes based on vehicle-road cooperation, characterized in that, Includes the following steps: S1 constructs a neural network model for detecting traffic violations using the following strategy: The model input consists of a series of images and includes two-stage inference. In the first stage, the model performs target detection on each image frame. The targets include vehicles, road center lines, and vehicle license plates. The system also obtains information related to detected traffic violations, including the center coordinates of the vehicle detection frame, the vehicle's orientation, the set of points on the lane lines, the type of lane lines, and whether the vehicle has a license plate. The types of lane lines include solid center lines, double solid center lines, solid lines on both sides of the lanes, and lane dividers. In the second stage, the model takes the vehicle detection boxes and feature vectors of the continuous frames detected in the first stage as input, performs vehicle target association and tracking, and outputs the tracking ID and associated trajectory information of each vehicle target. S2 uses image data collected during road tests to initialize and train the neural network model used for detecting traffic violations; S3 sends the image data to be detected collected by the road test into the trained neural network model to obtain the detection violation information in the continuous multi-frame images, as well as the trajectory information of each vehicle, and judges whether the vehicle has violated the rules based on this information, including the vehicle crossing the center solid line or the center double solid line, driving in the wrong direction, speeding, and illegal overtaking. In this process, lane lines are defined as straight lines, and the coordinate system is a graphical coordinate system. The set of points along the lane lines is fitted to a straight line using the least squares method. This yields the slope of the lane lines and a coordinate that exists both within the set of points along the lane lines, called the lane line coordinate. Determining whether a vehicle has committed a traffic violation involves the following steps: Step 31 uses the following algorithm to determine whether a vehicle is crossing the solid center line or the double solid center line: Let the center coordinates of the vehicle detection bounding box be... The coordinates and slopes of the center solid line and the center double solid line are respectively and ; Based on the coordinates of the center solid line and the center double solid line and slope The equation for the solid lane line can be obtained as follows: ， The center coordinates of the detection frame for each vehicle For each pair of solid center lines or double solid center lines, calculate the shortest distance from the vehicle's center point to the solid line: First, set the coordinates of the vehicle's center point. Substituting into the equation of the solid line, we obtain the projected coordinates of the point on the solid line. : ， ， Then, calculate the distance from the vehicle's center point to the solid line. : ， If the shortest distance If the value is less than the set threshold, the vehicle is judged to have crossed the solid line; Step 32 uses the following algorithm to determine whether the vehicle is traveling in the wrong direction: Let the center coordinates of the vehicle detection bounding box be... The directions of travel of the vehicles are respectively The coordinates and slope of the solid center line between the forward and reverse lanes are respectively... and Vehicle direction of travel This means there are only two possible directions of vehicle travel: diagonal travel is also considered as the closer to straight travel. Vehicles on either side of the solid center line travel in opposite directions. Assuming the correct travel direction of the upper lane of the solid center line is 1 and the correct travel direction of the lower lane is -1, the following calculation can be used to determine whether a vehicle is traveling in the wrong direction: For each vehicle coordinate Calculate the value of the equation of the central solid line: ， like, Then the vehicle Above the center solid line Then the vehicle Below the center solid line Then the vehicle On the center solid line; like This means the vehicle was not traveling in the wrong direction. This indicates that the vehicle is traveling in the wrong direction. Step 32 uses the following algorithm to determine whether the vehicle is speeding: First, a series of vehicle numbers are obtained through target tracking, namely... Let the center coordinates of the vehicle detection bounding box be... ,vehicle The coordinates of consecutive frames are represented as Set the road speed limit as Then, the following formula can be used to calculate the vehicle speed: Calculate vehicles Displacement between two frames: ， ， vehicle The time interval between the two frames is From this, the vehicle can be calculated. speed for: ， Finally, the vehicle was judged. speed Is it above the speed limit? If it is greater than that, it is judged as speeding; Step 34 uses the following algorithm to determine whether the vehicle illegally overtook: First, a series of vehicle numbers are obtained through target tracking, namely... Let the center coordinates of the vehicle detection bounding box be... This example uses a two-lane road in the same direction for ease of understanding; it does not mean this technology cannot be applied to other types of lanes. The coordinates and slopes of the solid lines on both sides of the road are as follows: and The coordinates and slope of the dividing line between the two lanes in the same direction are as follows: and ; The following method can be used to determine the number. The lane where the vehicle is located and the order of the vehicles: First, calculate the distances from the vehicle to the solid line of the left lane, the solid line of the right lane, and the lane divider line respectively: ， ， ， Then determine the vehicle's lane using the following formula: ， At this point, based on the corresponding direction of the lane in the coordinate system, the front and rear sequence of vehicles in the left and right lanes can be obtained. Each frame saves a sequence, and a detection is performed every certain number of frames to obtain the vehicle position information in these image frames. The system finds all vehicles in these image frames that are in the left high-speed lane in a certain frame, then appear in the right lane in a subsequent frame, and then return to the left lane. The system then determines whether the distance between these vehicles and the following vehicle after overtaking is less than the safe distance. If it is less than the safe distance, it is judged as overtaking. S4: When a vehicle is determined to be in violation and contains a license plate, the license plate number is identified from the image of the license plate of the detected violating vehicle. When a vehicle is determined to be in violation and does not have a license plate, the detected image frame of the vehicle and the annotation information given by the neural network model are transmitted to other nearby vehicles, notifying them to re-identify the vehicle. The nearby vehicles use the vehicle re-identification algorithm to find the violating vehicle, perform target detection, detect the license plate image of the violating vehicle and send it back, and then identify the license plate number on the returned license plate image. After identification, the data of vehicles violating traffic rules is uploaded to the cloud.

2. The method according to claim 1, characterized in that, The first stage of inference uses the Yolov5 model.

3. The method according to claim 2, characterized in that, The second stage of inference uses the DeepSORT model.

4. The method according to claim 1, characterized in that, In step S4, the transmission of the image frame and annotation information of the vehicle violating the traffic rules, as well as the transmission of license plate images to other nearby vehicles, all use the PC5 protocol.

5. The method according to claim 1, characterized in that, In step S4, the Yolov5+DeepSORT algorithm is used to re-identify other nearby vehicles to find the vehicle that violated the rules.

6. A system for detecting illegal lane changes based on vehicle-road cooperation, characterized in that, The system includes a roadside smart camera pole, on which a camera unit, an RSU, and a roadside communication unit are electrically connected in sequence. The RSU can communicate with the OBU on the vehicle via the roadside communication unit through V2I. The system detects illegal lane changes of vehicles by performing the steps of the method described in any of claims 1-5.

7. The system according to claim 6, characterized in that, The camera unit is a bullet camera.

Images