System and method for detecting illegal lane change based on vehicle infrastructure cooperation

Abstract

The invention belongs to the field of intelligent vehicle-road cooperation, and particularly relates to a system and method for detecting illegal lane change based on vehicle-road cooperation, and the method comprises the following steps: constructing a neural network model for detecting illegal lane change, the model input is continuous multi-frame images, the method comprises two-stage reasoning, in the first stage, the model carries out the target detection of each frame of image, and in the second stage, the model carries out the target detection of each frame of image; in the second stage, the model associates and tracks the vehicle targets, and outputs the tracking ID and associated track information of each vehicle target; training a neural network model used for violation detection; and sending to-be-detected image data acquired by a road test to the trained neural network model to obtain violation detection related information in continuous multi-frame images and track information of each vehicle, judging whether the vehicle has a violation behavior or not according to the information, and uploading violation vehicle data to the cloud after identification. And the sensing capability and idle resources of the intelligent network connection automobile are fully utilized, so that the traffic safety of the road can be improved.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent vehicle-road collaboration, and specifically relates to a system and method for detecting illegal lane changes based on vehicle-road collaboration. Background Art

[0002] In modern society, the increase in road traffic violations and the frequent occurrence of traffic accidents have become a serious social problem. Among them, illegal lane change is one of the main causes of traffic accidents. Traditional illegal lane change detection methods and processes mainly rely on traffic cameras and monitoring equipment, but they have some limitations. For example, there is no complete system for comprehensive detection of illegal lane changes of vehicles. In addition, due to the viewing angle problem, the roadside camera sometimes cannot obtain clear license plate information of the illegal vehicle and cannot fully cover the illegal lane change behaviors on the road.

[0003] On the other hand, there are many idle computing resources in intelligent connected cars on the road that have nowhere to be used, and vehicle-road collaboration technology also provides a basis for utilizing these resources. Vehicle-road collaboration refers to the information exchange and collaborative work between vehicles and road infrastructure. Through the sensors carried on the vehicle and the communication with the road infrastructure, the vehicle can obtain information about the road environment, and the road infrastructure can also provide real-time road conditions and navigation information to the vehicle.

[0004] Therefore, there is an urgent need for a method and system that utilizes the idle computing resources of intelligent connected vehicles to accurately detect and effectively punish illegal lane changing behaviors. Summary of the invention

[0005] The purpose of the present invention is to utilize intelligent connected vehicles and roadside intelligent equipment, as well as vehicle-road collaborative fusion perception, to realize a complete illegal lane change detection mechanism, fully utilize the perception capabilities and idle resources of intelligent connected vehicles, thereby improving road traffic safety and saving a portion of human resource expenditure.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A method for detecting illegal lane changes based on vehicle-road collaboration includes the following steps:

[0008] S1 builds a neural network model for detecting violations according to the following strategy:

[0009] The model input is a continuous multi-frame image, including two-stage reasoning. In the first stage, the model detects targets in each frame of the image, including vehicles, road center lines and vehicle license plates.

[0010] and obtaining information related to the detection of traffic violations, the information related to the detection of traffic violations including the coordinates of the center of the vehicle detection frame, the positive and negative directions of the vehicle, the point set of the lane line, the type of the lane line, and whether the vehicle has a license plate;

[0011] In the second stage, the model takes the vehicle detection frame of the continuous frames detected in the first stage and the feature vector of the detection frame as input, associates and tracks the vehicle targets, and outputs the tracking ID and associated trajectory information of each vehicle target;

[0012] S2 uses the image data collected by the road test to initialize the training of the neural network model used to detect violations;

[0013] S3 sends the image data to be detected collected by the road test to the trained neural network model, obtains the detection violation related information in the continuous multi-frame image, and the trajectory information of each vehicle, and judges whether the vehicle has violated the traffic rules based on the information, including the vehicle crossing the center solid line or the center double solid line, driving in the wrong direction, speeding, and illegal overtaking;

[0014] S4: When a vehicle is determined to be in violation of traffic rules and has a license plate, the license plate number is recognized from the license plate image of the detected vehicle in violation of traffic rules;

[0015] When a vehicle is judged to be in violation of traffic rules and does not have a license plate, the detection image frame of the vehicle and the annotation information given by the neural network model are transmitted to other nearby vehicles, and other nearby vehicles are notified to re-identify the vehicle; nearby vehicles use the vehicle re-identification algorithm to find the illegal vehicle, perform target detection, detect the license plate image of the illegal vehicle and send it back, and then identify the license plate number on the returned license plate image;

[0016] After identification, the data of the illegal vehicles will be uploaded to the cloud.

[0017] Furthermore, the types of lane lines include a solid center line, a double solid center line, solid lane lines on both sides, and a lane dividing line.

[0018] Furthermore, in step 3, the lane line is set as a straight line, and the coordinate system is the image coordinate system. The point set of the lane line is fitted into a straight line using the least squares method, and the slope of the lane line and a coordinate that is both in the lane line point set and on the lane line can be obtained, which is called the lane line coordinate. Determining whether the vehicle has violated the traffic rules includes the following steps:

[0019] Step 31 uses the following algorithm to determine whether a vehicle is pressing the center solid line or the center double solid line:

[0020] Assume that the center coordinates of the vehicle detection frame detected by the target are (VX1, VY1), (VX2, VY2),…, (VX n ,VY n), the coordinates and slopes of the central solid line and the central double solid line are (LX1,LY1),…,(LX m ,LY m ) and K1, K2…, K o ;

[0021] According to the coordinates of the center solid line and the center double solid line (LX i ,LY i ) and slope K i , we can get the equation of the lane solid line:

[0022] y=K i *(x-LX i )+LY i

[0023] The center coordinates of the detection box of each vehicle (VX i ,VY i ) and the center solid line or the center double solid line, correspond to each other, and calculate the shortest distance from the center point of the vehicle to the solid line:

[0024] First, the coordinates of the vehicle center point (VX i ,VY i ) into the real line equation to obtain the projection coordinates of the point on the real line (PX i ,PY i ):

[0025] PX i =(K i *(K i *VX i -K i *LX i +VY i -LY i )) / (1+K i 2 )

[0026] PY i =K i *(PX i -LX i )+LY i

[0027] Then, calculate the distance d from the center point of the vehicle to the solid line:

[0028] d = sqrt((PX i -VX i ) 2 +(PY i -VY i ) 2 ),

[0029] If the shortest distance d is less than the set threshold, it is determined that the vehicle has crossed the solid line;

[0030] Step 32 uses the following algorithm to determine whether the vehicle is in a reverse state:

[0031] Assume that the center coordinates of the vehicle detection frame detected by the target are (VX1, VY1), (VX2, VY2),…, (VX n ,VY n ), the directions of the vehicles are VF1,...,VF n , the coordinates and slopes of the center solid line between the forward and reverse lanes are (LX c ,LY c ) and K c , vehicle travel direction VF1,...,VF n ∈[-1,1], that is, there are only two possible driving directions of the vehicle. The oblique direction is also judged as the approaching straight direction. The vehicles on both sides of the center solid line are driving in opposite directions. Assuming that the correct driving direction of the upper lane on the center solid line is 1, and the correct driving direction of the lower lane is -1, the following calculation can be used to determine whether the vehicle is driving in the wrong direction:

[0032] For each vehicle coordinate (VX i ,VY i ), calculate the value of the central solid line equation:

[0033] LValue i =(VY-LY c )-K c *(VX-LX c ),

[0034] If LValue i >0, then the vehicle (VX i ,VY i ) is on the upper side of the center solid line, LValue i <0, then the vehicle (VX i ,VY i ) is below the center solid line, LValue i =0, then the vehicle (VX i ,VY i ) on the solid centre line;

[0035] If LValue i *VF i >0, it means the vehicle is not going against the flow, LValue i *VF i <0, indicating that the vehicle is traveling in the wrong direction;

[0036] Step 32 uses the following algorithm to determine whether the vehicle is speeding:

[0037] First, a series of vehicle numbers are obtained by target tracking, namely V1, V2, ..., V n , let the center coordinates of the vehicle detection frame detected by the target be (VX1, VY1), (VX2, VY2),…, (VX n ,VY n ), vehicle V i The coordinates of the consecutive frames are expressed as Let the road speed limit be v max , the following formula can be used to calculate the vehicle speed:

[0038] Calculate vehicle V i The displacement between two frames:

[0039]

[0040] Vehicle V i The time interval between two frames is Δt, from which the vehicle V i The speed v i for:

[0041]

[0042] Finally, determine the vehicle V i The speed v i Is it greater than the speed limit v max , if it is greater than, it is considered as speeding;

[0043] Step 34 uses the following algorithm to determine whether the vehicle is overtaking illegally:

[0044] First, a series of vehicle numbers are obtained by target tracking, namely V1, V2, ..., V n , let the center coordinates of the vehicle detection frame detected by the target be (VX1, VY1), (VX2, VY2),…, (VX n ,VY n ), the case of a two-lane road in the same direction is taken here just for the sake of understanding, and it does not mean that this technology cannot be applied to other types of lanes. The coordinates and slopes of the solid lines of the lanes on both sides are (LX1, LY1), (LX2, LY2) and K1, K2 respectively, and the coordinates and slopes of the lane dividing line in the middle of the two-lane road in the same direction are (LX3, LY3) and K3 respectively;

[0045] The following method is used to determine the number V i The lanes where the vehicles are located and the order of the vehicles:

[0046] First, calculate the distance from the vehicle to the solid line of the left lane, the solid line of the right lane, and the lane dividing line:

[0047] d1=|(VY i -LY1)-K1*(VX i -LX1)|

[0048] d2=|(VY i -LY2)-K2*(VX i -LX2)|

[0049] d3=|(VY i -LY3)-K3*(VX i -LX3)|

[0050] Then determine the lane the vehicle is in according to the following formula:

[0051]

[0052] At this time, according to the corresponding direction of the lane direction in the coordinate system, the front and rear sorting sequence of vehicles in the left and right lanes can be obtained. A sorting sequence is saved in each frame, and a detection is performed every certain number of frames to obtain the vehicle position information in these image frames. Find out all the vehicles in these image frames that are in the left lane high-speed lane in a certain frame, and then appear in the right lane in subsequent frames and then return to the left lane. Then, judge whether the distance between these vehicles and the rear vehicle after overtaking is less than the safe distance. If it is less than the safe distance, it is judged as overtaking.

[0053] Furthermore, the Yolov5 model is used in the first stage of reasoning.

[0054] Furthermore, the DeepSORT model is used in the first stage of reasoning.

[0055] Furthermore, in step S4, the transmission of the image frame and the annotation information of the illegal vehicle and the transmission of the license plate partial image to other nearby vehicles all adopt the PC5 protocol;

[0056] Furthermore, in step S4, Yolov5+DeepSORT algorithm is used to re-identify other nearby vehicles to find illegal vehicles.

[0057] The present invention also discloses a system for detecting illegal lane changes based on vehicle-road collaboration, including a roadside intelligent camera pole, on which a camera unit, an RSU and a roadside communication unit electrically connected in sequence are provided, and the RSU can perform V2I communication with the OBU on the vehicle through the roadside communication unit; the system detects the illegal lane change of the vehicle by executing the aforementioned method steps.

[0058] Furthermore, the camera unit is a gun-type camera.

[0059] The present invention makes full use of the perception capabilities and idle resources of intelligent networked vehicles to detect illegal lane changes, forming a complete and effective detection mechanism, thereby improving road traffic safety and saving a portion of human resource expenditure. It also avoids the situation where clear license plate information of illegal vehicles cannot be obtained due to a single viewing angle, thereby improving detection efficiency and effectiveness. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a schematic diagram of a system for detecting illegal lane changes based on vehicle-road collaboration and its application scenario in an embodiment of the present invention.

[0061] Figure 2 This is a flow chart of a method for detecting illegal lane changes based on vehicle-road collaboration in an embodiment of the present invention. DETAILED DESCRIPTION

[0062] Through specific examples, we will explain the embodiments of the present invention in detail so that those skilled in the art can easily understand other advantages and effects of the present invention. In addition, the present invention can also be implemented or applied through different specific embodiments, and the various details in this specification can be modified or changed according to different viewpoints and applications, provided that they do not deviate from the core idea of ​​the present invention. It should be noted that in the following embodiments, the schematic diagrams we provide are only intended to summarize the basic concept of the present invention, and features can be combined with each other without conflict.

[0063] The illustrations are only used as examples and are only schematic diagrams, not actual pictures, and should not be considered as limiting the present invention. In order to better illustrate the embodiments of the present invention, we may omit, enlarge or reduce some parts in the drawings, but this does not represent the size of the actual product. For those skilled in the art, it is understandable that some known structures and their descriptions are omitted in the drawings.

[0064] The system for detecting illegal lane changes based on vehicle-road collaboration in this example is basically as follows Figure 1 As shown, it includes a roadside intelligent camera pole, on which a gun-type camera, an RSU and a roadside communication unit are electrically connected in sequence. The RSU can perform V2I communication with the OBU on the vehicle through the roadside communication unit; the system detects the vehicle's illegal lane change through a method for detecting illegal lane changes based on vehicle-road collaboration, and then uploads the violation-related information to the remote end through 5G communication.

[0065] In this example, the method process of detecting illegal lane changes based on vehicle-road collaboration is basically as follows: Figure 2As shown in the figure, in the early stage of model building, it is necessary to collect image data through roadside intelligent camera poles to establish a data set, and divide the collected data into training set, test set and validation set for initializing the model used for training violation detection, hereinafter referred to as the detection model; the training set, test set and validation set are divided according to a certain ratio. For example, if the total data set has 1,000 pictures, 700 training sets, 100 test sets and 200 validation sets can be allocated.

[0066] The roadside smart camera pole in this example includes a gun-type camera, RSU, GPS device, roadside communication unit and storage unit, but is not limited to this. The gun-type camera in this example is used for target capture and video detection, with a frame rate of 30fps / s and a resolution of 1920*1080; RSU is used to provide computing power on the roadside and execute algorithms such as target detection and target tracking on the roadside. The roadside communication unit is used to communicate with the vehicle, the GPS device is used to determine the location of the roadside device, and the storage unit is used to save the deep learning model weights, algorithm code and some temporary information.

[0067] The input of the detection model in this example is continuous image frames, which need to be preprocessed and manually labeled. The image preprocessing used in this example is to scale the resolution of the input model image to 640*640 and fill the empty space with gray;

[0068] Assuming the input image length and width are L and W respectively, the scaling ratio is calculated by the following formula:

[0069]

[0070] Here we assume that the length of the input image is always greater than the width, and the selected scaling factor Scale3 is the smaller of the two factors, that is, Scale1, and the formula is as follows:

[0071] Scale3=Min(Scale1,Scale2)=Scale1

[0072] The length of the scaled image is 640 and the width is Then we can get the pixel height that needs to be filled at both ends of the image. Then the preprocessing part is completed by filling the upper and lower sides of the image with gray pixels of height H1. The final size of the image input to the model is 3*640*640.

[0073] In this example, the software used to annotate the image is Labelimg, which is an image annotation tool that generates an xml annotation file.

[0074] The detection model in this example uses Yolov5+DeepSort two-stage reasoning. After preprocessing, the continuous image frames first enter the Backbone network in the Yolov5 model for feature extraction, and finally obtain the feature map M1 of 20*20 size, the feature map M2 of 40*40 size, and the feature map M3 of 80*80 size. The three feature maps are input into the Neck network in the Yolov5 model for feature fusion, and then the fused feature maps are input into the Head network to predict a series of bounding box positions and object categories. Among them, the NMS (Non-Maximum Suppression) algorithm is used to screen out the most representative bounding boxes. NMS will select the best bounding box based on the confidence score and overlap. In this part of Yolov5 model reasoning, we can get the center coordinates of the vehicle detection frame, the positive and negative directions of the vehicle, the point set of the lane line (including the center solid line, the center double solid line, the solid lines of the lanes on both sides, and the lane dividing line), whether the vehicle has a license plate, and other information. Since most lanes are straight lines, and in order to reduce the computational burden, the lane line is set as a straight line, and the coordinate system is the graph coordinate system. Here, the point set of the lane line is fitted into a straight line using the least squares method, and the slope of the lane line can be obtained, as well as a coordinate that is both in the lane line point set and on the lane line. This coordinate is called the lane line coordinate.

[0075] Next, the vehicle detection frames of multiple consecutive frames detected by Yolov5 and the feature vectors corresponding to each detection frame are used as input, and the DeepSort algorithm is used to associate and track the target. The DeepSort algorithm uses the Kalman filter to estimate and predict the state of the target, and uses the Hungarian algorithm to associate the target with the trajectory. By calculating the similarity between the feature vectors and combining the prediction results of the Kalman filter, DeepSort can establish a coordinate trajectory for each vehicle target and update the state of the trajectory. After that, the tracking ID and associated trajectory information of each target can be output. The ID of the vehicle and the coordinates of the vehicle at each time point in the continuous frame can be obtained.

[0076] The continuous image frames are first entered into the Yolov5 model for detection. After preprocessing, the images need to enter the Backbone network for feature extraction, and finally obtain the feature map M1 of 20*20 size, the feature map M2 of 40*40 size, and the feature map M3 of 80*80 size. The three feature maps are input into the Neck network for feature fusion, and the three feature maps obtained are input into the Head network to predict a series of bounding box positions and object categories. The NMS (Non-Maximum Suppression) algorithm is used to screen out the most representative bounding box. NMS will select the best bounding box based on the confidence score and overlap. In this part of Yolov5 model reasoning, we can get the center coordinates of the vehicle detection frame, the positive and negative directions of the vehicle, the point set of the lane line, the different types of lane lines (including the center solid line, the center double solid line, the solid lines of the lanes on both sides, and the lane dividing line), whether the vehicle has a license plate, and other information. Since most lanes are straight lines, and in order to reduce the computational burden, the lane line is set as a straight line and the coordinate system is the graph coordinate system. Here, the point set of the lane line is fitted into a straight line using the least squares method, and the slope of the lane line can be obtained, as well as a coordinate that is both in the lane line point set and on the lane line. This coordinate is called the lane line coordinate.

[0077] The vehicle detection frame of the continuous frames detected by Yolov5 and the feature vector of this detection frame are used as input, and the DeepSort algorithm is used to associate and track the target. DeepSort uses the Kalman filter to estimate and predict the state of the target, and uses the Hungarian algorithm to associate the target with the trajectory. By calculating the similarity between the feature vectors and combining the prediction results of the Kalman filter, DeepSort can establish a coordinate trajectory for each vehicle target and update the state of the trajectory. After that, the tracking ID and associated trajectory information of each target can be output. The ID of the vehicle and the coordinates of the vehicle at each time point in the continuous frame can be obtained.

[0078] After obtaining the trained detection model, the smart camera pole inputs the collected images into the detection model that has used the best weights, and obtains the target detection and target tracking results, including the center coordinates of the vehicle detection frame, the positive and negative directions of the vehicle, the vehicle ID at each time point in the continuous frame, the center coordinates of the vehicle detection frame, and whether the vehicle contains license plate information. The slope of the lane line (including the center solid line, the center double solid line, the solid lines of the lanes on both sides, and the lane dividing line) is obtained from the image of the first frame, as well as a coordinate that is both in the lane line point set and on the lane line, which is called the lane line coordinate.

[0079] After obtaining the detection results, it is necessary to judge whether the vehicle has illegal lane change behavior based on the obtained detection results, including the vehicle pressing the center solid line, pressing the center double solid line, driving in the wrong direction, speeding, and illegal overtaking. After determining that the vehicle has illegal behavior, the system will output the vehicle's illegal behavior category, the detection frame coordinates of the illegal vehicle, and the length and width of the detection frame, thereby cutting out the illegal vehicle image.

[0080] The complete process of the system is as follows: First, the intelligent camera pole acquires continuous image frames and inputs them into the model. Through different detection heads of the Yolov5 model and the DeepSORT model, the center coordinates of the vehicle detection frame, the positive and negative directions of the vehicle, the point set of lane lines (including the center solid line, the center double solid line, the solid lines of the lanes on both sides, and the lane dividing line), whether the vehicle has a license plate, and the vehicle ID at each time point in the continuous frame are extracted. This information is input into the illegal overtaking detection algorithm, the compaction line and double solid line detection algorithm, the speeding detection algorithm, and the reverse detection algorithm according to the needs to obtain the relevant information of the illegal vehicle, including the image frame where the illegal vehicle appears, the detection frame coordinates and its length and width.

[0081] Then, these image frames are input into the Yolov5 model to determine whether the smart camera pole has captured the license plate of the illegal vehicle. If the license plate is captured, the license plate part is cut out and the license plate number is extracted using OpenCV on the smart camera pole. If the license plate is not captured, the smart camera pole transmits the detection image information frame of the vehicle and the Yolov5+DeepSORT annotation information to other nearby vehicles through V2I, notifying them to re-identify the vehicle. Nearby vehicles use the Yolov5+DeepSORT vehicle re-identification algorithm to find the illegal vehicle, and use Yolov5 to determine whether it contains a license plate, and transmit the image of the illegal vehicle with the license plate to the smart camera pole through V2I. The smart camera pole inputs the image into the Yolov5 model, locates the license plate part, and uses OpenCV to recognize the license plate number. After the recognition is completed, the illegal vehicle data is uploaded to the cloud service center for processing and punishment. In the following algorithm, the coordinate system is the graph coordinate system, and the lane lines are all straight lines.

[0082] Based on the obtained vehicle detection frame coordinates and the coordinates and slopes of the lane lines (center solid line, center double solid line), it is determined whether the vehicle is crossing the center solid line or the center double solid line based on this information.

[0083] Use the following algorithm to determine whether a vehicle is crossing the solid center line or the double solid center line:

[0084] Assume that the center coordinates of the vehicle detection frame detected by the target are (VX1, VY1), (VX2, VY2),…, (VX n ,VY n), the coordinates and slopes of the central solid line and the central double solid line are (LX1,LY1),…,(LX m ,LY m ) and K1, K2…, K o ;

[0085] According to the coordinates of the center solid line and the center double solid line (LX i ,LY i ) and slope K i , we can get the equation of the lane solid line:

[0086] y=K i *(x-LX i )+LY i

[0087] For each vehicle’s detection box center coordinate (VX i ,VY i );

[0088] For each detected vehicle and the center solid line or center double solid line, correspond to each other and calculate the shortest distance from the center point of the vehicle to the solid line:

[0089] First, the coordinates of the vehicle center point (VX i ,VY i ) into the real line equation to obtain the projection coordinates of the point on the real line (PX i ,PY i ):

[0090] PX i =(K i *(K i *VX i -K i *LX i +VY i -LY i )) / (1+K i 2 )

[0091] PY i =K i *(PX i -LX i )+LY i

[0092] Then, calculate the distance d from the center point of the vehicle to the solid line:

[0093] d = Sqrt((PX i -VX i ) 2 +(PY i -VY i ) 2 )

[0094] The threshold Threshold is set to 0.7 meters. If the shortest distance d is less than the set threshold, it is determined that the vehicle has crossed the solid line.

[0095] Based on the vehicle direction and vehicle coordinates obtained, as well as the coordinates and slope of the center solid line, the lane the vehicle is in is calculated to determine whether the vehicle is in a reverse state. The following algorithm is used for judgment:

[0096] Assume that the center coordinates of the vehicle detection frame detected by the target are (VX1, VY1), (VX2, VY2),…, (VX n ,VY n ), the directions of the vehicles are VF1,...,VF n , the coordinates and slopes of the center solid line between the forward and reverse lanes are (LX c ,LY c ) and K c .

[0097] Assume that the vehicle travels in the direction VF1,...,VF n ∈[-1,1], that is, there are only two possible driving directions for the vehicle, and the oblique direction will also be judged as the approaching straight direction. The vehicles on both sides of the center solid line are driving in opposite directions. Assuming that the correct driving direction of the upper lane on the center solid line is 1, and the correct driving direction of the lower lane is -1, the following calculation can be used to determine whether the vehicle is going in the wrong direction:

[0098] For each vehicle coordinate (VX i ,VY i ) to do the following:

[0099] Calculate the value of the central solid line equation LValue i :

[0100] LValue i =(VY-LY c )-K c *(VX-LX c )

[0101] LValue i >0, then the vehicle (VX i ,VY i ) is above the solid centre line;

[0102] LValue i <0, then the vehicle (VX i ,VY i ) below the solid centre line;

[0103] LValue i =0, then the vehicle (VXi ,VY i ) on the solid center line.

[0104] If LValue i *VF i >0, it means the vehicle is not going against the flow; LValue i *VF i <0, indicating that the vehicle is traveling in the wrong direction.

[0105] Based on the ID and coordinates of the vehicle at each time point in the continuous frames obtained by target tracking, the following algorithm can be used to determine whether the vehicle is speeding:

[0106] First, a series of vehicle numbers are obtained by target tracking, namely V1, V2, ..., V n , let the center coordinates of the vehicle detection frame detected by the target be (VX1, VY1), (VX2, VY2),…, (VX n ,V Y n), vehicle V i The coordinates of the consecutive frames are expressed as Let the road speed limit be v max , the following formula can be used to calculate the vehicle speed:

[0107] Calculate vehicle V i The displacement between two frames:

[0108]

[0109] Vehicle V i The time interval between two frames is Δt, from which the vehicle V i The speed v i for:

[0110]

[0111] Finally, determine the vehicle V i The speed v i Is it greater than the speed limit v max If it is greater than , it is considered as speeding.

[0112] Based on the ID and coordinates of the vehicle at each time point in the continuous frames obtained through target tracking, as well as the coordinates and slopes of the lane lines (including the center solid line, the center double solid lines, the solid lines of the lanes on both sides, and the lane dividing line), the following algorithm can be used to determine whether it is an illegal overtaking:

[0113] First, a series of vehicle numbers are obtained by target tracking, namely V1, V2, ..., V n , let the center coordinates of the vehicle detection frame detected by the target be (VX1, VY1), (VX2, VY2),…, (VYn ,VY n ), the case of a two-lane road in the same direction is taken here just for the sake of understanding, and it does not mean that this technology cannot be applied to other types of lanes. The coordinates and slopes of the solid lines of the lanes on both sides are (LX1, LY1), (LX2, LY2) and K1, K2 respectively, and the coordinates and slopes of the lane dividing line in the middle of the two-lane road in the same direction are (LX3, LY3) and K3 respectively.

[0114] The following method is used to determine the number V i The lanes where the vehicles are located and the order of the vehicles:

[0115] First, calculate the distance from the vehicle to the solid line of the left lane, the solid line of the right lane, and the lane dividing line:

[0116] d1=|(VY i -LY1)-K1*(VX i -LX1)|

[0117] d2=|(VY i -LY2)-K2*(VX i -LX2)|

[0118] d3=|(VY i -LY3)-K3*(VX i -LX3)|

[0119] Then determine the lane the vehicle is in according to the following formula:

[0120]

[0121] At this time, according to the corresponding direction of the lane direction in the coordinate system, the front and rear sorting sequence of the vehicles in the left and right lanes can be obtained. A sorting sequence is saved in each frame, and a detection is performed every 300 frames to obtain the vehicle position information of the first 300 frames. In these 300 frames, all the vehicles that are in the left lane high-speed lane in a certain frame, then appear in the right lane in the subsequent frames, and then return to the left lane are found. Then, these vehicles are judged whether the distance between them and the rear vehicle after overtaking is less than the safe distance. If it is less than the safe distance, it is judged as overtaking.

[0122] When the smart camera device determines that a vehicle has violated traffic rules and the image taken by the smart camera pole fails to capture the license plate of the violating vehicle, the detection image information frame of the vehicle and the Yolov5+DeepSORT annotation information will be transmitted to other nearby vehicles through V2I, notifying other nearby vehicles to re-identify the vehicle. The nearby vehicles use the Yolov5+DeepSORT vehicle re-identification algorithm to find the violating vehicle and transmit the image of the violating vehicle with the license plate to the smart camera pole through V2I. The smart camera pole inputs the image into the Yolov5 model, finds the license plate part, and uses OpenCV to recognize the license plate number. After recognition, the violating vehicle data is uploaded to the cloud service center for punishment.

[0123] The vehicle and the roadside smart camera pole communicate through the PC5 protocol with the help of the OBU on the vehicle and the RSU on the smart camera pole.

[0124] The smart camera pole obtains the vehicle violation category, the detection frame coordinates, and the length and width of the detection frame obtained in step S4, and then transmits the image frame of the illegal vehicle and the Yolov5+DeepSort annotation information to other nearby vehicles through V2I, notifying the nearby vehicles to execute the Yolov5+DeepSORT algorithm for vehicle re-identification to find the illegal vehicle.

[0125] After the nearby vehicles find the image of the illegal vehicle through the Yolov5+DeepSORT re-identification model, the Yolov5 model is used to detect whether the image contains a license plate. If not, the Yolov5+DeepSORT re-identification model will continue to be executed until the image of the illegal vehicle with a license plate is detected or the smart camera pole notifies that the image of the illegal vehicle with a license plate has been obtained.

[0126] On the OBU, license plate number recognition is achieved through image processing using Python+OpenCV. After the image of the license plate position detected by Yolov5 is cut out, the image is thresholded, edge detected, and morphologically operated. The specific position of the license plate is identified based on the obtained contour features, and the license plate is segmented. Then, the image is subjected to image denoising grayscale and thresholding to segment the license plate characters. Finally, template matching is used to recognize and output the characters.

[0127] The above-described embodiment is only a preferred solution of the present invention, but it is not intended to limit the present invention. A person skilled in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. A method for detecting illegal lane changes based on vehicle-road collaboration, characterized in that: The following steps are involved: S1 builds a neural network model for detecting violations according to the following strategy: The model input is a continuous multi-frame image, including two-stage reasoning. In the first stage, the model detects targets in each frame of the image, including vehicles, road center lines and vehicle license plates. and obtaining information related to the detection of traffic violations, the information related to the detection of traffic violations including the coordinates of the center of the vehicle detection frame, the positive and negative directions of the vehicle, the point set of the lane line, the type of the lane line, and whether the vehicle has a license plate; In the second stage, the model takes the vehicle detection frame of the continuous frames detected in the first stage and the feature vector of the detection frame as input, associates and tracks the vehicle targets, and outputs the tracking ID and associated trajectory information of each vehicle target; S2 uses the image data collected by the road test to initialize the training of the neural network model used to detect violations; S3 sends the image data to be detected collected by the road test to the trained neural network model, obtains the detection violation related information in the continuous multi-frame image, and the trajectory information of each vehicle, and judges whether the vehicle has violated the traffic rules based on the information, including the vehicle crossing the center solid line or the center double solid line, driving in the wrong direction, speeding, and illegal overtaking; S4: When a vehicle is determined to be in violation of traffic rules and has a license plate, the license plate number is recognized from the license plate image of the detected vehicle in violation of traffic rules; When a vehicle is judged to be in violation of traffic rules and does not have a license plate, the detection image frame of the vehicle and the annotation information given by the neural network model are transmitted to other nearby vehicles, and other nearby vehicles are notified to re-identify the vehicle; nearby vehicles use the vehicle re-identification algorithm to find the illegal vehicle, perform target detection, detect the license plate image of the illegal vehicle and send it back, and then identify the license plate number on the returned license plate image; After identification, the data of the illegal vehicles will be uploaded to the cloud.

2. The method according to claim 1, characterized in that The types of lane lines include a solid center line, double solid center lines, solid lane lines on both sides, and a lane dividing line.

3. The method according to claim 1, characterized in that: In step 3, the lane line is set as a straight line, and the coordinate system is the image coordinate system. The point set of the lane line is fitted into a straight line using the least squares method. The slope of the lane line and a coordinate that is both in the lane line point set and on the lane line can be obtained, which is called the lane line coordinate. The following steps are included to determine whether the vehicle has violated the traffic rules: Step 31 uses the following algorithm to determine whether a vehicle is pressing the center solid line or the center double solid line: Assume that the center coordinates of the vehicle detection frame detected by the target are (VX1, VY1), (VX2, VY2),…, (VX n ,VY n ), the coordinates and slopes of the central solid line and the central double solid line are (LX1,LY1),…,(LX m ,LY m ) and K1, K2…, K o ; According to the coordinates of the center solid line and the center double solid line (LX i ,LY i ) and slope K i , we can get the equation of the lane solid line: y=K i *(x-LX i )+LY i The center coordinates of the detection box of each vehicle (VX i ,VY i ) and the center solid line or the center double solid line, correspond to each other, and calculate the shortest distance from the center point of the vehicle to the solid line: First, the coordinates of the vehicle center point (VX i , VY i ) into the real line equation to obtain the projection coordinates of the point on the real line (PX i , PY i ): PX i =(K i *(K i *VX i -K i *LX i +VY i -LY i )) / (1+K i 2 ) PY i =K i *(PX i -LX i )+LY i Then, calculate the distance d from the center point of the vehicle to the solid line: d=sqrt((PX i -VX i ) 2 +(PY i -VY i ) 2 ), If the shortest distance d is less than the set threshold, it is determined that the vehicle has crossed the solid line; Step 32 uses the following algorithm to determine whether the vehicle is in a reverse state: Assume that the center coordinates of the vehicle detection frame detected by the target are (VX1, VY1), (VX2, VY2), ..., (VX n , VY n ), the directions of the vehicles are VF1, ..., VF n , the coordinates and slopes of the center solid line between the forward and reverse lanes are (LX c , L.Y. c ) and K c , vehicle travel direction VF1, ..., VF n ∈[-1,1], that is, there are only two possible driving directions of the vehicle. The oblique direction is also judged as the approaching straight direction. The vehicles on both sides of the center solid line are driving in opposite directions. Assuming that the correct driving direction of the upper lane on the center solid line is 1, and the correct driving direction of the lower lane is -1, the following calculation can be used to determine whether the vehicle is driving in the wrong direction: For each vehicle coordinate (VX i , VY i ), calculate the value of the central solid line equation: LValue i =(VY-LY c )-K c *(VX-LX c ), If LValue i >0, then the vehicle (VX i , VY i ) is on the upper side of the center solid line, LValue i <0, then the vehicle (VX i , VY i ) is below the center solid line, LValue i =0, then the vehicle (VX i , VY i ) on the solid centre line; If LValue i *VF i >0, it means the vehicle is not going against the flow, LValue i *VF i <0, indicating that the vehicle is traveling in the wrong direction; Step 32 uses the following algorithm to determine whether the vehicle is speeding: First, a series of vehicle numbers are obtained by target tracking, namely V1, V2, ..., V n , let the center coordinates of the vehicle detection frame detected by the target be (VX1, VY1), (VX2, VY2), ..., (VX n , VY n ), vehicle V i The coordinates of the consecutive frames are expressed as Let the road speed limit be v max , the following formula can be used to calculate the vehicle speed: Calculate vehicle V i The displacement between two frames: Vehicle V i The time interval between two frames is Δt, from which the vehicle V i The speed v i for: Finally, determine the vehicle V i The speed v i Is it greater than the speed limit v max , if it is greater than, it is considered as speeding; Step 34 uses the following algorithm to determine whether the vehicle is overtaking illegally: First, a series of vehicle numbers are obtained by target tracking, namely V1, V2, ..., V n , let the center coordinates of the vehicle detection frame detected by the target be (VX1, VY1), (VX2, VY2), ..., (VX n , VY n ), the case of a two-lane road in the same direction is taken here just for the sake of understanding, and it does not mean that this technology cannot be applied to other types of lanes. The coordinates and slopes of the solid lines of the lanes on both sides are (LX1, LY1), (LX2, LY2) and K1, K2 respectively, and the coordinates and slopes of the lane dividing line in the middle of the two-lane road in the same direction are (LX3, LY3) and K3 respectively; The following method is used to determine the number V i The lanes where the vehicles are located and the order of the vehicles: First, calculate the distance from the vehicle to the solid line of the left lane, the solid line of the right lane, and the lane dividing line: d1=|(Y i -LY1)-K1*(VX i -LX1)| <h2 style=";text-align:left;direction:ltr">d2=|(VY<h2 style=";text-align:left;direction:ltr"> i <h2 style=";text-align:left;direction:ltr"> -LY2)-K2*(VX<h2 style=";text-align:left;direction:ltr"> i <h2 style=";text-align:left;direction:ltr"> -LX2)| d3=|(VY i -LY3)-K3*(VX i -LX3)| Then determine the lane the vehicle is in according to the following formula: At this time, according to the corresponding direction of the lane direction in the coordinate system, the front and rear sorting sequence of vehicles in the left and right lanes can be obtained. A sorting sequence is saved in each frame, and a detection is performed every certain number of frames to obtain the vehicle position information in these image frames. Find out all the vehicles in these image frames that are in the left lane high-speed lane in a certain frame, and then appear in the right lane in subsequent frames and then return to the left lane. Then, judge whether the distance between these vehicles and the rear vehicle after overtaking is less than the safe distance. If it is less than the safe distance, it is judged as overtaking.

4. The method according to claim 1, characterized in that: The Yolov5 model is used in the first stage of reasoning.

5. The method according to claim 4, characterized in that The DeepSORT model is used in the second stage of reasoning.

6. The method according to claim 1, characterized in that In step S4, the transmission of the image frame and annotation information of the illegal vehicle and the transmission of the license plate partial image to other nearby vehicles all adopt the PC5 protocol.

7. The method according to claim 1, characterized in that In step S4, Yolov5+DeepSORT algorithm is used to re-identify other nearby vehicles to find illegal vehicles.

8. A system for detecting illegal lane changes based on vehicle-road collaboration, characterized in that: It comprises a roadside intelligent camera pole, on which a camera unit, an RSU and a roadside communication unit which are electrically connected in sequence are arranged, and the RSU can perform V2I communication with an OBU on a vehicle through the roadside communication unit; the system detects an illegal lane change of a vehicle by executing any of the method steps described in claims 1-7.

9. The system according to claim 8, characterized in that The camera unit is a gun-type camera.

Images