Method for detecting illegal entry of expressway truck into passenger car lane based on roadside camera

Abstract

The invention discloses a method for detecting illegal entry of a highway truck into a passenger car lane based on a roadside camera, and mainly solves the problem that illegal vehicles cannot be efficiently and accurately detected and recorded due to dependence on manual judgment in the prior art. According to the implementation scheme, an expressway image is shot through a roadside camera, and a truck forbidding lane area is calibrated in the image; training a highway vehicle identification model, and detecting images collected by a camera by using the trained vehicle identification model to obtain a vehicle detection frame and a vehicle category; training a license plate detection model, and using the trained license plate detection model to detect images collected by a camera to obtain a license plate detection frame; judging whether the truck runs in the forbidden area or not by using the position information of the vehicle detection frame and the license plate boundary frame; and cutting out a license plate bounding box area of a truck which is detected to illegally drive in the forbidden area, inputting the license plate bounding box area into a license plate recognition model CRNN to recognize a license plate number, and recording the license plate number. The method is high in detection precision and can be used for safety management of the expressway.

Description

Technical Field

[0001] The present invention belongs to the technical field of image target detection, and in particular relates to a method for detecting trucks illegally entering a passenger lane, which can be used for traffic management on highways. Background Art

[0002] With the rapid development of highway traffic, the mixing of trucks and buses is increasing. Especially during peak hours, trucks are often seen illegally entering passenger lanes. This not only disrupts normal traffic flow but also increases the risk of accidents. Therefore, during highway traffic monitoring, effective detection of trucks illegally entering passenger lanes is essential. Existing detection methods mostly rely on manual monitoring to identify truck violations. Alternatively, methods exist that use fixed cameras to capture single-frame images to detect truck positions and directly determine violations.

[0003] Patent application number CN201820792865.X discloses a "mobile early warning inspection system." This system uses a mobile inspection device to capture images of trucks illegally entering passenger lanes. While the system can record trucks illegally entering passenger lanes, it still requires manual intervention to determine vehicle violations and lacks real-time detection, which can easily lead to missed detections and low detection efficiency.

[0004] The patent document with patent application number CN202011086337.0 discloses a "method and system for detecting illegal road occupation by vehicles". It mainly judges illegal road occupation by detecting whether the center point of the vehicle detection frame is within the specified area. Although the system can effectively judge whether the vehicle is in the corresponding lane when the monitoring angle is perpendicular to the road surface, when the monitoring angle is tilted or the vehicle detection frame is large, the center point of the detection frame may deviate from the lane where the vehicle is actually located, resulting in a deviation in the judgment of whether the vehicle is in the designated lane area. In addition, it is difficult for the system to accurately judge whether the vehicle is occupying the road in complex highway scenarios, and it is prone to misjudgment, resulting in a high detection error rate. Summary of the Invention

[0005] The purpose of the present invention is to address the deficiencies of the above-mentioned existing technologies and propose a method for detecting trucks illegally entering passenger lanes on highways based on roadside cameras, so as to improve the detection efficiency of illegal trucks entering passenger lanes and reduce the false detection rate of non-illegal vehicles in complex highway scenarios.

[0006] The technical approach to achieving the purpose of the present invention is: to use deep learning technology to perform real-time detection on monitoring images, replacing traditional manual judgment methods, and improving the efficiency of detecting illegal vehicles; by using the positional relationship between the license plate detection frame and the vehicle detection frame obtained by detection, the actual lane in which the vehicle is located is determined, thereby achieving accurate judgment of vehicle violations and reducing the false detection rate in complex scenarios.

[0007] According to the above ideas, the technical solution of the present invention includes the following steps:

[0008] This method first sets up a roadside camera, specifies a truck-restricted zone in the surveillance footage, then detects the vehicle and license plate. Finally, it determines whether a violation has occurred based on the relative positions of the vehicle and license plate detection frames and the truck-restricted zone. This method is applicable not only to front-facing cameras but also to side-facing cameras to detect trucks illegally occupying the road.

[0009] To achieve the above objectives, the present invention provides a method for detecting trucks illegally entering passenger lanes on highways based on roadside cameras, which is characterized by comprising the following steps:

[0010] (1) A surveillance camera is set up on the roadside. In the captured highway image, a trapezoidal lane area is formed with lane lines as the boundary. The lane area where trucks are prohibited from entering is marked. The four corner points P1, P2, P3, and P4 of the marked area D are recorded in a clockwise direction. The pixel coordinates corresponding to the four corner points are (x1, y1), (x2, y2), (x3, y3), and (x4, y4), respectively.

[0011] (2) The vehicle images collected in advance from the highway monitoring screen are annotated to form a dataset, which is then input into the existing YOLOv5 network and trained using the cosine function descent method to obtain a trained highway vehicle recognition model;

[0012] (3) Input the current highway image captured by the camera into the trained highway vehicle recognition model to detect two types of vehicles: trucks and non-trucks. The identified vehicles are framed with a rectangular recognition frame, and the pixel coordinates of the bottom corner points of the detection frame and the vehicle category are recorded;

[0013] (4) The license plate images of various vehicles collected in advance are annotated to form a data set, and these data sets are used to train the license plate detection model of the DBNet network;

[0014] (5) Input the highway image into the trained license plate detection model to obtain the license plate type and the coordinates of the detection box corners, including the upper left corner and the lower right corner;

[0015] (6) For trucks, the truck detection frame position and license plate bounding frame position information are used to determine whether the truck is driving in a restricted area:

[0016] (6.1) Obtain the center point K(x,y) from the coordinates of the corner points of the license plate detection frame and project it vertically downward until it aligns with the lower edge of the truck detection frame along the vertical coordinate y b The coordinates of the vertical point L(x,y) obtained by intersection are (K(x),y b );

[0017] (6.2) Shoot a ray horizontally to the right from the vertical landing point L(x,y). Count the number of times the ray intersects the edges of the polygon enclosed by the marked area D, and determine the parity:

[0018] If the number is an odd number, it means that L(x,y) is inside the marked area D, then the truck is judged to be driving in the prohibited area and step (7) is executed;

[0019] Otherwise, it is determined that the truck is not traveling in the prohibited area;

[0020] (7) The license plate boundary box area of ​​the truck detected to be illegally driving in the prohibited area is cropped out and input into the existing license plate recognition model CRNN to identify the license plate number and record it.

[0021] Compared with the prior art, the present invention has the following advantages:

[0022] The present invention determines whether a vehicle is in a restricted area by combining the relative positions of a vehicle detection frame and a license plate detection frame, and determines whether a truck has illegally entered the passenger lane by the type of vehicle and license plate, thereby improving the accuracy of the determination. The invention is not only suitable for monitoring trucks illegally occupying the road with front-facing cameras, but also suitable for monitoring trucks illegally occupying the road with side-facing cameras. Compared with the existing technology, the invention has good adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is an implementation flow chart of the present invention;

[0024] Figure 2 This is a test result diagram of the original image using the vehicle recognition model of the present invention;

[0025] Figure 3 This is a test result diagram of the original image using the license plate recognition model of the present invention;

[0026] Figure 4 This is a result diagram of a simulation of violation detection on an original image captured by a camera using the invented method. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0028] It should be noted that the step numbers in the specification and claims of the present invention are only for the purpose of clearly describing the embodiments of the present invention and facilitating understanding, and the order of the step numbers is not limited.

[0029] Reference Figure 1 , the implementation steps of this example are as follows:

[0030] Step 1: Mark the pixel range of the forbidden area.

[0031] 1.1) Capture highway road images with a camera;

[0032] 1.2) Using the lane markings as boundaries, demarcate the lane area into a trapezoidal shape, marking the lane sections where trucks are prohibited from entering. Record the four corner points P1, P2, P3, and P4 of the marked area D in a clockwise direction. The corresponding pixel coordinates of these four corner points are (x1, y1), (x2, y2), (x3, y3), and (x4, y4), respectively.

[0033] Step 2: Train the highway vehicle recognition model.

[0034] 2.1) Collect vehicle images in a variety of environments, including buses and trucks, covering different body sizes, shooting angles, and various lighting conditions;

[0035] 2.2) Annotate the collected vehicle image dataset and generate annotation files in YOLO format, including the location and category information of the target objects in each image. Then perform normalization, data augmentation, and dataset segmentation preprocessing on the annotated image data.

[0036] 2.3) Input the preprocessed image dataset and the corresponding annotation file into the YOLOV5 network, set the network initial parameters, including the training cycle epochs, the initial learning rate initial_lr, the upper limit max_lr and the lower limit min_lr of the learning rate, and set min_lr to initial_lr;

[0037] 2.4) Use the cosine annealing learning rate adjustment method to dynamically adjust the learning rate lr in each training cycle:

[0038]

[0039] 2.5) At the beginning of each training cycle, based on the current training cycle epoch, epoch = 1, 2, ..., epochs and the total training cycle epochs, use the above formula to calculate the learning rate lr and update the model parameters;

[0040] 2.6) In each training cycle, use the currently calculated learning rate lr for forward and backward propagation of the model; every 5 training cycles, halve the learning rate lower limit min_lr and recalculate the learning rate for the current training cycle;

[0041] 2.7) Repeat 2.3) to 2.6) until Get the trained model.

[0042] Step 3: Collect license plate image data and train the license plate detection model DBNet.

[0043] 3.1) Collect license plate images from multiple sources and environments, including license plates of passenger cars and trucks, with images of different license plate sizes and under different lighting conditions;

[0044] 3.2) Preprocess the collected license plate images of buses and trucks and their annotated data by adjusting image size, normalizing pixel values, and performing data augmentation. The preprocessed image data is converted into the format required for DBNet model training. The license plate image data of buses and trucks is then divided into a training set and a validation set in an 8:2 ratio.

[0045] 3.3) Input the training set into the DBNet network, set the initial learning rate, batch size, and optimizer parameters, and start model training;

[0046] 3.4) In each training cycle, the loss function L is calculated using forward propagation, and the model parameters are updated through back propagation. The loss function is composed of the classification loss L cls and regression loss L reg The specific calculation formula is as follows:

[0047] L=λ1L cls +λ2L reg

[0048] Among them, L cls is the classification loss, L reg is the regression loss, λ1 and λ2 are the weights of classification loss and regression loss, respectively, which are adjusted according to the classification and positioning performance of the validation set;

[0049] 3.5) After each training cycle, the validation set is used to evaluate the performance of the model. The performance indicators include accuracy, recall, and F1 score, which are calculated as follows:

[0050]

[0051]

[0052]

[0053] 3.6) Based on the evaluation results of the validation set, adjust the model parameters to optimize the model performance;

[0054] 3.7) Repeat steps 3.3) to 3.6) until the accuracy of the validation set exceeds 0.98 to obtain the final trained model.

[0055] Step 4: Determine the trucks that are prohibited from entering the prohibited area based on the positional relationship between the license plate detection frame and the vehicle detection frame.

[0056] 4.1) Input the current highway image captured by the camera into the trained highway vehicle recognition model to detect vehicles, identifying two vehicle types: passenger cars and trucks, and marking the identified trucks with rectangular boxes;

[0057] 4.2) Input the current highway image captured by the camera into the trained license plate detection model to perform license plate detection, obtain the license plate category and license plate detection frame, and extract the corner coordinates of the license plate detection frame from the license plate detection frame;

[0058] 4.3) Obtain the center point K(x, y) of the license plate detection frame using the coordinates of the corner points, and calculate the vertical point L(x, y) where its projection onto the lower edge of the truck detection frame falls:

[0059]

[0060] L(x, y) = (K(x, y) b )

[0061] Among them, (x min ,y min )、(x max ,y max ) are the coordinates of the upper left corner and the lower right corner of the license plate detection frame, K(x) is the horizontal coordinate of K(x, y), and y b is the vertical coordinate of the lower edge of the truck detection box;

[0062] 4.4) Shoot a horizontal ray yL from point L(x, y) to the right, and represent any side of the polygon enclosed by region D as two vertices (xa ,y a ) and (x b ,y b ) and calculate the intersection point P of the horizontal ray with the edge k (x k ,y k )'s coordinates:

[0063]

[0064] y k =y L ;

[0065] 4.5) According to the horizontal coordinate x of the intersection point k The relationship between the horizontal coordinate L(x) of the point L(x, y) determines the number of intersections between the horizontal ray and the edge:

[0066] If x k ≤L(x), it means that the edge has no intersection with the horizontal ray;

[0067] If x k > L(x), it means that the edge has an intersection with the horizontal ray, and go to step 4.6);

[0068] If y a =y b , it means that the edge is a horizontal line segment and will not intersect with the horizontal ray;

[0069] 4.6) Count the number of times the ray intersects all edges of region D and determine whether they are even or parity:

[0070] If the number is an odd number, it means that L(x, y) is inside the marked area D, then the truck is judged to be driving in the prohibited area and step 5 is executed;

[0071] Otherwise, it is determined that the truck is not traveling in the prohibited area.

[0072] Step 5: Input the license plate image of the illegal vehicle into the existing CRNN model to record the license plate text.

[0073] The CRNN model includes a convolutional layer for extracting image features, a recursive layer for capturing the temporal dependencies of feature sequences, and a connection layer for connecting output features. The steps of using this model to record illegal license plates include the following:

[0074] 5.1) Crop the license plate bounding box area of ​​the detected illegal vehicle;

[0075] 5.2) The CRNN model first processes the image through the convolutional layer to extract its features. Then, the recursive layer identifies the temporal information and associations in the features, converts the features into character sequences, and finally generates the text recognition results of the license plate through the connection layer.

[0076] 5.3) Perform post-processing on the output recognition results to correct common recognition errors and remove illegal characters, and format the processed license plate text into a standardized output form;

[0077] 5.4) Record the recognized license plate text content.

[0078] The effects of the present invention can be further illustrated by the following test results:

[0079] 1. Test conditions:

[0080] The hardware platform for the test experiment of the present invention is: the processor is Intel i7-10870H CPU, the main frequency is 2.21GHz, the graphics card is RTX2080Ti, and the memory is 16GB.

[0081] The software platforms for the test experiments of the present invention are: Windows 10 operating system and Pycharm 2023.

[0082] 2. Test content and results:

[0083] Simulation 1, using the recognition model of the present invention to test the original image, obtain the vehicle's recognition frame, category and confidence level, as shown in Figure 2 shown.

[0084] Simulation 2, using the recognition model of the present invention to test the original image, obtain the recognition frame, category and confidence of the license plate, such as Figure 3 shown.

[0085] from Figure 2 and Figure 3 It can be seen that the present invention can achieve accurate detection and recognition of both vehicles and license plates.

[0086] Simulation 3, the method of the present invention is used to test the original image taken by the camera, and the results are as follows Figure 4 shown.

[0087] from Figure 4 It can be seen that both trucks and buses are accurately detected, and it can be accurately determined whether the truck is driving in the prohibited area, indicating that the present invention can accurately determine whether the truck has illegally entered the passenger lane within the allowable error.

Claims

1. A method for detecting a truck illegally entering a passenger lane on a highway based on a roadside camera, characterized in that: The steps include: (1) A surveillance camera is set up on the roadside. In the highway image captured, a trapezoidal lane area is formed with lane lines as boundaries. The lane area where trucks are prohibited from entering is marked. The four corner points P1, P2, P3, and P4 of the marked area D are recorded in a clockwise direction. The pixel coordinates corresponding to the four corner points are (x1, y1), (x2, y2), (x3, y3), and (x4, y4), respectively. (2) The vehicle images collected in advance from the highway monitoring screen are annotated to form a data set, which is input into the existing YOLOv5 network and trained using the cosine function descent method to obtain a trained highway vehicle recognition model; (3) Input the current highway image captured by the camera into the trained highway vehicle recognition model to detect two types of vehicles: trucks and non-trucks. Frame the identified vehicles with a rectangular recognition frame, and record the pixel coordinates of the bottom corner points of the detection frame and the vehicle category. (4) The license plate images of various vehicles collected in advance are annotated to form a data set, and these data sets are used to train the license plate detection model of the DBNet network; (5) Input the highway image into the trained license plate detection model to obtain the type of license plate and the coordinates of the corner points of the detection frame, including the upper left corner and the lower right corner; (6) For trucks, the truck detection frame position and license plate boundary frame position information are used to determine whether the truck is driving in the prohibited area: (6.1) Obtain the center point K(x, y) through the coordinates of the corner points of the license plate detection frame, and project it downward in the vertical direction until it coincides with the lower edge of the truck detection frame along the vertical coordinate y b The coordinates of the vertical point L(x, y) obtained by intersection are (K(x), y b ); (6.2) Shoot a ray horizontally to the right from the vertical point L(x, y), calculate the number of times the ray intersects the edge of the polygon enclosed by the marked area D, and determine its parity: If the number is an odd number, it means that L(x, y) is inside the marked area D, then the truck is judged to be driving in the prohibited area and step (7) is executed; Otherwise, it is determined that the truck is not traveling in the prohibited area; (7) The bounding box area of ​​the license plate of the truck detected to be driving illegally in the prohibited area is cropped out and input into the existing license plate recognition model CRNN to recognize the license plate number and record it.

2. The method according to claim 1, characterized in that In step (2), the vehicle image dataset is input into the existing YOLOv5 network and trained using the cosine function descent method. The implementation steps include the following: (2a) Annotate the original vehicle image dataset and generate annotation files in YOLO format, including the location and category information of the target object in each image. Perform normalization, data enhancement and preprocessing of the segmented dataset on the annotated image data. (2b) Input the processed image dataset and the corresponding annotation file into the YOLOV5 network, set the initial parameters of the network, including the training cycle epochs, the initial learning rate initial_lr, the upper limit max_lr and the lower limit min_lr of the learning rate. At the beginning of training, set min_lr to initial_lr; (2c) Use the cosine annealing learning rate adjustment method to dynamically adjust the learning rate lr in each training cycle: (2d) At the beginning of each training cycle, according to the current training cycle epoch, epoch = 1, 2, ..., epochs, and the total training cycle epochs, the learning rate lr is calculated using the above formula to update the parameters of the model; (2e) In each training cycle, the currently calculated learning rate lr is used for forward propagation and backward propagation of the model; every 5 training cycles, the lower limit of the learning rate min_lr is halved and the learning rate of the current training cycle is recalculated; (2f) Repeat (2b) to (2e) until Get the trained model.

3. The method according to claim 1, characterized in that In step (4), the license plate image dataset is input into the DBNet model for training, which is implemented as follows: (4a) Collect license plate images from multiple sources and different environments, including two types of license plates: passenger cars and trucks, different license plate sizes, and license plate images under different lighting conditions; (4b) Convert the license plate images and annotation data of buses and trucks into the format required for DBNet model training, adjust the image size and normalize the image pixel value to form a license plate image dataset of trucks and buses, and divide it into a training set and a validation set in a ratio of 8:2; (4c) Input the processed training set into the DBNet network and set the initial learning rate, batch size and optimizer parameters; (4d) In each training cycle, the loss function L is calculated through forward propagation, and the model parameters are updated through back propagation. The loss function uses the classification loss L cls and regression loss L reg The weighted sum of is as follows: L=λ1L cls +λ2L reg Where L cls and L reg are classification loss and regression loss respectively, while λ1 and λ2 are the weights of these two loss terms, which are adjusted according to the license plate classification and regional positioning performance on the validation set; (4e) After each training cycle, the validation set is used to evaluate the detection performance of the model. The evaluation indicators include accuracy, recall and F1 score. The calculation formula is as follows: (4f) According to the evaluation results of the validation set, adjust the model parameters to optimize the model performance; (4g) Repeat (4c) to (4f) until the accuracy of the validation set satisfies accuracy>=0.98, and the trained model is obtained.

4. The method according to claim 1, characterized in that The center point K(x, y) and K(x, y) obtained in step (6.1) are projected to the vertical point L(x, y) at the lower edge of the vehicle detection frame, respectively, as follows: L(x,y)=(K(x),y b ) Among them, (x min ,y min )、(x max ,y max ) are the coordinates of the upper left corner and lower right corner of the license plate detection frame, K(x) is the horizontal coordinate of K(x,y), and y b It is the vertical coordinate of the lower edge of the vehicle detection box.

5. The method according to claim 1, characterized in that In step (6.2), the number of times the ray emitted from point L(x, y) intersects the edge of the polygon enclosed by the marked area D is calculated as follows: (6.2a) A horizontal ray y is emitted from point L(x,y) to the right. L , represent any edge of the polygon enclosed by area D as two vertices (x a ,y a ) and (x b ,y b ) and calculate the intersection point P of the horizontal ray with the edge. k (x k ,y k )’s coordinates: and k =and L ; (6.2b) According to the horizontal coordinate x of the intersection point k The relationship between the horizontal coordinate L(x) of the point L(x, y) determines the number of intersections between the horizontal ray and the edge: If x k ≤L(x), it means that the edge has no intersection with the horizontal ray; If x k >L(x), it means that the edge has an intersection with the horizontal ray, and execute step (6.2c); If y a =y b , it means that the edge is a horizontal line segment and will not intersect with the horizontal ray; (6.2c) The sum of the number of intersections between the ray and all the edges of region D is the number of intersections between the ray yL and the polygon enclosed by the marked region D.

6. The method according to claim 1, characterized in that In step (7), the cropped license plate image is input into the CRNN model for license plate text recognition. The implementation steps include the following: (7a) The CRNN model first processes the image through its convolutional layer to extract features, then identifies the temporal information and dependencies in the features through the recursive layer, serializes the features into a sequence of characters, and finally outputs the text content of the license plate, i.e., the license plate number, through the connection layer; (7b) The output license plate text is post-processed to remove illegal characters and correct common recognition errors, and the processed license plate number and category are formatted as standard output.

Images