Target detection and segmentation technology-based traffic violation identification method
Through the combination of target detection and segmentation technology, the accurate judgment of whether the vehicle is pressed on the diversion line and the identification of the license plate number is achieved, which solves the problems of high difficulty and low accuracy in the existing technology, and improves the automation of traffic management and Security.
Patent Information
- Application Number
- CN202410962232.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-18
- Publication Date
- 2024-11-08
AI Technical Summary
In the prior art, it is difficult to accurately identify whether the vehicle is pressing the diversion line in traffic violation identification, and the accuracy of license plate identification is low at high speed and at different angles, resulting in limited traffic management efficiency and safety.
Using a method based on object detection and segmentation technology, the diversion line area and vehicle position are identified through deep learning models, combined with the behavior analysis module to determine whether the vehicle is pressing the diversion line, and an alarm is triggered when a violation occurs, and a license plate recognition model is called to obtain the license plate number , realize automated traffic violation detection.
It improves the automation level of traffic violation detection, reduces interference from human factors, improves the efficiency and safety of traffic management, provides detailed violation data support, and adapts to different traffic scenarios and lighting conditions.
Smart Images

Figure CN118918508A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of computer image processing, and in particular to a method for identifying traffic violations based on target detection and segmentation technology. Background Art
[0002] With the acceleration of urbanization, urban traffic pressure continues to increase. Traffic violations such as vehicles running over guide lines and illegal parking not only affect traffic flow, but may also cause traffic accidents, posing a serious threat to public safety. Especially on highways and urban expressways, guide lines are important traffic markings, and their role is to guide vehicles to drive along the prescribed routes and reduce the occurrence of traffic accidents. Guide lines are a type of prohibited marking. Vehicles are not allowed to run over or over the lines, nor are they allowed to turn around or park in the guide line area at will. However, due to the negligence or violations of the driver, vehicles running over guide lines often occur, which brings challenges to traffic management.
[0003] Traditional traffic violation detection mainly relies on manual inspections or simple camera monitoring. These methods are inefficient and easily affected by human factors and environmental conditions, making it difficult to achieve comprehensive and efficient supervision. With the rapid development of artificial intelligence technology, especially breakthroughs in image recognition, object detection and segmentation technology in the field of computer vision, new solutions have been provided for the automatic and intelligent detection of traffic violations.
[0004] The application of deep learning technology in the field of image recognition has greatly improved the accuracy and efficiency of target detection and target segmentation. Target detection technology can identify multiple targets and their locations in an image, while target segmentation technology can further distinguish targets from backgrounds and achieve pixel-level positioning. The application of these technologies provides a technical basis for the development of an automated traffic violation detection system.
[0005] Object detection technology is a key area in computer vision, which aims to identify multiple target objects from images or videos and locate their positions. Early object detection mainly relied on manually designed features, but this method is limited by feature expression capabilities and computing resources. With the development of deep learning technology, object detection methods based on deep neural networks have become mainstream. These methods usually include region proposal networks (RPN) and subsequent classification and bounding box regression steps. Typical object detection models such as Faster R-CNN, YOLO (You Only LookOnce) and SSD (Single Shot MultiBox Detector) have achieved different degrees of balance between speed and accuracy.
[0006] In the traffic violation identification system, object detection technology is used to monitor road conditions in real time and identify and locate large vehicles on the road. Since a large amount of video stream data needs to be processed, the object detection model needs to have a high processing speed and low latency to meet the real-time requirements.
[0007] Object segmentation technology aims to assign each pixel in an image to a specific category, providing a more sophisticated understanding of image content than object detection. Unlike object detection, object segmentation requires not only identifying the object in the image, but also accurately locating the shape and outline of the object. Deep learning technology, especially convolutional neural networks (CNNs), has made significant progress in the field of object segmentation. Popular object segmentation models include fully convolutional networks (FCNs), U-Net, Mask R-CNN, etc. These models use encoder-decoder architectures, combined with upsampling and skip connections, to generate high-quality segmentation maps while preserving image details. The application of the YOLO series in the field of object segmentation usually combines object detection with pixel-level segmentation, such as models such as YOLACT (You Only Look Once for semantic / instance Segmentation). These models add a segmentation branch based on YOLO, which can simultaneously output the category and location of the object and the category label of each pixel.
[0008] License plate recognition technology involves locating the license plate area from the vehicle image and extracting the license plate number from it. This process usually includes three steps: license plate location, character segmentation and character recognition. License plate location aims to detect the location of the license plate from the vehicle image; character segmentation separates the characters in the license plate to prepare for the recognition of each character; the final character recognition uses optical character recognition (OCR) technology to convert the segmented characters into corresponding numbers or letters.
[0009] With the application of deep learning technology, the accuracy and robustness of license plate recognition have been significantly improved. The license plate recognition system based on CNN can handle the challenges of different lighting, different angles and different license plate styles.
[0010] The application of existing technologies in traffic violation identification is mainly concentrated in scenarios such as illegal parking, speeding, and traffic light violations. Object detection technology is used to identify illegally parked vehicles, object segmentation technology is used to identify traffic signs or signals from complex backgrounds, and license plate recognition technology is used to record information about illegal vehicles. However, there are still some challenges in applying these technologies to the identification of vehicles pressing guide lines. Guide lines are usually slender and may be difficult to identify under different road surfaces and lighting conditions; the intersection judgment of vehicles and guide lines requires precise spatial analysis; and license plate recognition needs to maintain high accuracy when the vehicle is driving at high speeds and at different angles.
[0011] Based on the above problems, it is necessary to provide an identification method for traffic violations based on target detection and segmentation technology, which can automatically identify and monitor whether the vehicle presses the guide line, and recognize the license plate of the violating vehicle, thereby improving the automation level of traffic violation detection, reducing the interference of human factors, and improving the efficiency and safety of traffic management. Summary of the invention
[0012] The purpose of the present invention is to provide a method for identifying traffic violations based on target detection and segmentation technology, which can automatically identify and monitor whether a vehicle presses a guide line, and perform license plate recognition on the violating vehicle, thereby improving the automation level of traffic violation detection, reducing the interference of human factors, and improving the efficiency and safety of traffic management; the research results are expected to be widely used in the fields of intelligent traffic monitoring, traffic violation analysis and processing, and have important social and economic value.
[0013] In order to solve the above technical problems, the technical solution adopted by the present invention is that the identification method of traffic violations based on target detection and segmentation technology specifically includes the following steps:
[0014] S1: The image acquisition device collects real-time video stream data to form a data set; trains the guide line segmentation model, vehicle detection model, and license plate recognition model;
[0015] S2: After initializing the real-time video stream data, the guide line area is obtained through the guide line segmentation model and the guide line area is saved; after initialization, the vehicle position is located according to the real-time video stream data vehicle detection model;
[0016] S3: Combine the guide line area and vehicle position to determine whether the vehicle crosses the guide line through the behavior analysis module. If not, return to step S2; if yes, use the license plate recognition model combined with the illegal vehicle image in the alarm generation module to identify the license plate number, and then send the result to the data management system to record the illegal vehicle information.
[0017] The above technical solution is adopted to identify vehicles and guide lines through target detection and target segmentation algorithms. Specifically, firstly, the guide line area is accurately located by the deep learning model through target segmentation technology, and then the target detection algorithm is used to monitor and identify the vehicle in real time, and then the intersection of the vehicle and the guide line is analyzed to determine whether the vehicle presses the guide line and trigger an alarm when a violation occurs. After the alarm is triggered, the vehicle image is automatically captured and the license plate recognition model is called to obtain the license plate number; this method can accurately determine whether the vehicle presses the guide line and accurately identify the license plate number, thereby improving the automation level of traffic violation detection, reducing interference from human factors, and improving the efficiency and safety of traffic management; at the same time, it effectively assists traffic supervision and management, and provides data support for subsequent traffic violation analysis and processing.
[0018] Preferably, the data set in step S1 includes a target segmentation guide line data set, a target detection vehicle data set and a license plate service data set; wherein the target segmentation guide line data set and the target detection vehicle data set are first annotated by the image data in the real-time video stream data, and then the annotation file is converted into a txt label file suitable for YOLO algorithm processing, and the txt label file includes a category label and a corresponding bounding box coordinate; the license plate service data set is first annotated by the image data in the real-time video stream data to obtain the coordinate position of the license plate and the label of the license plate number, and then the license plate image is cropped according to the license plate coordinates to form a format suitable for LPRNet character recognition and YOLO license plate detection model training.
[0019] Preferably, the specific steps of training the flow line segmentation model in step S1 are:
[0020] S1-11: Divide the dataset into training set, validation set and test set;
[0021] S1-12: The guide line images and label files in the training set are configured and input into the YOLOv8s-seg network. The guide line segmentation model gradually learns to distinguish guide lines from other road elements by learning the features in the image and the shape and texture information of the guide lines;
[0022] S1-13: Use the validation set to adjust the hyperparameters of the model training and use data augmentation technology to optimize the model; the hyperparameters include learning rate, batch size and number of iterations; train to obtain the flow line segmentation model;
[0023] S1-14: Use the test set to evaluate the flow line segmentation model through the F1-Confidence curve; that is, use the F1 score for evaluation. The F1 score calculation formula is:
[0024]
[0025] Among them, Precision represents accuracy, Recall represents recall rate; F1 score is the harmonic mean of precision and recall rate.
[0026] Preferably, the specific steps of training the vehicle detection model in step S1 are:
[0027] S1-21: Divide the dataset into training set, validation set and test set;
[0028] S1-22: Configure the images containing vehicles in the training set and their label files, input them into the yolov8 target detection network, and gradually extract the vehicle features in the image through a series of convolutional layers, pooling layers, activation functions, etc. After multiple rounds of iterative training, the weight file for vehicle detection is finally obtained;
[0029] S1-23: Use the transfer learning method to use the weight file obtained by YOLOv8s pre-training as the starting point, use the validation set to adjust the hyperparameters, and learn the specific features of the vehicle to obtain the vehicle detection model;
[0030] S1-24: The final performance of the vehicle detection model is obtained using the test set evaluation.
[0031] Preferably, the license plate recognition model in step S1 includes a license plate detection model and a character recognition model, the license plate detection model is responsible for locating the license plate area in the image, and the character recognition model is used to identify each character license plate detection on the license plate; the license plate detection model is trained using the yolov5s network, and the input sample is a cropped vehicle picture without other road background features. After model tuning and learning of the yolov5 network, a license plate detection model that can locate the license plate position is finally obtained.
[0032] Preferably, the character recognition model is trained using an end-to-end LPRNet network. The LPRNet network is trained using a convolutional neural network (CNN) combined with a connection temporal classification (CTC) method, including a backbone, a global context feature extraction module, and a recognition head. The backbone uses a classic CNN structure to extract local features of the license plate image. A 13x1 convolution module is designed at the end of the backbone to extract context information in the sequence direction. A fully connected layer is used outside the backbone to flatten the feature map output by the backbone into a one-dimensional vector, and extract global context features through nonlinear transformation. The features after feature fusion are input to the recognition head for final license plate character recognition. At the same time, LPRNet uses a CTC loss function during training, and the input sample is a cropped license plate image. After the LPRNet network structure and the CTC loss function, the character recognition model will repeatedly iterate the images and label files in the training data set, and then optimize the network parameters through the back propagation algorithm to finally obtain the optimal LPRNet license plate recognition model. The calculation formula for the test accuracy of the LPRNet license plate recognition model is:
[0033]
[0034] Among them, Tp is the number of samples truly recognized as license plates, that is, the samples correctly recognized by the model, Tn_1 and Tn_2 refer to the number of samples assumed to be non-license plates but are actually license plates, that is, the samples that the model mistakenly recognizes license plates as non-license plates.
[0035] Preferably, the specific steps of step S2 are:
[0036] S21: extracting the features of the guide line by using the guide line segmentation model to obtain the guide line area;
[0037] S211: within the first second of the real-time video stream data flow input, the first few frames of the video stream data are processed by the flow line segmentation model;
[0038] S212: Identify the position of the guide line frame by frame and generate a corresponding mask;
[0039] S213: performing an averaging process on all the masks obtained to obtain a guide line mask integrating information of multiple frames, thereby determining a guide line region with a maximum probability that is specific to a fixed guide line region relative to the image acquisition device; and saving the guide line region;
[0040] S22: Real-time monitoring and identification of vehicles based on real-time video stream data;
[0041] S221: Analyze the first frame in the video stream data in real time using the vehicle detection model to obtain the coordinate position of the target detection frame of each vehicle;
[0042] S222: When the vehicle detection model detects a vehicle, the target tracking model is started, and the DeepSORT algorithm is used to take the coordinate position of the target detection frame of the received vehicle as input, and a tracking ID is assigned to each vehicle, and the vehicle is tracked to extract the features of the vehicle.
[0043] S223: The DeepSORT algorithm continuously updates the status of the tracker along with the video stream data to ensure the tracking of the vehicle position, thereby achieving real-time monitoring and identification of the vehicle.
[0044] By adopting the above technical solution, combining YOLOv8 with DeepSORT can fully utilize the advantages of both and realize real-time monitoring and identification of vehicles. YOLOv8 provides fast and accurate target detection, while DeepSORT ensures stable tracking of vehicles in complex traffic environments. This combination not only improves the real-time performance of the system, but also enhances the ability to analyze vehicle behavior, providing a powerful tool for traffic monitoring. YOLOv8 contains four different target detection models: n / s / m / l / x. The YOLOv8s model is selected because it meets the requirements for real-time detection of vehicles in terms of accuracy and performance. The YOLOv8 target detection model is combined with the DeepSORT target tracking algorithm to achieve efficient monitoring of vehicles on the road and accurately track the coordinate frame position of each vehicle.
[0045] Preferably, in step S22, if the tracker does not track the corresponding detection frame in multiple consecutive frames, it is determined that the target has disappeared and is removed from the tracking list. If a new vehicle appears, a new tracker is initialized for the new vehicle and added to the tracking list.
[0046] Preferably, the specific steps of using the license plate recognition model in combination with the illegal vehicle image in the alarm generation module to recognize the license plate number in step S3 are:
[0047] S31: Determine whether the vehicle is crossing the guide line through the behavior analysis module;
[0048] S311: extracting the bottom edge coordinates from the target detection frame of the vehicle;
[0049] S312: Processing the mask image of the guide line to ensure its binary form;
[0050] S313: traverse each pixel point of the bottom edge of the vehicle, and check whether each pixel point is located in the mask area of the guide line. If one or more pixel points of the bottom edge of the vehicle are located in the mask area of the guide line, the bottom edge of the vehicle overlaps with the guide line, and it is determined that the vehicle has crossed the line and entered the guide line area.
[0051] S32: Record the boundary information of the target detection box of the vehicle crossing the line and the tracking ID assigned by the DeepSORT algorithm, and use a timing mechanism to track the duration of the vehicle's crossing the line. If the duration of the vehicle's crossing the line exceeds a preset threshold, the vehicle is judged to be an illegal vehicle; and the vehicle is marked as an alarm vehicle;
[0052] S33: intercepting the image data containing the warning vehicle from the original video stream according to the boundary information of the target detection frame, cropping it, and passing the cropped image data to the license plate recognition model to asynchronously trigger the license plate detection service;
[0053] S34: Use the license plate recognition model to locate the license plate and recognize the characters, and complete the recording and alarm of the illegal vehicles;
[0054] S341: using the received cropped image data of the warning vehicle as input, and using the license plate detection model to locate the license plate position;
[0055] S342: The character recognition model is then used to recognize the characters on the license plate, obtain the license plate number and record it, and transmit the information of the alarmed vehicle to the data management system for subsequent analysis and query, providing the traffic management department with a complete violation record.
[0056] With the above technical solution, through the steps, the line-crossing judgment logic can not only provide accurate detection results, but also adapt to different traffic scenes and lighting conditions, thereby ensuring the efficient operation of the traffic monitoring system. In addition, the bottom edge of the vehicle target frame is specially selected to calculate the intersection with the guide line area instead of the entire vehicle target frame. The advantage of this strategy is that it avoids false detection caused by camera angle deviation when the vehicle body is close to the guide line area. By focusing on the overlap of the bottom edge of the vehicle and the guide line, it is possible to more accurately judge whether the vehicle is actually crossing the line, thereby improving the judgment accuracy and reliability of the system.
[0057] Preferably, the identification system for traffic violations based on target detection and segmentation technology includes a data acquisition module, a target segmentation module, a target detection module, a behavior analysis module, an alarm generation module, a vehicle identification module and a data management system; the data acquisition module is responsible for collecting real-time video stream data from high-definition cameras installed at key traffic nodes, and the real-time video stream data serves as the basis for system analysis; the target segmentation module is a target segmentation technology based on deep learning, which accurately identifies the guide line area from the complex road background and provides clear spatial boundaries for vehicle behavior analysis; the target detection module uses the YOLOv8 algorithm to quickly and accurately detect the position and size of the vehicle from the video stream, and provide necessary spatial information for subsequent analysis ; The behavior analysis module is used to combine the results of target detection and target segmentation, analyze the relative position of the vehicle and the guide line through advanced image processing algorithms, and determine whether the vehicle is pressing the guide line; the alarm generation module is used to immediately trigger an alarm when the behavior analysis module detects that the vehicle is pressing the guide line, and transmit the image data of the alarm vehicle to the license plate recognition module and the data management system; the license plate recognition module uses OCR technology and deep learning models to automatically identify the license plate number from the screenshot of the image data of the alarm vehicle, and store the data in the database; the data management system is responsible for storing and managing all detected vehicle information, violation records and license plate recognition results. It provides data support for the system to facilitate subsequent query, analysis and report generation. The recognition system monitors traffic conditions in real time, and automatically identifies whether the vehicle is pressing the guide line, and identifies the license plate number in the case of violation.
[0058] Compared with the prior art, the present invention has the following beneficial effects: it utilizes a combination of multiple technologies to improve recognition capability and robustness, and significantly improves detection accuracy and efficiency; it can improve recognition accuracy and reduce missed detection and false detection while ensuring the ability to process a large amount of road monitoring data in real time; it provides more detailed data support for traffic management departments, and helps to accurately track and manage violations; specifically:
[0059] (1) Accurate identification of guide line areas: Through advanced target segmentation technology, the system can accurately identify guide line areas from complex road backgrounds. This requires not only that the algorithm has a high degree of recognition of the geometric shape and color features of the guide lines, but also that it can adapt to different lighting conditions and road conditions.
[0060] (2) Real-time vehicle monitoring and analysis: Using an efficient target detection algorithm, the system can monitor road conditions in real time, identify large vehicles on the road, and analyze the relationship between their driving trajectories and the guide line area to determine whether there is any line crossing behavior;
[0061] (3) Intelligent post-processing logic: After detecting possible violations of the guide line, the system will further verify the violation through a series of carefully designed logical judgments, such as the relative position of the vehicle and the guide line, the length of time the line has been violated, etc., to reduce false alarms;
[0062] (4) License plate recognition and data association: Once the system confirms that a vehicle has crossed the guide line, it will automatically capture the vehicle image and call the license plate recognition service to obtain the license plate number. This not only involves accurate license plate recognition, but also includes associating license plate information with violation data to provide a basis for subsequent violation processing. BRIEF DESCRIPTION OF THE DRAWINGS
[0063] Figure 1 The framework diagram and flow chart of the identification system for traffic violations based on target detection and segmentation technology of the present invention;
[0064] Figure 2 A schematic diagram of a target segmentation guide line data set of the method for identifying traffic violations based on target detection and segmentation technology of the present invention;
[0065] Figure 3 It is a schematic diagram of guide line segmentation sample annotation of the method for identifying traffic violations based on target detection and segmentation technology of the present invention;
[0066] Figure 4 For the corresponding Figure 3 Category labels and bounding box coordinates of the midline;
[0067] Figure 5 A schematic diagram of a target detection vehicle data set with a target frame for the method for identifying traffic violations based on target detection and segmentation technology of the present invention;
[0068] Figure 6 A vehicle data image with a target frame in the method for identifying traffic violations based on target detection and segmentation technology of the present invention;
[0069] Figure 7 For the corresponding Figure 6 The class labels and bounding box coordinates of the vehicles in ;
[0070] Figure 8 A schematic diagram of a clear partial vehicle image cut out of the method for identifying traffic violations based on target detection and segmentation technology of the present invention;
[0071] Fig. 9 A schematic diagram of a format suitable for LPRNet character recognition and YOLO license plate detection model training of the method for identifying traffic violations based on target detection and segmentation technology of the present invention;
[0072] Fig.10It is a trend chart of the changes of various indicators of the training set and the validation set with the number of training iterations during the training process of the guide line segmentation model of the method for identifying traffic violations based on target detection and segmentation technology of the present invention;
[0073] Fig.11 The ability of the guide line segmentation model of the method for identifying traffic violations based on target detection and segmentation technology of the present invention to identify guide lines at different confidence levels;
[0074] Fig.12 The change trend of various indicators of the training set and the validation set during the training process of the vehicle detection model of the method for identifying traffic violations based on target detection and segmentation technology of the present invention with the number of training iterations;
[0075] Fig.13 It is a partial iteration process of the LPRNet in the training process of the method for identifying traffic violations based on target detection and segmentation technology of the present invention and the final test accuracy (Test Accuracy) and test speed (TestSpeed);
[0076] Fig.14 It is a schematic diagram of the visualization of the change of the complete loss value of the LPRNet model training with the number of iterations (Epoch) of the identification method of traffic violations based on the target detection and segmentation technology of the present invention;
[0077] Fig.15 The test results and recognition confidence of the license plate detection and license plate number recognition method based on the target detection and segmentation technology in traffic violations of the present invention are shown. Figure 1 ;
[0078] Fig.16 The test results and recognition confidence of the license plate detection and license plate number recognition method based on the target detection and segmentation technology in traffic violations of the present invention are shown. Figure 2 ;
[0079] Fig.17 The test results and recognition confidence of the license plate detection and license plate number recognition method based on the target detection and segmentation technology in traffic violations of the present invention are shown. Figure 3 ;
[0080] Fig.18 Schematic diagram of the warning image display content of the method for identifying traffic violations based on target detection and segmentation technology of the present invention Figure 1 ;
[0081] Fig.19 The present invention is a schematic diagram of the present invention's method for identifying traffic violations based on target detection and segmentation technology. Figure 2 . DETAILED DESCRIPTION
[0082] The technical solution is further described below in conjunction with the accompanying drawings.
[0083] Example: Figure 1 As shown in the figure, the identification system for traffic violations based on target detection and segmentation technology includes a data acquisition module, a target segmentation module, a target detection module, a behavior analysis module, an alarm generation module, a vehicle identification module and a data management system; the data acquisition module is responsible for collecting real-time video stream data from high-definition cameras installed at key traffic nodes, and the real-time video stream data serves as the basis for system analysis; the target segmentation module is a target segmentation technology based on deep learning, which accurately identifies the guide line area from the complex road background and provides a clear spatial boundary for vehicle behavior analysis; the target detection module uses the YOLOv8 algorithm to quickly and accurately detect the position and size of the vehicle from the video stream, and provide the necessary spatial information for subsequent analysis ; The behavior analysis module is used to combine the results of target detection and target segmentation, analyze the relative position of the vehicle and the guide line through advanced image processing algorithms, and determine whether the vehicle is crossing the line; the alarm generation module is used to immediately trigger an alarm when the behavior analysis module detects the behavior of the vehicle crossing the guide line, and transmit the image data of the alarm vehicle to the license plate recognition module and the data management system; the license plate recognition module uses OCR technology and deep learning models to automatically identify the license plate number from the screenshot of the image data of the alarm vehicle, and store the data in the database; the data management system is responsible for storing and managing all detected vehicle information, violation records and license plate recognition results. It provides data support for the system to facilitate subsequent query, analysis and report generation. The call of the license plate recognition service is an efficient and intelligent process. When the target detection module detects an illegal vehicle and generates a screenshot of the alarm vehicle, the system will asynchronously trigger the license plate detection service. This asynchronous call mechanism means that license plate detection does not occupy real-time monitoring resources. The license plate recognition service will only be activated when a violation is detected and license plate information is required. Once the license plate number is identified and confirmed, the system will automatically record the relevant information into the data management system for subsequent analysis and inquiry, providing the traffic management department with a complete record of violations.
[0084] like Figure 1 As shown, the identification method for traffic violations based on target detection and segmentation technology specifically includes the following steps:
[0085] S1: The image acquisition device collects real-time video stream data to form a data set; trains the guide line segmentation model, vehicle detection model, and license plate recognition model;
[0086] The data set in step S1 includes a target segmentation guide line data set, a target detection vehicle data set and a license plate service data set; wherein the target segmentation guide line data set and the target detection vehicle data set are firstly annotated by the image data in the real-time video stream data, and then the annotated file is converted into a txt label file suitable for YOLO algorithm processing, and the txt label file includes a category label and a corresponding bounding box coordinate; the license plate service data set is firstly annotated by the image data in the real-time video stream data to obtain the coordinate position of the license plate and the label of the license plate number, and then the license plate image is cropped according to the license plate coordinates to form a format suitable for LPRNet character recognition and YOLO license plate detection model training;
[0087] In this embodiment, the target segmentation guide line dataset is collected from the Internet and the camera video streams on the highway. The selected camera should be able to clearly capture the traffic scene with guide lines, ensuring that the video stream contains diverse vehicle behaviors and different traffic conditions; static images are extracted from these video streams, ensuring that the selected images contain different time, lighting conditions and traffic flow, so as to increase the diversity and coverage of the dataset, such as Figure 2 As shown; then, use the labelme tool to annotate the extracted images with flow line segmentation samples; Figure 3 As shown in the figure, Labelme is an intuitive graphical annotation tool that enables you to accurately mark the location of the guide line and create a clear segmentation mask. Finally, convert the segmentation json data annotated by labelme into a txt file that can be processed by YOLO. This step is to adapt the annotation data to the input format of the YOLOv8 model to ensure that the model can correctly read and learn the label information. The converted txt file contains the necessary information for training the YOLOv8 model, including the category label and the corresponding bounding box coordinates, as shown in the figure. Figure 4 As shown, the corresponding Figure 3 The category labels and bounding box coordinates of the guide lines in the image are obtained. Through this series of steps, a dataset containing accurate guide line segmentation annotations is created. This dataset is used to train and verify the target segmentation algorithm to ensure that the guide line segmentation model can accurately learn and identify the guide line area.
[0088] The target detection vehicle dataset in this embodiment is made by collecting a large number of pictures with vehicles on the Internet and pictures extracted by high-speed cameras. These pictures cover a variety of vehicle types, colors and background environments to ensure the diversity and practicality of the dataset. Using the labelImg tool, the vehicle position in each picture can be accurately marked to form a dataset with target boxes, such as Figure 5As shown in the figure, after the labeling is completed, the labeling information is converted into a txt file that can be processed by the YOLO algorithm through the data preprocessing step. These files contain the coordinate information and category labels of each vehicle target box, providing direct data input for model training; Figure 7 Shown is the corresponding Figure 6 The class labels and bounding box coordinates of the vehicles in;
[0089] The license plate service data set in this embodiment mainly includes the data of license plate detection using yolo and the data of LPRNet character recognition. In order to build an efficient and comprehensive license plate recognition module, the vehicle target detection model is first used to process a large number of collected vehicle images, accurately locate the position of the vehicle, and crop out clear local vehicle images from them, such as Figure 8 As shown in , this step ensures that our dataset only contains images related to vehicles, improving the quality and availability of the data; First, we filter the cropped vehicle data to ensure that the dataset covers blue, green, yellow, and license plates in different regions and different light scenes, such as Figure 8 As shown. In the license plate labeling stage, PaddleOCR was used for semi-automatic labeling. PaddleOCR is a powerful optical character recognition library that can help us quickly identify the characters on the license plate. Although this process is semi-automatic, the recognition results are still carefully reviewed and corrected manually to ensure the accuracy of the labeling. Next, Python was used to process the data annotated by PPOCRLabel to obtain the coordinate position of the license plate, the label of the license plate number, and the license plate image according to the license plate coordinates. Finally, it was organized into a format suitable for LPRNet character recognition and YOLO license plate detection model training, as shown below Fig. 9 As shown;
[0090] The specific steps of training the flow line segmentation model in step S1 are:
[0091] S1-11: Divide the dataset into training set, validation set and test set;
[0092] S1-12: The guide line images and label files in the training set are configured and input into the YOLOv8s-seg network. The guide line segmentation model gradually learns to distinguish guide lines from other road elements by learning the features in the image and the shape and texture information of the guide lines;
[0093] S1-13: Use the validation set to adjust the hyperparameters of model training, and use data enhancement technology to optimize the model; hyperparameters include learning rate, batch size and number of iterations; train to obtain the guide line segmentation model; YOLOv8-seg includes four models of different scales: s / m / l / x; after comparison, the YOLOv8s-seg model is selected as the guide line segmentation model in this embodiment; and this embodiment trains the guide line segmentation model end-to-end, from image input to accurate segmentation mask output of the guide line, and the entire process is completed within the YOLOv8 framework. First, the organized guide line images and label files are configured and input into the YOLOv8s-seg network. The model gradually learns to distinguish guide lines from other road elements by learning the features in the image and the shape, texture and other information of the guide lines. During the training process, the performance of the model is optimized and its generalization ability is improved by adjusting hyperparameters such as learning rate, batch size and number of iterations, and using data enhancement technology. While the model is being trained, a part of the data set is used as a validation set to regularly evaluate the performance of the model. This helps monitor overfitting and adjust the training strategy in a timely manner. By comparing the performance of the model on different validation sets, we can further optimize the model structure and learning process to improve its accuracy and robustness in practical applications; Fig.10 As shown in the figure, it shows the changing trend of various indicators of the training set and the validation set with the number of training iterations during the training process of the model; Fig.10 As shown, on the training set, the bounding box loss (train / box_loss) and segmentation loss (train / seg_loss) are very low, 0.00628 and 0.00469 respectively, indicating that the model performs well in locating and segmenting objects. The classification loss (train / cls_loss) and target loss (train / obj_loss) are also very low, which means that the model is very accurate in classifying and identifying objects. In addition, the precision (precision) and recall (recall) of the model on different size categories (B represents large, M represents medium) are very high, which shows that the model can accurately identify and recall objects with almost no false detections or missed detections. The performance on the validation set is consistent with that on the training set, indicating that the model can maintain high performance on unseen data. The average precision (mAP) is very high at different IoU thresholds (0.5 and 0.5:0.95) and size categories, which shows that the model can maintain high accuracy detection in different situations;
[0094] S1-14: Use the test set to evaluate the flow line segmentation model through the F1-Confidence curve; that is, use the F1 score for evaluation. The F1 score calculation formula is:
[0095]
[0096] Wherein, Precision represents accuracy, Recall represents recall rate; F1 score is the harmonic mean of precision and recall rate; in this embodiment, target segmentation is a key link, which is responsible for accurately identifying the guide line area from the complex traffic scene. Fig.11 The ability of the flow line segmentation model to identify flow lines at different confidence levels. Compared with traditional segmentation models such as Mask R-CNN, U-Net, and DeepLab, YOLOv8-Seg has shown obvious advantages in speed and efficiency, end-to-end learning, and generalization. Compared with two-stage detectors such as Mask R-CNN, the single-stage design of YOLOv8-Seg greatly reduces the amount of calculation and achieves faster processing speed. Although U-Net performs well in the field of medical image segmentation, its symmetrical architecture leads to high computational costs and is not suitable for application scenarios that require real-time processing. Although DeepLab introduces dilated convolution to capture a wider range of contextual information, it relies on the CRFs post-processing step, which limits its real-time performance to a certain extent. The YOLOv8-Seg model has significant advantages in real-time, accuracy and flexibility, which make it very suitable for traffic violation identification tasks. Specifically, firstly, YOLOv8-Seg inherits the efficient detection capability of the YOLO series models. By introducing a deeper network structure, a more efficient feature fusion mechanism and an advanced training strategy, the YOLOv8-Seg model has achieved significant performance improvement in the target segmentation task, and can complete the target segmentation task while ensuring a high frame rate. Secondly, YOLOv8-seg adopts an innovative encoder-decoder architecture, and effectively improves the model's detection and segmentation capabilities for targets of different sizes by introducing feature pyramid networks (FPN) and multi-scale feature fusion technology. This multi-scale processing strategy enables YOLOv8-seg to adapt to the size changes of targets more flexibly when processing complex scenes, improves the segmentation accuracy of the model in complex scenes, and enables it to more accurately locate the location area of the guide line.
[0097] The specific steps of training the vehicle detection model in step S1 are:
[0098] S1-21: Divide the dataset into training set, validation set and test set;
[0099] S1-22: Configure the images containing vehicles in the training set and their label files, input them into the yolov8 target detection network, and gradually extract the vehicle features in the image through a series of convolutional layers, pooling layers, activation functions, etc. After multiple rounds of iterative training, the weight file for vehicle detection is finally obtained;
[0100] S1-23: Use the transfer learning method to use the weight file obtained by YOLOv8s pre-training as the starting point, use the validation set to adjust the hyperparameters, and learn the specific features of the vehicle to obtain the vehicle detection model;
[0101] S1-24: The final performance of the vehicle detection model is obtained by using the test set evaluation; When building a target detection system, the key factors we need to consider first include detection speed, accuracy, model size, and real-time processing capabilities. The YOLO model adopts a single-stage detection strategy and has become a benchmark in the field of target detection with its fast and efficient performance. From YOLOv1 to YOLOv7, the series has been continuously optimized, introducing technologies such as anchor boxes, multi-scale predictions, and deeper network structures to improve detection speed and accuracy, especially in real-time application scenarios. As the latest work in the series, YOLOv8 inherits and develops the advantages of the previous generation models, providing faster detection speed and higher accuracy. It simultaneously predicts bounding boxes, category probabilities, and target locations through a single network structure, achieving near real-time detection performance, which is very suitable for traffic violation detection scenarios that process large amounts of video stream data; The DeepSORT algorithm is an advanced target tracking model that further provides stable tracking capabilities based on YOLOv8 detection; DeepSORT can effectively predict the motion trajectory of vehicles and maintain accurate target associations in the interaction between vehicles by combining the Kalman filter and the Hungarian algorithm. This algorithm is particularly suitable for dealing with scenes with dense vehicles and partial occlusions, because it can distinguish and track individual vehicles through appearance features and motion information;
[0102] The YOLOv8 target detection model is used to train the vehicle dataset. First, a large number of images containing vehicles and their label files are collected and screened, and then configured and input into the yolov8 target detection network. After a series of convolutional layers, pooling layers, activation functions, etc., the vehicle features in the image are gradually extracted. After multiple rounds of iterative training, the weight file for vehicle detection is finally obtained; during the training process, the image adopts the transfer learning method, using the pre-trained YOLOv8s model weights as the starting point, and fine-tuning is performed on this basis to learn the specific features of the vehicle. In this way, the model can converge faster and improve the training efficiency. In addition, in order to evaluate the performance of the model, the cross-validation method is used to divide the dataset into a training set, a validation set, and a test set. The model is trained on the training set, the hyperparameters are adjusted on the validation set, and the final performance of the model is evaluated on the test set. Fig.12 The changing trend of various indicators of the training set and validation set with the number of training iterations during the training process of the model is shown. It can be seen that the loss value is gradually decreasing, while the precision and recall rate are gradually increasing, which indicates that the performance of the model is improving during the training process;
[0103] Table 4.1 shows the average accuracy of each type of car in the training set and the validation set. It can be seen from the table that the model has achieved high accuracy in both the training set and the validation set, that is, the model is more accurate in predicting each type of car;
[0104] Table 4.1 Average accuracy of each type of car in the training set and validation set
[0105]
[0106]
[0107] The license plate recognition model in step S1 includes a license plate detection model and a character recognition model, wherein the license plate detection model is responsible for locating the license plate area in the image, and the character recognition model is used to identify each character on the license plate for license plate detection; the license plate detection model is trained using a yolov5s network, and the input sample is a cropped vehicle image without other road background features. After model tuning and learning of the yolov5 network, a license plate detection model capable of locating the license plate position is finally obtained;
[0108] The character recognition model is trained using an end-to-end LPRNet network. The LPRNet network training uses a convolutional neural network (CNN) combined with a connection time classification (CTC) method, including a backbone, a global context feature extraction module, and a recognition head. The backbone uses a classic CNN structure to extract local features of license plate images. A 13x1 convolution module is designed at the end of the backbone to extract context information in the sequence direction. A fully connected layer is used outside the backbone to flatten the feature map output by the backbone into a one-dimensional vector, and extract global context features through nonlinear transformation. The features after feature fusion are input into the recognition head for final license plate character recognition. At the same time, LPRNet uses a CTC loss function during training, and the input sample is a cropped license plate image. After the LPRNet network structure and the CTC loss function, the character recognition model will repeatedly iterate the images and label files in the training data set, and then optimize the network parameters through the back propagation algorithm to finally obtain the optimal LPRNet license plate recognition model. Fig.13 The following figure shows some of the iterations of LPRNet during training and the final test accuracy and test speed. The complete loss value is visualized as it changes with the number of epochs. Fig.14 It can be seen that the loss gradually decreases with the number of iterations, showing the faster fitting ability of the model; the calculation formula for the test accuracy of the LPRNet license plate recognition model is:
[0109]
[0110] Among them, Tp is the number of samples that are truly recognized as license plates, that is, the samples correctly recognized by the model, Tn_1 and Tn_2 refer to the number of samples that are assumed to be non-license plates but are actually license plates, that is, the samples where the model mistakenly recognizes license plates as non-license plates; the core idea of this formula is to evaluate its performance by comparing the correct and incorrect predictions of the model, which represents the proportion of license plates correctly recognized by the model in all test samples. Therefore, the higher this ratio, the stronger the recognition ability of the model. The test accuracy of this model reached 0.998, so it can be seen that the character recognition ability of the model is good, and the average recognition speed in the test of 729 pictures reached 0.149ms, which shows that the model has a fast recognition speed; Fig.13 , Fig.14 , Fig.15 The test results and recognition confidence of license plate detection and license plate number recognition are shown respectively;
[0111] S2: After initializing the real-time video stream data, the guide line area is obtained through the guide line segmentation model and the guide line area is saved; at the same time, the vehicle position is located according to the real-time video stream data vehicle detection model;
[0112] The specific steps of step S2 are:
[0113] S21: extracting the features of the guide line by using the guide line segmentation model to obtain the guide line area;
[0114] S211: within the first second of the real-time video stream data flow input, the first few frames of the video stream data are processed by the flow line segmentation model;
[0115] S212: Identify the position of the guide line frame by frame and generate corresponding masks; these masks represent the precise position and shape of the guide line in each frame;
[0116] S213: Averaging all the masks obtained to obtain a guide line mask that integrates multi-frame information, thereby determining the guide line area with the highest probability and specific to the fixed and unchanging area relative to the image acquisition device; and saving the guide line area; this process effectively smooths the instantaneous errors caused by camera jitter or light changes, ensuring the accuracy and stability of the guide line position; the advantages of this processing process are two-fold: first, it greatly reduces resource usage, because we do not need to perform real-time target segmentation inference on each frame of the video stream; second, it avoids changes in the mask area caused by temporary occlusion of the guide line area by vehicles or other objects, which may interfere with the judgment of whether the vehicle presses the guide line; through this method, we not only improve the efficiency of the system, but also enhance the robustness of the system. The static feature extraction of the guide line provides a stable and reliable reference for vehicle behavior analysis, allowing the system to more accurately judge whether the vehicle passes through the guide line area;
[0117] S22: Real-time vehicle monitoring and identification based on real-time video stream data;
[0118] S221: Analyze the video stream data in real time using the vehicle detection model to obtain the coordinate position of the target detection frame of each vehicle;
[0119] S222: When the vehicle detection model detects a vehicle, the target tracking model is started, and the DeepSORT algorithm is used to take the coordinate position of the target detection frame of the received vehicle as input, assign a tracking ID to each vehicle, start tracking the vehicle, and extract the features of the vehicle; the DeepSORT algorithm uses the appearance features and motion information of the vehicle, such as color, shape, and position changes in consecutive frames, to extract the unique features of each vehicle;
[0120] S223: The DeepSORT algorithm continuously updates the status of the tracker along with the video stream data to ensure the tracking of the vehicle position, thereby achieving real-time monitoring and identification of the vehicle;
[0121] In step S22, if the tracker does not track the corresponding detection frame in multiple consecutive frames, it is determined that the target has disappeared and is removed from the tracking list. If a new vehicle appears, a new tracker is initialized for the new vehicle and added to the tracking list.
[0122] S3: judging whether the vehicle is crossing the guide line by combining the guide line area and the vehicle position through the behavior analysis module; if not, returning to step S2, specifically returning to step S22; if yes, using the license plate recognition model combined with the illegal vehicle image in the alarm generation module to identify the license plate number, and then sending the result to the data management system to record the illegal vehicle information;
[0123] The specific steps of using the license plate recognition model in combination with the illegal vehicle image in the alarm generation module to recognize the license plate number in step S3 are:
[0124] S31: Determine whether the vehicle is crossing the guide line through the behavior analysis module;
[0125] S311: extracting the bottom edge coordinates from the target detection box of the vehicle; specifically, if the vehicle bounding box is defined by coordinates (x_min, y_min, x_max, y_max), then the bottom edge coordinates are a series of points from (x_min, y_min) to (x_max, y_min);
[0126] S312: Processing the mask image of the guide line to ensure its binary form; meaning that the guide line area is marked as white (255), while the rest of the image is black (0);
[0127] S313: traverse each pixel point of the bottom edge of the vehicle, and check whether each pixel point is located in the mask area of the guide line. If one or more pixel points of the bottom edge of the vehicle are located in the mask area of the guide line, the bottom edge of the vehicle overlaps with the guide line, and it is determined that the vehicle has crossed the line and entered the guide line area.
[0128] S32: Record the boundary information of the target detection frame of the vehicle that crosses the line and the tracking ID assigned by the DeepSORT algorithm, and use a timing mechanism to track the duration of the vehicle's crossing the line. If the duration of the vehicle's crossing the line exceeds a preset threshold, the vehicle is judged to be an illegal vehicle; the vehicle is marked as an alarm vehicle; in the process of the system monitoring the road video stream in real time, once it is detected that a vehicle in the current frame has violated the rules and crossed the guide line, the system will immediately record the target boundary frame of the vehicle and the tracking ID assigned by the DeepSORT algorithm; in order to accurately determine whether the vehicle continues to violate the rules, the system uses a timing mechanism to track the duration of the vehicle's crossing the line; if the vehicle's crossing the line continues for a certain period of time, the system will automatically determine that the vehicle is an illegal vehicle; once the vehicle is determined to be illegal, the system will immediately intercept the picture containing the illegal vehicle from the original video stream based on the target boundary box information; then, these pictures will be passed to the license plate recognition service for further license plate recognition and processing to complete the entire illegal vehicle recording and alarm process;
[0129] S33: intercepting the image data containing the warning vehicle from the original video stream according to the boundary information of the target detection frame, cropping it, and passing the cropped image data to the license plate recognition model to asynchronously trigger the license plate detection service;
[0130] S34: Use the license plate recognition model to locate the license plate and recognize the characters, and complete the recording and alarm of the illegal vehicles;
[0131] S341: using the received cropped image data of the warning vehicle as input, and using the license plate detection model to locate the license plate position;
[0132] S342: The character recognition model is then used to recognize the characters on the license plate, obtain the license plate number and record it, and transmit the information of the alarmed vehicle to the data management system for subsequent analysis and query, providing the traffic management department with a complete violation record.
[0133] In this embodiment, the license plate number recognition process is an important part of the intelligent transportation system, which aims to accurately extract the license plate number information from the input warning vehicle picture, and is generally divided into the license plate positioning stage and the character recognition stage. First, the system receives the cropped warning vehicle picture as input. Since the vehicle picture may contain redundant background information, direct character recognition will result in a decrease in accuracy. Therefore, the YOLOv5s target detection model is used to accurately locate the license plate position. With its excellent detection speed and accuracy, YOLOv5s can accurately frame the area where the license plate is located in a short time, providing high-quality license plate images for subsequent steps. In the character recognition stage, we use LPRNet, a deep learning model specially designed for license plate recognition. The core advantage of LPRNet lies in its end-to-end training method. Unlike the complex process of traditional license plate recognition systems that first need to perform character segmentation and then recognize one by one, LPRNet can directly recognize the license plate text from the original image without any intermediate steps. In addition, LPRNet abandons the traditional recurrent neural network (RNN) and adopts a pure convolutional neural network (CNN) combined with a connection time classification (CTC) method. LPRNet mainly consists of three parts: backbone, global context feature extraction module and recognition head. The backbone adopts the classic CNN structure to extract local features of license plate images. A 13x1 convolution module is designed at the end of the backbone to extract context information in the sequence direction. In addition, in order to capture global context features, a fully connected layer is used outside the backbone to flatten the feature map output by the backbone into a one-dimensional vector, and extract global context features through nonlinear transformation. The features after feature fusion are input to the recognition head for final license plate character recognition. Since the output of the decoder is different from the target character sequence length, LPRNet uses the CTC loss function during training, which can well cope with end-to-end training that does not require character segmentation and alignment. Through this process, our license plate number recognition system can efficiently and accurately extract license plate information from warning vehicle pictures.
[0134] The system input is real-time video stream data. First, in the initialization stage, the target segmentation technology is used to accurately extract the guide line area from the video stream. Then, the target detection model is used to monitor and identify the vehicle in real time, ensuring real-time tracking of vehicle dynamics. When the system detects the intersection of the vehicle and the guide line, that is, when the vehicle crosses the line, the system will automatically trigger the alarm mechanism. After the alarm is triggered, the system not only records the video footage at the moment of the alarm, but also captures the image of the illegal vehicle. Subsequently, the system calls the license plate recognition service, uses the previously trained license plate recognition model to process the captured vehicle image, and automatically identifies and outputs the license plate number. Finally, the system sends the video footage and license plate number and other information of the alarm vehicle to the business system for further processing and analysis by the traffic management department. Fig.18 and Fig.19 The following is the warning image display content (the license plate number is fictitious to protect privacy).
[0135] The guide line area is obtained using the target segmentation model during task initialization, and the vehicle target frame is obtained in real time through the target detection model. When the vehicle violates the guide line rule through logical judgment, the license plate recognition service is called to identify the license plate number of the illegal vehicle and transmit it to the back end. Finally, the guide line area and target frame can be drawn through the front end, and the corresponding warning words can be printed in the upper right corner of the picture.
[0136] Through the analysis of the results of the system deployment, we can see that the intelligent traffic monitoring system has shown high efficiency and accuracy in practical applications. It can process a large amount of video data in real time, quickly and accurately identify traffic violations, and reduce manual intervention through automated processes, thereby improving the level of intelligent traffic management. In addition, the deployment of the system also helps to improve road safety, reduce accidents caused by traffic violations, and provide technical support for building a safer and more efficient traffic environment.
[0137] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention, such as changes in certain materials or reaction conditions, should be included in the scope of protection of the present invention.
Claims
1. A method for identifying traffic violations based on target detection and segmentation technology, characterized in that: The specific steps include: S1: The image acquisition device collects real-time video stream data to form a data set; trains the guide line segmentation model, vehicle detection model, and license plate recognition model; S2: After initializing the real-time video stream data, the guide line area is obtained through the guide line segmentation model and the guide line area is saved; after initialization, the vehicle position is located according to the real-time video stream data vehicle detection model; S3: Combine the guide line area and vehicle position to determine whether the vehicle crosses the guide line through the behavior analysis module. If not, return to step S2; if yes, use the license plate recognition model combined with the illegal vehicle image in the alarm generation module to identify the license plate number, and then send the result to the data management system to record the illegal vehicle information.
2. The method for identifying traffic violations based on target detection and segmentation technology according to claim 1 is characterized in that: The data sets in step S1 include a target segmentation guide line data set, a target detection vehicle data set and a license plate service data set; wherein the target segmentation guide line data set and the target detection vehicle data set are firstly annotated with the image data in the real-time video stream data, and then the annotated file is converted into a txt label file suitable for YOLO algorithm processing; the txt label file includes a category label and a corresponding bounding box coordinate; the license plate service data set is firstly annotated with the image data in the real-time video stream data to obtain the coordinate position of the license plate and the label of the license plate number, and then the license plate image is cropped according to the license plate coordinates to form a format suitable for LPRNet character recognition and YOLO license plate detection model training.
3. The method for identifying traffic violations based on target detection and segmentation technology according to claim 2 is characterized in that: The specific steps of training the flow line segmentation model in step S1 are: S1-11: Divide the dataset into training set, validation set and test set; S1-12: The guide line images and label files in the training set are configured and input into the YOLOv8s-seg network. The guide line segmentation model gradually learns to distinguish guide lines from other road elements by learning the features in the image and the shape and texture information of the guide lines; S1-13: Use the validation set to adjust the hyperparameters of the model training and use data augmentation technology to optimize the model; the hyperparameters include learning rate, batch size and number of iterations; train to obtain the flow line segmentation model; S1-14: Use the test set to evaluate the flow line segmentation model through the F1-Confidence curve; that is, use the F1 score for evaluation. The F1 score calculation formula is: Among them, Precision represents accuracy, Recall represents recall rate; F1 score is the harmonic mean of precision and recall rate.
4. The method for identifying traffic violations based on target detection and segmentation technology according to claim 3 is characterized in that: The specific steps of training the vehicle detection model in step S1 are: S1-21: Divide the dataset into training set, validation set and test set; S1-22: Configure the images containing vehicles in the training set and their label files, input them into the yolov8 target detection network, and gradually extract the vehicle features in the image through a series of convolutional layers, pooling layers, and activation functions. After multiple rounds of iterative training, the weight file for vehicle detection is finally obtained; S1-23: Use the transfer learning method to use the weight file obtained by YOLOv8s pre-training as the starting point, use the validation set to adjust the hyperparameters, and learn the specific features of the vehicle to obtain the vehicle detection model; S1-24: The final performance of the vehicle detection model is obtained using the test set evaluation.
5. The method for identifying traffic violations based on target detection and segmentation technology according to claim 4 is characterized in that: The license plate recognition model in step S1 includes a license plate detection model and a character recognition model. The license plate detection model is responsible for locating the license plate area in the image, and the character recognition model is used to identify each character on the license plate for license plate detection; the license plate detection model is trained using the yolov5s network, and the input sample is a cropped vehicle picture without other road background features. After model tuning and learning of the yolov5 network, a license plate detection model that can locate the license plate position is finally obtained.
6. The method for identifying traffic violations based on target detection and segmentation technology according to claim 4 is characterized in that: The character recognition model is trained using an end-to-end LPRNet network. The LPRNet network is trained using a convolutional neural network (CNN) combined with a connection temporal classification (CTC) method, including a backbone, a global context feature extraction module, and a recognition head. The backbone uses a classic CNN structure to extract local features of license plate images. A 13x1 convolution module is designed at the end of the backbone to extract context information in the sequence direction. A fully connected layer is used outside the backbone to flatten the feature map output by the backbone into a one-dimensional vector, and extract global context features through nonlinear transformation. The features after feature fusion are input into the recognition head for the final license plate character recognition; at the same time, LPRNet uses the CTC loss function during training, and the input sample is a cropped license plate image. After the LPRNet network structure and CTC loss function, the character recognition model will repeatedly iterate the images and label files in the training data set, and then optimize the network parameters through the back propagation algorithm to finally obtain the optimal LPRNet license plate recognition model; the calculation formula for the test accuracy of the LPRNet license plate recognition model is: Among them, Tp is the number of samples truly recognized as license plates, that is, the samples correctly recognized by the model, Tn_1 and Tn_2 refer to the number of samples assumed to be non-license plates but are actually license plates, that is, the samples that the model mistakenly recognizes license plates as non-license plates.
7. The method for identifying traffic violations based on target detection and segmentation technology according to claim 4 is characterized in that: The specific steps of step S2 are: S21: extracting the features of the guide line by using the guide line segmentation model to obtain the guide line area; S211: within the first second of the real-time video stream data flow input, the first few frames of the video stream data are processed by the flow line segmentation model; S212: Identify the position of the guide line frame by frame and generate a corresponding mask; S213: performing an averaging process on all the masks obtained to obtain a guide line mask integrating information of multiple frames, thereby determining a guide line area that is fixed relative to the image acquisition device; and saving the guide line area; S22: Real-time monitoring and identification of vehicles based on real-time video stream data; S221: Analyze the video stream data in real time using the vehicle detection model to obtain the coordinate position of the target detection frame of each vehicle; S222: When the vehicle detection model detects a vehicle, the target tracking model is started, and the DeepSORT algorithm is used to take the coordinate position of the target detection frame of the received vehicle as input, and a tracking ID is assigned to each vehicle, and the vehicle is tracked to extract the features of the vehicle. S223: The DeepSORT algorithm continuously updates the status of the tracker along with the video stream data to ensure the tracking of the vehicle position, thereby achieving real-time monitoring and identification of the vehicle.
8. The method for identifying traffic violations based on target detection and segmentation technology according to claim 7 is characterized in that: In step S22, if the tracker does not track the corresponding detection frame in multiple consecutive frames, it is determined that the target has disappeared and is removed from the tracking list. If a new vehicle appears, a new tracker is initialized for the new vehicle and added to the tracking list.
9. The method for identifying traffic violations based on target detection and segmentation technology according to claim 8 is characterized in that: The specific steps of using the license plate recognition model in combination with the illegal vehicle image in the alarm generation module to recognize the license plate number in step S3 are: S31: Determine whether the vehicle is crossing the guide line through the behavior analysis module; S311: extracting the bottom edge coordinates from the target detection frame of the vehicle; S312: Processing the mask image of the guide line to ensure its binary form; S313: traverse each pixel point of the bottom edge of the vehicle, and check whether each pixel point is located in the mask area of the guide line. If one or more pixel points of the bottom edge of the vehicle are located in the mask area of the guide line, the bottom edge of the vehicle overlaps with the guide line, and it is determined that the vehicle has crossed the line and entered the guide line area. S32: Record the boundary information of the target detection box of the vehicle crossing the line and the tracking ID assigned by the DeepSORT algorithm, and use a timing mechanism to track the duration of the vehicle's crossing the line. If the duration of the vehicle's crossing the line exceeds a preset threshold, the vehicle is judged to be an illegal vehicle; and the vehicle is marked as an alarm vehicle; S33: intercepting the image data containing the warning vehicle from the original video stream according to the boundary information of the target detection frame, cropping it, and passing the cropped image data to the license plate recognition model to asynchronously trigger the license plate detection service; S34: Use the license plate recognition model to locate the license plate and recognize the characters, and complete the recording and alarm of the illegal vehicles; S341: using the received cropped image data of the warning vehicle as input, and using the license plate detection model to locate the license plate position; S342: The character recognition model is then used to recognize the characters on the license plate, obtain the license plate number and record it, and transmit the information of the alarmed vehicle to the data management system for subsequent analysis and query, providing the traffic management department with a complete violation record.
10. The method for identifying traffic violations based on target detection and segmentation technology according to claim 9, characterized in that: The identification system for traffic violations based on target detection and segmentation technology includes a data acquisition module, a target segmentation module, a target detection module, a behavior analysis module, an alarm generation module, a vehicle identification module and a data management system; the data acquisition module is responsible for collecting real-time video stream data from high-definition cameras installed at key traffic nodes, and the real-time video stream data serves as the basis for system analysis; the target segmentation module is a target segmentation technology based on deep learning, which accurately identifies the guide line area from the road background and provides a clear spatial boundary for vehicle behavior analysis; the target detection module uses the YOLOv8 algorithm to detect the position and size of the vehicle from the video stream and provide necessary spatial information for subsequent analysis; the behavior analysis module is used to combine the results of target detection and target segmentation, analyze the relative position of the vehicle and the guide line through the image processing algorithm, and judge whether the vehicle is pressing the line; the alarm generation module is used to trigger an alarm when the behavior analysis module detects the behavior of the vehicle pressing the guide line, and transmit the image data of the alarmed vehicle to the license plate recognition module and the data management system; The license plate recognition module uses OCR technology and deep learning models to automatically identify license plate numbers from screenshots of image data of the alarm vehicle, and stores the data in a database; the data management system is responsible for storing and managing all detected vehicle information, violation records and license plate recognition results. It provides data support for the system to facilitate subsequent query, analysis and report generation.
Citation Information
Cited By
Dynamic road toll charging method and system based on vehicle type and vehicle driving path
CN119723699A
Freight vehicle roadside illegal parking detection method and system based on image recognition
CN120088994A
Intelligent dust removal system and dust removal method for cutting machine and electronic equipment
CN120134045A
Intelligent dust removal system and method for cutting machine, and electronic device
CN120134045B
Intelligent monitoring method for illegal behaviors of personnel in elimination and control room
CN120655884A
