Deep learning-based method and system for discriminating illegal left turn of non-motor vehicles at intersection

Abstract

The invention discloses an intersection non-motor vehicle illegal left-turn discrimination method based on deep learning, and relates to the field of intelligent traffic, and the method comprises the steps: dividing an area where non-motor vehicles appear at an intersection, and marking a left-turn waiting area, a left-turn inevitable area and a left-turn exit in the area; capturing a picture of a non-motor vehicle passing through the area through a camera installed in the divided area, and carrying out data preprocessing; using a pre-trained convolutional neural network model RestNet-50 to perform feature extraction on pictures captured by a regional camera, and constructing three different feature databases; respectively calculating feature similarities of sample images in the left-turn waiting area, the left-turn inevitable area and the left-turn exit, retaining the picture with the highest similarity, and establishing a corresponding relation database; according to the method, whether the vehicle turns left illegally or not is judged through the database of the corresponding relation between the pictures, the illegal left turning behaviors are effectively reduced, and intersection traffic safety is assisted.

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation, and particularly to a method and system for discriminating illegal left turns of non-motor vehicles at intersections based on deep learning. Background Art

[0002] As an important traffic flow center in transportation, intersections frequently witness traffic accidents. However, studies have found that some traffic accidents are caused by illegal driving of vehicles at intersections, including motor vehicles and non-motor vehicles. In recent years, with the development of new energy, the number of non-motor vehicles has been increasing day by day, and their illegal driving behaviors have brought huge potential safety hazards and troubles to traffic safety. Especially at intersections, the illegal left-turn behaviors of non-motor vehicles usually occur in the visual blind spots of normally driving motor vehicles, thus often causing various traffic accidents, and the impacts and hazards caused are huge. Therefore, it is necessary to explore a method for discriminating and reminding illegal left-turn behaviors of non-motor vehicles at intersections. Summary of the Invention

[0003] Object of the Invention: The present invention proposes a method and system for discriminating illegal left turns of non-motor vehicles at intersections based on deep learning, aiming to solve the problem of potential safety hazards caused by illegal left turns of non-motor vehicles at current intersections.

[0004] Technical Solution: To achieve the object of the invention, the technical solution adopted by the present invention is as follows: A method for discriminating illegal left turns of non-motor vehicles at intersections based on deep learning, including the following steps: Divide the area where non-motor vehicles appear at the intersection, and mark the left-turn waiting area, left-turn necessary area, and left-turn exit within this area; Capture pictures of non-motor vehicles passing through this area by cameras installed in the divided area, and perform data preprocessing; Use the pre-trained convolutional neural network model RestNet-50 to extract features from the pictures captured by the area cameras respectively, and construct three different feature databases; Calculate the feature similarity between the sample images in the left-turn waiting area and the left-turn necessary area, and the left-turn exit respectively, retain the picture with the highest similarity, and establish a corresponding relationship database; Judge whether the vehicle makes an illegal left turn through the corresponding relationship database between pictures.

[0005] Further, the step of dividing the area where non-motor vehicles appear at the intersection and marking the left-turn waiting area, left-turn necessary area, and left-turn exit within this area includes: Take the area extending around from the intersection point of the zebra crossing as the area where non-motor vehicles appear at the intersection.

[0006] Further, the step of capturing pictures of non-motor vehicles passing through this area by cameras installed in the divided area and performing data preprocessing includes: Capture non-motor vehicle images passing through the area by area cameras, organize the captured images, and group them according to the area. Use the object detection model YOLOv8 to preprocess each group of images and crop out the non-motor vehicle images.

[0007] Furthermore, use the pre-trained convolutional neural network model RestNet-50 to extract features from the pictures captured by the area cameras respectively, and construct three different feature databases, including: Use RestNet-50 to extract 512-dimensional feature vectors of vehicle images in different areas respectively, and construct feature databases for three areas.

[0008] Furthermore, calculate the feature similarity of the sample images in the left-turn waiting area, the left-turn necessary area, and the left-turn exit respectively, retain the pictures with the highest similarity, and establish a corresponding relationship database, including: Calculate the cosine similarity of the sample images between the left-turn waiting area and the left-turn exit, and between the left-turn waiting area and the left-turn necessary area; Retain the group of pictures with the highest cosine similarity value, and establish a corresponding relationship database between them.

[0009] Furthermore, determine whether a vehicle makes an illegal left turn through the corresponding relationship database between pictures, including: For the picture corresponding relationship database, when the values of the attribute columns left-turn waiting area, left-turn necessary area, and left-turn exit are all 1, the non-motor vehicle represented by the picture tuple makes a normal left turn; While for the picture tuple with the values of the attribute columns left-turn waiting area and left-turn exit being 1 and the value of the left-turn necessary area being 0, the non-motor vehicle represented makes an illegal left turn.

[0010] Furthermore, use the area extending from the intersection of the zebra crossings to the surrounding areas as the area where non-motor vehicles appear at the intersection, including: A circular area covered by a circle with the intersection of the zebra crossings as the center and a radius of 1m; Mark the possible appearance position areas as A, B, C, D respectively, and correspond A, B, C to the left-turn waiting area, the left-turn necessary area, and the left-turn exit respectively.

[0011] Furthermore, capture non-motor vehicle images passing through the area by area cameras, organize the captured images, and group them according to the area, including: Classify the pictures taken from different areas into the groups of that area; The groups are the left-turn waiting area, the left-turn necessary area, and the left-turn exit respectively.

[0012] Further, preprocessing each group of images using the target detection model YOLOv8 and cropping out non-motor vehicle images includes: Locating the target bounding box through the Detection Head; Predicting whether each pixel within the bounding box belongs to the target through the Segmentation Head and generating a mask based on the prediction results; Cropping the image according to the generated mask to obtain the non-motor vehicle object in the captured image; Further, calculating the cosine similarity between the left-turn waiting area and the left-turn exit, and between the left-turn waiting area and the necessary left-turn area for sample images includes: Calculating the global feature similarity between feature maps according to the formula of cosine similarity: Cosine similarity = A * B / (|A| * |B|) where A and B respectively represent the 512-dimensional feature vectors after normalization of the feature maps, and |A| and |B| respectively represent the vector norms; Further, retaining the group of images with the highest cosine similarity value and establishing a corresponding relationship database between them includes: Regarding the group of images with the highest cosine similarity as the successfully matched group of images; Taking whether it appears in the three areas as the judgment criterion, and using 1 and 0 boolean values to represent "yes" and "no". For the images that are successfully matched between the left-turn waiting area and the necessary left-turn area, and between the left-turn waiting area, set their values to 1, otherwise 0.

[0013] Further, using RestNet-50 to extract 512-dimensional feature vectors of vehicle images in different areas and constructing a feature database for the three areas includes: Loading the pre-trained RestNet-50 pre-trained model and removing the original fully connected layer of the model; The non-motor vehicle object images cropped by YOLOv8 need to be standardized, resized, and center-cropped in Resnt-50; Extracting the feature vectors and normalizing them; Further, locating the target bounding box through the Detection Head includes: Dividing and performing Anchor-free prediction on the multi-scale feature maps from the Neck network; Performing bounding box decoding; Further, predicting whether each pixel within the bounding box belongs to the target through the Segmentation Head and generating a mask based on the prediction results includes: ROI Extraction: Crop the corresponding region from the fused high-level feature map according to the predicted coordinate box of the detection head. Mask Generation: Generate K prototype masks through the convolutional layer of the segmentation head. The detection head additionally outputs a K-dimensional coefficient vector for each instance, and generates an instance-level mask by linearly combining the prototype masks: Maskinstance = ∑ coefficientk * ProtoMaskk (k = 1…K) Activation Binarization: Map the mask to [0, 1] through the sigmoid activation function and set a threshold to generate a binary mask; Furthermore, cropping the image according to the generated mask to obtain non-motor vehicle objects in the captured image includes: Mask Post-Processing: Crop the mask to the target area according to the finally retained bounding box, thereby obtaining the target non-motor vehicle object; Furthermore, extracting the feature vector and normalizing it includes: Gradually extract features through convolutional layers and residual blocks; Compress the output of the last convolutional layer through global average pooling; Add a fully connected layer after the pooling layer to reduce the 2048-dimensional feature vector to a 512-dimensional feature vector; Normalize the 512-dimensional feature vector to obtain a normalized feature vector for subsequent similarity calculation; Furthermore, partitioning the multi-scale feature map from the Neck network and performing Anchor-free prediction includes: Partition the feature map into an S*S grid, and each grid predicts the center offset (∆x, ∆y) and width and height (w, h) of the target. Adopt the Anchor-free mechanism to avoid the limitation of predefined anchor boxes; Output (4 + 1 + C) values for each grid prediction; Furthermore, performing bounding box decoding includes: Convert the Anchor-free prediction value to an actual coordinate value: Ximg = (σ(∆x) + gx) * s Yimg = (σ(∆y) + gy) * s Where gx, gy represent the coordinates of the upper left corner of the grid, s is the downsampling step of the feature map, and σ is the sigmoid function; Furthermore, outputting (4 + 1 + C) for each grid prediction includes: Output 4: Bounding box coordinates, center point offset and width and height, normalized to 01; Output 1: Confidence, that is, the probability of the existence of the target, activated through the sigmoid function. Output C: Category probability, i.e., Softmax and Sigmoid multi-label classification; The present invention also provides a non-motor vehicle illegal left-turn discrimination system at intersections based on deep learning, including a data collection module, a feature extraction module, a left-turn data module, an illegal left-turn determination module, a voice reminder module, and a visualization module; The data collection module, which is composed of cameras installed in various areas, is used to collect non-motor vehicle images in various areas, and crop and organize the non-motor vehicle images to form different databases; The feature extraction module extracts the features of the processed sample images and forms a 512-dimensional feature vector through convolutional neural network processing; The left-turn data module is used to calculate the similarity of the sample images, establish a corresponding relationship database according to the similarity matching relationship, and store the corresponding relationships of non-motor vehicles; The illegal left-turn determination module is used to discriminate the illegal left-turn behavior of non-motor vehicles according to the values in the corresponding relationship database; The voice reminder module reminds non-motor vehicles not to make illegal left-turns in the non-motor vehicle left-turn waiting area, and gives a secondary reminder to non-motor vehicles that make illegal left-turns at the left-turn exit; The visualization module is used to reflect statistical data in the system, including the non-motor vehicle flow at intersections, the non-motor vehicle flow of illegal left-turns, the number of voice reminders, and the comparison of the number of illegal left-turn behaviors of non-motor vehicles before and after using the system.

[0014] Compared with the prior art, the technical solution of the present invention has the following beneficial effects: The method and system for discriminating illegal left-turns of non-motor vehicles at intersections based on deep learning provided by the present invention particularly considers the complex and diverse traffic conditions and large vehicle flow at intersections. By obtaining image data to discriminate the driving behaviors of non-motor vehicles, it first determines whether a non-motor vehicle makes a left turn, and then on this basis, judges whether the non-motor vehicle is illegal through its trajectory. By combining methods such as regional division, deep learning, and image recognition, the discrimination of illegal behaviors of non-motor vehicles becomes easier and more intelligent; at the same time, by setting up a voice reminder module, non-motor vehicles can be reminded, thus effectively regulating the driving behaviors of non-motor vehicles; by collecting and processing the driving data of non-motor vehicles through an integrated system and displaying it through a visualization interface, it helps to analyze the occurrence of illegal behaviors of non-motor vehicles at intersections, thereby reducing the occurrence of traffic accidents. Description of the Drawings

[0015] Figure 1 is the working flow chart of the method for discriminating illegal left-turns of non-motor vehicles at intersections based on deep learning of the present invention; Figure 2Schematic diagram of the non-motor vehicle illegal left-turn discrimination system module at intersections based on deep learning of the present invention; Figure 3 Flowchart of the specific operation of the non-motor vehicle illegal left-turn discrimination method and system at intersections based on deep learning of the present invention; Figure 4 Schematic diagram of intersection area division and illegal left-turn behavior determination for the non-motor vehicle illegal left-turn discrimination method at intersections based on deep learning. Specific implementation manner

[0016] The technical solution of the present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0017] Figure 1 The flowchart of the non-motor vehicle illegal left-turn discrimination method based on deep learning provided for the embodiments of the present invention. The non-motor vehicle illegal left-turn discrimination method at intersections includes: Dividing the area where non-motor vehicles appear at intersections and marking the left-turn waiting area, left-turn necessary area, and left-turn exit within this area; Further, taking the area extending around from the intersection point of the zebra crossing as the area where non-motor vehicles appear at intersections. The main steps are as follows: Taking the area extending around from the intersection point of the zebra crossing as the area where non-motor vehicles appear at intersections.

[0018] Further, taking the area extending around from the intersection point of the zebra crossing as the area where non-motor vehicles appear at intersections mainly includes: A circular area covered by a circle with a radius of 1 m centered at the intersection point of the zebra crossings; Marking the possible position areas as A, B, C, D respectively, and corresponding A, B, C to the left-turn waiting area, left-turn necessary area, and left-turn exit respectively.

[0019] As Figure 4 shown, the schematic diagram of intersection area division and illegal left-turn behavior determination for the non-motor vehicle illegal left-turn discrimination method at intersections based on deep learning marks the possible position areas A, B, C, D of non-motor vehicles at intersections, as well as the divided left-turn waiting area, left-turn necessary area, and left-turn exit.

[0020] Capturing non-motor vehicle pictures passing through this area by cameras installed in the divided area and performing data preprocessing; Further, capturing non-motor vehicle pictures passing through this area by cameras installed in the divided area and performing data preprocessing mainly includes: Capturing non-motor vehicle images passing through this area by area cameras, organizing the captured images, and grouping them according to the area; Preprocess each group of images using the target detection model YOLOv8 to crop out non-motor vehicle images.

[0021] Preferably, capture non-motor vehicle images passing through the area via area cameras, organize the captured images, and group them according to the area. The steps are as follows: Classify the pictures taken from different areas into the groups of that area; The groups are the left-turn waiting area, the necessary area for left-turn, and the left-turn exit respectively; Furthermore, the steps of preprocessing each group of images using the target detection model YOLOv8 to crop out non-motor vehicle images are as follows: Locate the target bounding box through the Detection Head; Predict whether each pixel within the bounding box belongs to the target through the Segmentation Head, and generate a mask according to the prediction results; Crop the image according to the generated mask to obtain the non-motor vehicle object in the captured image; Furthermore, the steps of locating the target bounding box through the Detection Head are as follows: Divide and perform Anchor-free prediction on the multi-scale feature maps from the Neck network; Perform bounding box decoding; Furthermore, the steps of dividing and performing Anchor-free prediction on the multi-scale feature maps from the Neck network are as follows: Divide the feature map into an S*S grid, and each grid predicts the center offset (∆x, ∆y) and width and height (w, h) of the target. Adopt the Anchor-free mechanism to avoid the limitation of predefined anchor boxes; Output (4 + 1 + C) values for each grid prediction; Preferably, the steps of outputting (4 + 1 + C) values for each grid prediction are as follows: Output 4: Bounding box coordinates, center point offset, and width and height, normalized to 01; Output 1: Confidence, that is, the probability of the existence of the target, activated through the sigmoid function; Output C: Class probability, that is, Softmax and Sigmoid multi-label classification; Furthermore, the steps of performing bounding box decoding are as follows: Convert the Anchor-free prediction values to actual coordinate values: Ximg = (σ(∆x) + gx) * s Yimg = (σ(∆y) + gy) * s Among them, gx and gy represent the coordinates of the upper left corner of the grid, s is the downsampling step of the feature map, and σ is the sigmoid function; Furthermore, the method for predicting whether each pixel within the bounding box belongs to the target through the Segmentation Head and generating a mask according to the prediction results is as follows: ROI extraction: Crop the corresponding region from the fused high-level feature map according to the predicted coordinate box of the detection head Mask generation: Generate K prototype masks through the convolutional layer of the segmentation head. The detection head additionally outputs a K-dimensional coefficient vector for each instance, and generates an instance-level mask by linearly combining the prototype masks: Maskinstance = ∑ coefficientk * ProtoMaskk (k = 1…K) Activation binarization: Map the mask to [0, 1] through the sigmoid activation function and set a threshold to generate a binary mask; Furthermore, the method for cropping the image according to the generated mask to obtain non-motor vehicle objects in the captured image is as follows: Mask post-processing: Crop the mask to the target area according to the finally retained bounding box, thereby obtaining the target non-motor vehicle object; The method of using the pre-trained convolutional neural network model RestNet-50 to extract features from the pictures captured by the area cameras respectively and construct three different feature databases is as follows: Use RestNet-50 to extract 512-dimensional feature vectors of vehicle images in different regions respectively, and construct feature databases for three regions.

[0022] Furthermore, the method of using RestNet-50 to extract 512-dimensional feature vectors of vehicle images in different regions respectively and construct three regional feature databases is as follows: Load the pre-trained RestNet-50 pre-trained model and remove the original fully connected layer of the model; The non-motor vehicle object images cropped by YOLOv8 need to be standardized, resized, and center-cropped in Resnt-50; Extract the feature vectors and normalize them; Furthermore, the method of extracting the feature vectors and normalizing them is as follows: Gradually extract features through convolutional layers and residual blocks; Compress the output of the last convolutional layer through global average pooling; Add a fully connected layer after the pooling layer to reduce the 2048-dimensional feature vector to a 512-dimensional feature vector; Normalize the 512-dimensional feature vector to obtain a normalized feature vector, which is convenient for subsequent similarity calculation; Calculate the feature similarities of the sample images in the left-turn waiting area, the left-turn necessary area, and the left-turn exit respectively, retain the image with the highest similarity, and establish a corresponding relationship database; Further, the steps of calculating the feature similarities of the sample images in the left-turn waiting area, the left-turn necessary area, and the left-turn exit respectively, retaining the image with the highest similarity, and establishing a corresponding relationship database are as follows: Calculate the cosine similarities of the sample images in the left-turn waiting area and the left-turn exit, and in the left-turn waiting area and the left-turn necessary area; Retain the group of images with the highest cosine similarity value, and establish a corresponding relationship database between them; Further, the specific method for calculating the cosine similarities of the sample images in the left-turn waiting area and the left-turn exit, and in the left-turn waiting area and the left-turn necessary area is as follows: According to the formula of cosine similarity, calculate the global feature similarity between feature maps: cosine similarity = A * B / (|A| * |B|), where A and B respectively represent the normalized 512-dimensional feature vectors of the feature maps, and |A| and |B| respectively represent the vector norms; Further, the specific method for retaining the group of images with the highest cosine similarity value and establishing a corresponding relationship database between them is as follows: Regard the group of images with the highest cosine similarity as the successfully matched group of images; Taking whether it appears in the three areas as the judgment criterion, and using 1 and 0 boolean values to represent "yes" and "no", for the images that are successfully matched in the left-turn waiting area and the left-turn necessary area, and the left-turn waiting area, set their values to 1, otherwise 0; Judge whether a vehicle makes an illegal left turn through the image correspondence relationship database; Further, the steps of judging whether a vehicle makes an illegal left turn through the image correspondence relationship database are as follows: For the image correspondence relationship database, when the values of the attribute columns of the left-turn waiting area, the left-turn necessary area, and the left-turn exit are all 1, the non-motor vehicle represented by the image tuple is making a normal left turn; While for the image tuple with the values of the attribute columns of the left-turn waiting area and the left-turn exit being 1 and the value of the left-turn necessary area being 0, the non-motor vehicle represented by it is making an illegal left turn; According to another example of the present invention, as Figure 2 shown, there is also provided a non-motor vehicle illegal left-turn discrimination system based on deep learning, which includes: a data collection module 1, a feature extraction module 2, a left-turn data module 3, an illegal left-turn determination module 4, a voice reminder module 5, and a visualization module 6; Data collection module 1, which consists of cameras installed in each area, is used to collect non-motor vehicle images in each area, crop and organize the non-motor vehicle images to form different databases; Feature extraction module 2, which extracts the features of the processed sample images and forms a 512-dimensional feature vector through convolutional neural network processing; Left-turn data module 3, which is used to calculate the similarity of sample images, establish a corresponding relationship database according to the similarity matching relationship, and store the corresponding relationship of non-motor vehicles; Violation left-turn determination module 4, which is used to determine the violation left-turn behavior of non-motor vehicles according to the values in the corresponding relationship database; Voice reminder module 5, which reminds non-motor vehicles not to make illegal left-turns in the non-motor vehicle left-turn waiting area and gives a second reminder to non-motor vehicles making illegal left-turns at the left-turn exit; Visualization module 6, which is used to reflect statistical data in the system, including the non-motor vehicle flow at intersections, the flow of non-motor vehicles making illegal left-turns, the number of voice reminders, the number of traffic accidents at intersections, and the comparison of the number of illegal left-turn behaviors of non-motor vehicles before and after using the system.

[0023] According to the example of the present invention, as Figure 3 shown, there is also provided a specific working flow chart of a method and system for discriminating illegal left-turns of non-motor vehicles at intersections based on deep learning, which mainly includes the following steps: S1. Define three areas at the intersection; It should be explained that the three areas defined at the intersection are the left-turn waiting area, the necessary left-turn area, and the left-turn exit. At the intersection, a left-turn waiting area may also serve as the left-turn exit in the other direction, but in this example, only the situation in the left-turn direction shown Figure 4 is discussed.

[0024] S2. Area cameras acquire non-motor vehicle images in different areas and establish area image databases; It should be explained that the images of non-motor vehicles passing through the three divided areas are collected by cameras installed at the positions of the three divided areas and stored in the image databases of the three areas respectively for subsequent use and processing.

[0025] S3. Use YOLOv8 object detection to crop the acquired images; It should be noted that image cropping is to more prominently display the features of the object to be extracted - non-motor vehicles, and avoid interference from the surrounding environment on subsequent feature extraction.

[0026] S4. Use a deep learning neural network model to extract image features; It should be noted that the deep learning neural network model adopted here is RestNet-50. By performing convolution, pooling, and fully connected operations on the image features, the most important features that can represent the sample image are obtained.

[0027] S5. Calculate the feature similarity of the sample images in different regional databases; It should be noted that calculating the feature similarity between different images mainly involves calculating the feature similarity of the images taken at the left-turn waiting area and the left-turn exit, and between the left-turn waiting area and the necessary left-turn area, laying a foundation for subsequent establishment of corresponding relationships and determination of illegal left turns.

[0028] S6. Perform image matching according to the similarity and establish a corresponding relationship database; It should be noted that the values in the corresponding relationship database are mainly represented by 0 and 1 Boolean values, where 1 indicates that a non-motor vehicle has passed through this area, and 0 indicates that it has not. When the images taken at the left-turn waiting area and the left-turn exit of the non-motor vehicle are successfully matched, and the non-motor vehicle has appeared in both areas, the values of the corresponding left-turn waiting area and left-turn exit attribute columns in the database are both 1.

[0029] S7. Determine whether the non-motor vehicle makes a left turn; It should be noted that determining whether the non-motor vehicle makes a left turn is mainly judged by the values of each area stored in the established database. For all vehicles in the left-turn waiting area, the value of the left-turn waiting area attribute column is 1. If a left-turn behavior occurs, the value of the left-turn exit attribute column is 1; if no left-turn behavior occurs, the value of the left-turn exit attribute column is 0.

[0030] S8. Determine whether the non-motor vehicle violates the regulations; It should be noted that determining whether the non-motor vehicle makes an illegal left turn is based on determining whether the non-motor vehicle has a left-turn behavior. On this basis, if the value in the database satisfies that the value of the necessary left-turn area attribute column is 1, it means that the non-motor vehicle follows the traffic rules and makes a normal left turn; if the value of the necessary left-turn area attribute column is 0, it means that the non-motor vehicle has an illegal left-turn behavior.

[0031] S9. Voice reminder of the illegal behavior of non-motor vehicles; For the vehicles that have made illegal left turns mentioned above, the system will perform user profiling and record them. When they pass through the left-turn exit, the voice reminder device will receive the system instruction and remind them of their illegal behavior.

[0032] S10. The system visual interface displays the results.

[0033] It should be noted that the system visualization interface is necessary, which can improve the interaction experience between the system and users. At the same time, through the statistical data of the system, the traffic flow at intersections can be analyzed. Most importantly, the number of non-motor vehicle violations and the number of traffic accidents at intersections before and after using the methods and systems provided by the present invention can be compared to obtain the effectiveness of the methods and systems provided by the present invention.

[0034] According to the example of the present invention, as Figure 4 shown, a schematic diagram of intersection area division and illegal left-turn behavior determination for the non-motor vehicle illegal left-turn discrimination method based on deep learning at intersections is also provided. In this schematic diagram, the possible position areas A, B, C, and D at the intersection, the left-turn waiting area, the necessary left-turn area, the left-turn exit, the installation position of the voice reminder device, and the possible routes for normal left-turn and illegal left-turn of non-motor vehicles are marked.

[0035] Specifically, for better understanding by those skilled in the art, the technical terms or some nouns that the present application may involve are now explained: The YOLOv8 object detection model in the example of the present invention is the latest breakthrough in the YOLO series of object detection models. Compared with YOLOv5, it is more suitable for real-time object detection such as unmanned driving. When performing detection output, YOLOv8 returns a more structured Results object, directly providing xy-format coordinates, while YOLOv5 needs to obtain xy coordinates through intermediate conversion.

[0036] The RestNet-50 model in the example of the present invention is a convolutional neural network in the field of deep learning, which belongs to residual learning at the core and performs excellently in image classification, object detection, and image segmentation.

[0037] In summary, by means of the above technical solutions of the present invention, the present invention divides the left-turn waiting area, the necessary left-turn area, and the left-turn exit at the intersection, and obtains non-motor vehicle images passing through the three areas through cameras set in the three areas, and establishes corresponding regional image databases. The YOLOv8 object detection model is used to crop the sample images in the database, and then the most core features that can represent each image are extracted through operations such as convolution and pooling of the RestNet-50 model. By calculating the feature similarity between the images in different regional image databases, the matching relationship between the images is explored, and an image correspondence database with the left-turn waiting area, the necessary left-turn area, and the left-turn exit as attribute columns, each non-motor vehicle as a tuple, and the value as a boolean value 0, 1 is established, and whether a non-motor vehicle has an illegal behavior is determined according to the values in this database.

[0038] Meanwhile, the present invention proposes a system that integrates a voice reminder module. The system will create user profiles and record non-motor vehicles that violate the left-turn rule, remind them of the violation at the left-turn exit, and detect whether the vehicle observes traffic rules during subsequent driving. Moreover, the system provided by the present invention has a visual interface that can count the non-motor vehicle driving data at intersections, thus facilitating the quantification of the implementation effect of the present invention.

Claims

1. A discriminant method for non-motor vehicle illegal left-turn at intersections based on deep learning, characterized in that It includes the following steps: Divide the area where non-motor vehicles appear at the intersection, and mark the left-turn waiting area, the necessary area for left-turn, and the left-turn exit within this area; Capture pictures of non-motor vehicles passing through this area by the cameras installed in the divided area, and perform data preprocessing; Use the pre-trained convolutional neural network model RestNet-50 to extract features from the pictures captured by the area cameras respectively, and construct three different feature databases; Calculate the feature similarity of the sample images in the left-turn waiting area and the necessary area for left-turn and the left-turn exit respectively, retain the pictures with the highest similarity, and establish a corresponding relationship database; Judge whether the vehicle makes an illegal left turn through the corresponding relationship database between pictures; 2. The method for discriminating illegal left-turn of non-motor vehicles at intersections based on deep learning according to claim 1, characterized in that, Dividing the area where non-motor vehicles appear at the intersection and marking the left-turn waiting area, the necessary area for left-turn and the left-turn exit within this area includes: Taking the area extending around from the intersection point of the zebra crossing as the area where non-motor vehicles appear at the intersection, including: A circular area covered by a circle with the intersection point of the zebra crossing as the center and a radius of 1m; Mark the possible position areas as A, B, C, D respectively, and correspond A, B, C to the left-turn waiting area, the necessary area for left-turn, and the left-turn exit respectively; 3. The method for discriminating illegal left turns of non-motor vehicles at intersections based on deep learning according to claim 1, wherein, Capture pictures of non-motor vehicles passing through this area by the cameras installed in the divided area, and perform data preprocessing. The steps are as follows: Capture the images of non-motor vehicles passing through this area by the area cameras, and sort out the captured images, grouping them according to the area, including: classifying the pictures taken from different areas into the groups of this area; the groups are the left-turn waiting area, the necessary area for left-turn, and the left-turn exit respectively; Use the object detection model YOLOv8 to preprocess each group of images and crop out the images of non-motor vehicles; 4. The method for discriminating illegal left turns of non-motor vehicles at intersections based on deep learning according to claim 2, characterized in that Use the object detection model YOLOv8 to preprocess each group of images and crop out the images of non-motor vehicles. The steps are as follows: Locate the target bounding box through the detection head; Predict whether each pixel within the bounding box belongs to the target through the segmentation head, and generate a mask according to the prediction result; Crop the image according to the generated mask to obtain the non-motor vehicle object in the captured image, including: mask post-processing: crop the mask to the target area according to the finally retained bounding box, so as to obtain the target non-motor vehicle object; 5. The method for discriminating illegal left turns of non-motor vehicles at intersections based on deep learning according to claim 4, characterized in that, Locate the target bounding box through the detection head. The steps are as follows: Divide and perform Anchor-free prediction on the multi-scale feature maps from the Neck network, including the following steps: Divide the feature map into a grid of S*S, and each grid predicts the center offset (∆x, ∆y) and width and height (w, h) of the target. Adopt the Anchor-free mechanism to avoid the limitation of predefined anchor boxes; Output (4+1+C) values for each grid prediction. The steps are as follows: Output 4: Bounding box coordinates, center point offset and width and height, normalized to 01; Output 1: Confidence, that is, the probability of the existence of the target, activated by the sigmoid function; Output C: Class probability, that is, Softmax and Sigmoid multi-label classification; Perform bounding box decoding, including the following steps: Convert the Anchor-free prediction values into actual coordinate values: Ximg = (σ(∆x) + gx) * s, Yimg=(σ(Δy)+gy)*s, Among them, gx, gy represent the coordinates of the upper left corner of the grid, s is the feature map downsampling step, and σ is the sigmoid function.

6. The method for discriminating illegal left turns of non-motor vehicles at intersections based on deep learning according to claim 4, wherein The segmentation head predicts whether each pixel in the bounding box belongs to the object and generates a mask based on the prediction results. The steps are as follows: ROI extraction: Crop the corresponding area from the fused high-level feature map according to the predicted coordinate frame of the detection head; Mask generation: K prototype masks are generated through the convolutional layer of the segmentation head. The detection head additionally outputs a K-dimensional coefficient vector for each instance, and the instance-level mask is generated by linearly combining the prototype masks: Maskinstance=∑coefficientk*ProtoMaskk (k=1…K), Activation binarization: The mask is mapped to [0, 1] through the sigmoid activation function, and the threshold is set to generate a binary mask.

7. The method for discriminating illegal left turns of non-motor vehicles at intersections based on deep learning according to claim 1, characterized in that, Use the pre-trained convolutional neural network model RestNet-50 to extract features from the images captured by the regional cameras and build three different feature databases. The steps are as follows: RestNet-50 is used to extract 512-dimensional feature vectors of vehicle images in different regions, and feature databases of three regions are constructed. RestNet-50 is used to extract 512-dimensional feature vectors of vehicle images in different regions, and feature databases of three regions are constructed. The steps are as follows: Load the pre-trained RestNet-50 model and remove the original fully connected layer of the model; For non-motor vehicle object images cropped by YOLOv8, they need to be standardized, resized, and center-cropped in Resnt-50; Extract feature vectors and normalize them, including: Features are gradually extracted through convolutional layers and residual blocks; Compress the output of the last convolutional layer through global average pooling; A fully connected layer is added after the pooling layer to reduce the 2048-dimensional feature vector to a 512-dimensional feature vector; The 512-dimensional feature vector is normalized to obtain a normalized feature vector for subsequent similarity calculation.

8. The method for discriminating illegal left turns of non-motor vehicles at intersections based on deep learning according to claim 1, wherein Calculate the feature similarity of sample images in the left-turn waiting area, the left-turn must-go area, and the left-turn exit, retain the image with the highest similarity, and establish a corresponding relationship database. The steps are as follows: Calculate the cosine similarity of the sample images of the left-turn waiting area and the left-turn exit, and the left-turn waiting area and the left-turn must-go area, including: According to the formula of cosine similarity, the global feature similarity between feature maps is calculated: Cosine similarity = A*B / (|A|*|B|), Where A and B represent the normalized 512-dimensional feature vectors of the feature graph, and |A| and |B| represent the modulus length of the vector respectively; The image groups with the highest cosine similarity values are retained, and a corresponding database between them is established, including: The image group with the highest cosine similarity is regarded as the image group with successful matching; The criterion is whether the image appears in the three areas, with 1 and 0 Boolean values representing "yes" and "no". For images that successfully match the left-turn waiting area with the left-turn must-pass area and the left-turn waiting area, the value is set to 1, otherwise it is 0.

9. The method for discriminating illegal left-turn of non-motor vehicles at intersections based on deep learning according to claim 1, characterized in that Through the database of correspondence between pictures, we can determine whether the vehicle has made an illegal left turn. The steps are as follows: For the image correspondence database, when the values of the attribute columns left-turn waiting area, left-turn must-pass area, and left-turn exit are all 1, the non-motorized vehicle represented by the image tuple is turning left normally; The non-motorized vehicle represented by the image tuple whose attribute columns have values of 1 for the left-turn waiting area and the left-turn exit and value of 0 for the left-turn must-pass area is making an illegal left turn.

10. A system for implementing the non-motor vehicle illegal left-turn discrimination method for deep learning according to any one of claims 1 to 9, characterized in that, The system includes: data collection module, feature extraction module, left turn data module, illegal left turn determination module, voice reminder module and visualization module; The data collection module is composed of cameras installed in various areas, which are used to collect non-motor vehicle images in various areas, and to crop and organize the non-motor vehicle images to form different databases; The feature extraction module extracts the features of the processed sample image and forms a 512-dimensional feature vector through convolutional neural network processing; The left-turn data module is used to calculate the similarity of sample images, and to establish a corresponding relationship database based on the similarity matching relationship to store the corresponding relationship of non-motor vehicles; An illegal left turn determination module is used to determine the illegal left turn behavior of non-motor vehicles according to the values in the corresponding relationship database; The voice reminder module reminds non-motor vehicles not to make illegal left turns in the non-motor vehicle left turn waiting area, and gives a second reminder to non-motor vehicles that make illegal left turns at the left turn exit; The visualization module is used to reflect statistical data in the system, including the flow of non-motor vehicles at intersections, the flow of non-motor vehicles making illegal left turns, the number of voice reminders, the number of traffic accidents at intersections, and the comparison of the number of illegal left turns of non-motor vehicles before and after using the system.

Images