A traffic light perception method and system for assisting vehicle driving

Abstract

The present invention discloses a method and system for identifying traffic lights, belonging to the technical field of vehicle assisted driving. Specifically: annotate the images of traffic lights with directions to construct a dataset of traffic lights with directions. A direction perception model based on ResNet101 is proposed, and the annotated traffic light dataset is trained using this model to obtain a network model that can identify traffic lights with directions. This model can effectively solve the problems of low recognition accuracy for small traffic light targets and direction recognition; input the trained model into an intelligent perception system for assisted driving vehicles proposed in the present invention. This system can be deployed to terminal devices to judge the color and direction of traffic lights and give voice reminders. The present invention proposes a complete set of recognition methods and systems for detecting traffic lights, which can not only identify traffic lights, but also identify the directions of traffic lights, and through voice reminders, can effectively avoid potential safety hazards caused by driver negligence.

Description

Technical Field

[0001] The invention relates to a method and system for identifying traffic lights, belonging to the technical field of auxiliary vehicle driving. Background Art

[0002] With the advancement of technology and the economy, transportation is becoming increasingly convenient, and car use is becoming more widespread. However, this also presents a challenge: the increasing number of cars has led to negative consequences such as deteriorating road conditions and frequent traffic accidents. Assisted driving can help drivers pay more attention to road conditions and effectively maintain traffic safety.

[0003] Traffic light recognition is a key focus in assisted driving. Current methods primarily focus on identifying traffic light colors. With urban development, traffic is becoming increasingly complex. Traffic lights are no longer limited to red lights, but now also include red lights for left turns, red lights for right turns, red lights for U-turns, and green lights for left turns, right turns, and U-turns. With these complex traffic light directional indicators, even the slightest inattention can lead to red light violations and even accidents. To address this issue, the present invention proposes a method and system for identifying traffic light directional indicators. Summary of the Invention

[0004] The technical problem solved by the present invention is to provide a method and system for identifying traffic direction traffic lights.

[0005] Because traffic lights are small, some images captured by actual vehicle-mounted sensing devices show only 10×10 pixels in size and have directional characteristics, which falls under the category of small target detection. This paper proposes a direction-aware target detection method based on the ResNet101 network, which can effectively identify small traffic light targets with directional characteristics. The specific technical solution is as follows:

[0006] As a first aspect, the present invention provides a traffic light perception method for assisting vehicle driving, the method steps being as follows:

[0007] Step 1: Obtain various traffic light image frame data with directions from the vehicle-mounted video data as image data frames to be labeled;

[0008] Step 2: Label the traffic light image frame data obtained in various scenarios. The labeled traffic light image frame data is divided into nine categories according to actual traffic rules, namely: left turn at red light, go straight at red light, right turn at red light, U-turn at red light, left turn at green light, go straight at green light, right turn at green light, U-turn at green light, and yellow light.

[0009] Step 3: Based on ResNet101, construct a feature fusion with different levels of depth.

[0010] Step 4: For the features fused in Step 2 of the present invention, use a one-stage YOLO object detection framework to input the data set belonging to Step 2 of the present invention into the one-stage object detection framework for network training. After iterating more than 200 times, a trained network model is obtained.

[0011] As a second aspect, the present invention provides a perception system for an assisted driving vehicle. This system is installed on a terminal device, which includes a mobile phone, and is not limited to terminal devices such as mobile phones. In the present invention, this system is called an assisted driving vehicle intelligent perception system (hereinafter referred to as the intelligent perception system). The intelligent perception system includes: network model deployment, which obtains traffic light images in real time and outputs traffic light information and direction information lights.

[0012] The network model deployment: The intelligent perception system in the present invention deploys the network model trained in Step 4 to the terminal device, which includes a mobile phone and is not limited to terminal devices such as mobile phones.

[0013] Fix the mobile phone with the intelligent perception system of the present invention at the front end of the vehicle and adjust the angle to be able to capture the traffic lights. Then turn on the intelligent perception system on the mobile phone or terminal device.

[0014] After the intelligent perception system is turned on, it will call the navigation interface of Amap (or other map navigation). When it senses a traffic light signal ahead through the navigation system, it will call the mobile phone camera interface to capture the image of the current traffic light signal in real time and input it into the network model calculation system. The network model calculation system outputs the color and direction of the signal, and the intelligent perception system will play it to the driver through voice to remind the driver to drive safely, greatly improving the traffic safety factor. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The following further describes the present invention with reference to the drawings.

[0016] Figure 1 is the overall flowchart of the method of the present invention, and Table 1 is the naming of the labels in the present invention.

[0017] Figure 2 is the illustration of the directional traffic lights marked in the present invention.

[0018] Figure 3 is the schematic diagram of the network with deep and shallow feature fusion constructed by the method described in Example 1;

[0019] Figure 4 (a) is the traffic picture to be detected;

[0020] Figure 4 (b) is Figure 4 (a) the schematic diagram of the experimental results of the traffic picture;

[0021] Figure 4 (c) is the traffic picture to be detected;

[0022] Figure 4 (d) is Figure 4 (d) Schematic diagram of the experimental result of the traffic picture in (c); Specific implementation mode

[0023] The technical solution of the present invention will be further specifically described below through embodiments in combination with the accompanying drawings. Obviously, the described embodiments are only partial embodiments of the present invention. The present invention uses the camera of a mobile phone or a terminal device to identify traffic lights based on the object detection of the ResNet101 network. First, there is a traffic light data set in the present invention. This data set has approximately more than 30,000 traffic light pictures. The present invention no longer uses the traditional three types of labels of red lights, green lights, and yellow lights, but uses nine types of labels. As shown in Table 1.

[0024] The detailed steps of the present invention are as follows:

[0025] Step 1: First, more than 30,000 traffic light pictures are labeled in the present invention. The labels are as shown in Table 1. The position is labeled by the horizontal and vertical coordinates of the upper left corner and the horizontal and vertical coordinates of the lower right corner. As shown in the traffic light annotation box in Figure 2 , the remote sensing image set with the labeled targets is input into the object detection framework based on the ResNet101 network described in the present invention for training.

[0026] The object detection framework based on the ResNet101 network described in the present invention is as shown in Figure 3 and is specifically described in detail as follows:

[0027] First, the features of pool1 are extracted from ResNet101 in the present invention. Assume that the feature M obtained by poo11 has a dimension of m×m×d. Among them, M l represents a certain feature matrix among them, represents the i-th eigenvalue in the feature matrix M l , 1≤x≤m, 1≤y≤m, represents M l The mean value of the eigenvalues of the matrix, which can be expressed as

[0028]

[0029] For M, the set of average feature values of each channel can be expressed as Calculate The sum in the set, denoted as MAP, then MAP can be expressed as

[0030]

[0031] Here, it is necessary to find such a channel in M: Finally, obtain the channel C = {c1, c2, c3,..., c η}, take the first 3 channels, and the feature set of this channel is denoted as: Let dim = m×m×3, then, flatten MAP aim into a feature vector with a dimension of 1dim, denoted as F shallow .

[0032] Based on the ResNet101 network, in the present invention, the last layer of features in this network is obtained, denoted as F deep , and the size of its features is 1×LD. The F s h allow features with a size of 1dim obtained in the present invention are dimensionally reduced by PCA, and the dimension after dimensional reduction is 1×LD. The features after PCA are represented as:

[0033] The features F deep obtained in the present invention and those in the present invention are spliced and fused in a linear vector in a way from deep to shallow, represented as Then the fused vector F aggreage has a dimension of 1×(LD + LD). At this time, PCA dimensional reduction is performed again to reduce the dimension of F aggreage to 1×LD, and the features after dimensional reduction are represented as F Last . Thus, the fusion of deep and shallow features in the traffic light recognition process is completed.

[0034] Combine the above process with the YOLO detection framework to form a traffic light recognition and detection network. After multiple training iterations on more than 30,000 pictures with traffic light annotations, it is trained into a traffic light detection model for direction perception.

[0035] Step 2: Deploy and export the traffic light detection network model obtained in Step 1 into the ONNX Runtime format for running on mobile phones or terminal mobile devices.

[0036] Step 3, the perception system of the assisted driving vehicle is responsible for loading the network file generated in the ONNX Runtime mode on the mobile phone or terminal device. The loading process includes parsing the model file, creating a model object and preparing for inference.

[0037] Step 4: When the driver is driving, clip the in-vehicle mobile phone holder in front of the driver's seat position and activate the intelligent perception system, which will turn on the mobile phone navigation system.

[0038] Step 5: The intelligent perception system calls the navigation system interface. When the system receives a reminder of a traffic light signal ahead, at this time, the perception system of the assisted driving vehicle calls the camera module interface of the mobile phone or terminal device to take pictures and obtain the traffic light video in real time.

[0039] Step 6: The intelligent perception system inputs the traffic light video into the network model calculation system for inference and feeds the inference result back to the intelligent perception system.

[0040] Step 7: The intelligent perception system calls the voice device of the mobile phone or terminal to issue a voice reminder to remind and assist the driver to drive safely.

[0041] It should be understood that the above embodiments are only used to illustrate the present invention and not to limit the scope of the present invention. In addition, it should be understood that after reading the content of the present invention, those skilled in the art can make various changes and modifications to the present invention, and these equivalents also fall within the scope defined by the appended claims of this application.

[0042]

[0043] Table 1

Claims

1. A traffic light perception method and system for assisting vehicle driving, characterized in that, It includes the following steps: S1: Annotate multiple pictures of traffic lights, and input the remote sensing image set with the annotated targets into the object detection framework based on the ResNet101 network for training; Based on the ResNet101 network, the last layer features in the network are obtained and denoted as , whose feature size is 1×LD; based on the ResNet101 network, the shallow layer features in the network are obtained and denoted as , whose feature size is . PCA is used for dimensionality reduction, and the dimension after dimensionality reduction is 1×LD. The feature representation after PCA is: ; The features and are linearly concatenated and fused in a way from deep to shallow, denoted as . Then the dimension of the fused vector is . At this time, PCA dimensionality reduction is performed again to reduce the dimension of to 1×LD, and the reduced features are denoted as . Thus, the fusion of deep and shallow features in the traffic light recognition process is completed; Combine the above process with the YOLO detection framework to form a traffic light recognition and detection network; After multiple training iterations of multiple pictures with traffic light annotations, train it into a traffic light detection network model with direction perception; S2: Deploy and export the traffic light detection network model obtained in S1 into the ONNX Runtime format for running on mobile phones or terminal mobile devices; S3: The perception system of the assisted driving vehicle is responsible for loading the network file generated in the ONNX Runtime mode on the mobile phone or terminal device. The loading process includes parsing the model file, creating a model object and preparing for inference; S4: When the driver is driving, clamp the in-vehicle mobile phone holder in front of the driver's seat position and start the intelligent perception system. The intelligent perception system will turn on the mobile phone navigation system; S5: The intelligent perception system calls the navigation system interface. When the system receives a reminder that there is a traffic light signal ahead, at this time, the perception system of the assisted driving vehicle calls the camera module interface of the mobile phone or terminal device to take pictures and obtain the traffic light video in real time; S6: The intelligent perception system inputs the traffic light video into the network model calculation system network for inference and feeds the inference result back to the intelligent perception system; S7: The intelligent perception system calls the voice device of the mobile phone or terminal to issue a voice reminder to remind and assist the driver to drive safely.

2. The traffic light perception method and system for assisting vehicle driving according to claim 1, wherein, The object detection framework based on the ResNet101 network described in S1 specifically includes: First, extract the features of pool1 from the ResNet101 network. Assume the features obtained by pool1 are M, and the dimension of M is dimensions, where is used to represent a certain feature matrix among them, represents the feature matrix in the th eigenvalue, , , represents the mean value of the eigenvalues of the matrix, which is expressed as ， For M, the set of feature averages for each channel is denoted as , calculate the sum in the set, denoted as , then is denoted as ， Here, it is necessary to find such a channel in M: , and finally obtain the channel closest to the average value . Take the first 3 channels, and the feature set of this channel is denoted as: . Let . Then, is flattened into a feature vector as , denoted as .

3. A traffic light perception method and system for assisting vehicle driving according to claim 1, characterized in that, The annotation described in S1, and the labels of the annotation are as follows: 。

Images