A traffic light detection and recognition method based on visual images

Abstract

The application discloses a traffic light detection and recognition method based on visual images, realizes recognition of the traffic light based on a visual algorithm, combines two heat maps generated by a deep learning network model, and fuses inference results of the two heat maps, so that the method does not need to install communication equipment or depend on a high-precision map, effectively improves robustness of the deep learning network model, reduces a missed detection rate and a false detection rate of the model, and can improve a detection distance of the model to a certain extent, and has good generalization performance.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent driving, and particularly relates to a traffic light detection and recognition method based on visual images. BACKGROUND

[0002] In recent years, automatic driving technology has developed rapidly, and the detection and recognition of traffic lights is an indispensable part. At present, the mainstream traffic light recognition methods can be divided into two categories: a v2x-based recognition method and a visual image recognition method.

[0003] The v2x-based recognition method is realized through signal transmission. The traffic light sends a signal state in real time, and the vehicle recognizes the traffic light by analyzing the received signal. This method is relatively safe and responds quickly, but it requires that the traffic light and the vehicle end are both equipped with communication equipment, and the dependence on communication equipment is too high. Therefore, the popularization degree in the actual application scene is low, and the application cost is high.

[0004] The visual image recognition method mainly refers to acquiring images through a camera and detecting and recognizing the traffic light on the image through an algorithm. At present, the visual image recognition method can be divided into two types. One method first acquires a region of interest through a high-precision map, and then detects and recognizes the target region. The other method directly detects and recognizes the traffic light on the entire image through an algorithm. The high-precision map-based method can make the vehicle focus better on the traffic light it needs to identify, and reduce missed detection and false detection. However, this method must be supported by a high-precision map, and relies too much on map information. In addition, the construction cost of the high-precision map is high and quite complex. The visual algorithm-based recognition method can only detect ordinary traffic lights, that is, it can only recognize the color of the traffic light, and cannot recognize the case that multiple lights in a light frame are on. SUMMARY

[0005] In view of the above problems, the main purpose of the present application is to design a traffic light detection and recognition method based on visual images, which realizes the recognition of traffic lights based on visual algorithms, and solves the problem of recognizing the color of traffic lights.

[0006] In order to achieve the above purpose, the present application adopts the following technical scheme:

[0007] A traffic light detection and recognition method based on visual images, comprising the construction of a deep learning network model and the detection and recognition based on the deep learning network model, specifically comprising the following steps:

[0008] S1: constructing and training a deep learning network model based on a detection target;

[0009] The construction of the deep learning network model comprises:

[0010] S101: a deep learning network generates a heat map, and a peak value on the heat map corresponds to a center point of a detection target;

[0011] S102: after obtaining the center point of the detection target, a deep learning network is used to regress coordinates of the detection target and a category thereof;

[0012] S103: a traffic light information is output by a deep learning network model;

[0013] S2: a 2D image related to a traffic light of a preset road section is acquired;

[0014] S3: the acquired 2D image is sent to a deep learning network model that is built and trained to perform inference;

[0015] S4: coordinates of the traffic light in the 2D image and a category thereof are output.

[0016] As a further description of the present application, the coordinates are coordinate positions of a traffic light outer frame in the 2D image; and the category refers to a category of a light that is on in the traffic light outputting the coordinate position, and specifically includes color and shape information of the light.

[0017] As a further description of the present application, the shape information includes one or more or all of a round light, a digital light and an arrow light.

[0018] As a further description of the present application, the detection target includes the traffic light outer frame and the light that is on in the traffic light outer frame.

[0019] As a further description of the present application, in S101, the heat map generated by the deep learning network includes two heat maps; a peak value on one heat map corresponds to a center point of a traffic light outer frame, and a peak value on the other heat map corresponds to a center point of a light that is on in the traffic light outer frame.

[0020] As a further description of the present application, the heat map of the traffic light outer frame is generated by using an elliptical Gaussian kernel function, the heat map of the light that is on in the traffic light outer frame is generated by using a circular Gaussian kernel function, and a radius of the circular Gaussian kernel is enlarged.

[0021] As a further description of the present application, in S102, after obtaining the center point of the detection target as the traffic light outer frame, a deep learning network is used to regress a width and a height of the frame and category information of the light that is on in the frame;

[0022] After obtaining the center point of the detection target as the light that is on in the traffic light outer frame, a deep learning network is used to regress a width and a height of the light and a color and a specific shape thereof.

[0023] As a further description of the present application, in S103, the traffic light information output by the deep learning network model is the fusion result of two heat maps.

[0024] As a further description of the present application, the target point of the detection center obtained from the two heat maps includes three forms, which are: only obtaining the outer light frame of the traffic light, only obtaining the light on in the outer light frame of the traffic light, and simultaneously obtaining the outer light frame of the traffic light and the light on in the outer light frame of the traffic light.

[0025] As a further description of the present application, based on the three forms of the target point of the detection center obtained, the output of the fusion result of the two heat maps is:

[0026] Only obtaining the outer light frame of the traffic light, the output is the outer light frame of the traffic light and the color and default category of the light on in the outer light frame of the traffic light;

[0027] Only obtaining the light on in the outer light frame of the traffic light, the output is the position and color of the light on in the outer light frame of the traffic light;

[0028] Simultaneously obtaining the outer light frame of the traffic light and the light on in the outer light frame of the traffic light, the output is the position of the outer light frame of the traffic light and the color of the light on in the outer light frame of the traffic light.

[0029] Compared with the prior art, the technical effect of the present application is:

[0030] The present application provides a traffic light detection and recognition method based on visual images, which realizes the recognition of traffic lights based on visual algorithms, combines two heat maps generated by a deep learning network model, and fuses the inference results of the two heat maps, without the need to install communication equipment or rely on high-precision maps, effectively improving the robustness of the deep learning network model, reducing the missed detection and false detection rate of the model, and to a certain extent, improving the detection distance of the model, and having good generalization performance. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 It is a recognition flowchart based on a deep learning network model of the present application;

[0032] Figure 2 It is a fusion mode diagram in the present application;

[0033] Figure 3 It is an output traffic light information example diagram of the present application. DETAILED DESCRIPTION

[0034] The present application will be described in detail below with reference to the accompanying drawings:

[0035] A traffic light detection and recognition method based on visual images, referring to Figures 1-3As shown, the building of the deep learning network model and the detection and identification based on the deep learning network model, specifically including the following steps:

[0036] S1: building and training a deep learning network model based on the detection target;

[0037] The building of the deep learning network model includes:

[0038] S101: generating a heat map by the deep learning network, and the peak value on the heat map corresponds to the center point of the detection target;

[0039] S102: after obtaining the center point of the detection target, using the deep learning network to regress the coordinates and category of the detection target;

[0040] S103: outputting the traffic light information by the deep learning network model;

[0041] S2: obtaining a 2D image of the traffic light related to a preset road section;

[0042] S3: sending the obtained 2D image into the built and trained deep learning network model for inference;

[0043] S4: outputting the coordinates and category of the traffic light in the 2D image.

[0044] Specifically, the coordinates and category involved in the present embodiment are set as follows:

[0045] Coordinates: the coordinate position of the external light frame (large light) of the traffic light in the 2D image, such as Figure 3 (Xmin, Ymin) and (Xmax, Ymax) in the above formula;

[0046] Category: refers to the category of the light (small light) that is on in the output coordinate position, specifically including the color and shape information of the light;

[0047] Shape information, including one or more or all of round light, digital light, and arrow light.

[0048] The above specific small light information can help the decision-making and planning module in the field of intelligent driving to further develop.

[0049] More specifically, the above detection target includes the external light frame of the traffic light and the light that is on in the external light frame of the traffic light.

[0050] Based on the above disclosed steps, the building of the deep learning network model is analyzed in detail, and the analysis content is as follows:

[0051] In S101, the heat map generated by the deep learning network includes two heat maps; the peak value on one heat map corresponds to the center point of the outer light frame of the traffic light, and the peak value on the other heat map corresponds to the center point of the light on in the outer light frame of the traffic light;

[0052] The heat map of the outer light frame of the traffic light is generated by using an elliptical Gaussian kernel function, and the heat map of the light on in the outer light frame of the traffic light is generated by using a circular Gaussian kernel function, and the radius of the circular Gaussian kernel is expanded.

[0053] In S102, after obtaining the target point of the detection center as the outer light frame of the traffic light, the width and height of the light frame and the category information of the light on in the light frame are regressed by using the deep learning network; and after obtaining the target point of the detection center as the light on in the outer light frame of the traffic light, the width and height of the light, the color and the specific shape of the light are regressed by using the deep learning network.

[0054] In S103, the traffic light information output by the deep learning network model is the fusion result of the two heat maps.

[0055] Specifically, the target point of the detection center obtained by the two heat maps includes three forms, which are: only obtaining the outer light frame of the traffic light, only obtaining the light on in the outer light frame of the traffic light, and simultaneously obtaining the outer light frame of the traffic light and the light on in the outer light frame of the traffic light.

[0056] Based on the three forms of the target point of the detection center obtained, the output of the fusion result of the two heat maps is respectively:

[0057] Only the outer light frame of the traffic light is obtained, and the output is the outer light frame of the traffic light and the color and default category of the light on in the outer light frame of the traffic light (since the proportion of the circular light is higher in actual application, the default category in this embodiment is the circular light);

[0058] Only the light on in the outer light frame of the traffic light is obtained, and the output is the position and color of the light on in the outer light frame of the traffic light;

[0059] The outer light frame of the traffic light and the light on in the outer light frame of the traffic light are simultaneously obtained, and the output is the position of the outer light frame of the traffic light and the color of the light on in the outer light frame of the traffic light.

[0060] It should also be noted that, in this embodiment, the deep learning network model uses different heat maps for the external light frame of the traffic light and the lights lit in the external light frame of the traffic light, respectively. At the same time, in order to further improve the detection distance of the model, different Gaussian kernel functions are used to generate heat maps in view of the differences in the contours of the external light frame of the traffic light and the lights lit in the external light frame of the traffic light. In view of the fact that most of the external light frames of the traffic lights are rectangular, an elliptical Gaussian kernel function is used for the heat map of the external light frame of the traffic light, and the lights lit in the external light frame of the traffic light are basically circular, so a circular Gaussian kernel function is used. However, since the volume of the lights lit in the external light frame of the traffic light is too small, the heat map cannot have a good distribution effect when the original radius is used, so the original radius is expanded to 2 times. This method can effectively improve the detection distance of the deep learning network model and reduce the missed detection rate and false detection rate of the model.

[0061] In the present invention, the construction and training of the deep learning network model is an important part of the specific embodiment of the present invention. The present invention adopts a detection network based on the deep learning network, and on this basis, improves the deep learning network according to the characteristics of traffic lights, so that it has better performance in traffic light detection.

[0062] Specifically, based on the above disclosed content, first, the deep learning network generates a heat map, and the peak on the heat map corresponds to the center point of the detection target. After obtaining the center point of the target, the network is used to regress the width and height of the plane frame of the target and its category. Since the traffic light occupies a smaller image area than the entire image, it belongs to the detection of small targets. In order to improve the detection distance and robustness, in the present invention, the deep learning network generates two heat maps: the peak on one heat map corresponds to the center point of the light frame outside the traffic light, and regresses the width and height of the light frame and the color of the light therein, but does not identify the specific shape of the light; the peak on the other heat map corresponds to the center point of the light in the light frame outside the traffic light, and regresses the width and height of the light as well as its color and specific shape. The traffic light information output by the final network is the fusion of the two heat map reasoning results. Since the deep learning network model has the possibility of missed detection and false detection, different fusion schemes need to be adopted for different reasoning results. The specific fusion scheme is as follows. Figure 2 As shown, for the sake of ease of understanding and explanation, the headlights in the figure refer to the outer light frame of the traffic light, and the small lights refer to the lights in the outer light frame of the traffic light.

[0063] In this embodiment, a large number of cameras are used to collect and annotate 2D images as a training set to train the deep learning network model. The weight parameters of the model are updated using the loss function so that the deep learning network model can learn the characteristics of traffic lights. The loss function used in this embodiment includes three parts: the total loss function L of the heat map k , target length and width loss function L sizeand the loss function L of the target center point offset off .

[0064] The total loss function L det is as follows:

[0065] L det = L k + lambda off L off + lambda size L size

[0066] Wherein, lambda off = 1, lambda size = 0.5, respectively represent the weight of the corresponding loss function.

[0067] Specifically, each loss function is as follows:

[0068]

[0069] Wherein, alpha and beta are hyperparameters, N represents the number of image key points, represents the predicted value.

[0070]

[0071] Wherein, p represents the center point of the target frame, R represents the down-sampling multiple 4, represents the deviation value, represents the bias value of the network output.

[0072]

[0073] Wherein, represents the target length and width result of the network output, s k represents the actual length and width size of the target.

[0074] The traffic light detection and recognition method based on visual images disclosed by the application has lower cost, stronger generalization ability compared with the method based on v2x or high-precision map, and the application can more efficiently detect and recognize traffic lights, has long detection distance, strong robustness, and lower false detection rate and missed detection rate, and is very suitable for the field of automatic driving.

[0075] The application is different from the traditional traffic light detection and recognition method based on visual images, two heat maps are used in the deep learning network model to detect the light in the outer light frame of the traffic light and the light in the outer light frame of the green light respectively, so as to improve the detection performance and robustness of the deep learning network model.

[0076] The present application is directed to the characteristics of different detection objects, and the detection of the outer light frame of the traffic light adopts an elliptical Gaussian kernel function. The labeled target traffic light outer frame distributes key points to the feature map in the form of an elliptical Gaussian kernel, and the specific formula is as follows:

[0077]

[0078] wherein, a≈0.1155w,b≈0.1155h,w and h represent the width and height of the target frame, corresponding to the center coordinates, and (x, y) corresponds to the current position. For the detection of the light in the outer light frame of the green light, a circular Gaussian kernel function is adopted, and the radius of the circular Gaussian kernel is expanded, and the specific formula is as follows:

[0079]

[0080] wherein, σ p is a standard deviation related to the size of the target, represents the center coordinates of the target, and (x, y) corresponds to the current position.

[0081] In addition, unlike many existing technologies that can only identify the most common ordinary traffic light, various types of traffic light can be identified, such as a traffic light with multiple lights lit at the same time, a traffic light with an irregular shape, etc., which can have a higher degree of popularization. Unlike the traffic light detection and identification method based on v2x and high-precision map, it does not need to install communication equipment or rely on high-precision map, and only uses the images collected by the camera to achieve the effect of detecting and identifying the traffic light.

[0082] The above embodiments are only used to illustrate the technical solutions of the present application and not to limit it. Other modifications or equivalent replacements to the technical solutions of the present application made by those skilled in the art should be covered in the scope of the claims of the present application, as long as they do not deviate from the spirit and scope of the technical solutions of the present application.

Claims

1. A traffic light detection and recognition method based on visual images, characterized by: It includes the construction of a deep learning network model and detection and recognition based on the deep learning network model, specifically including the following steps: S1: Build and train a deep learning network model based on the detection target; The construction of the deep learning network model includes: S101: Using a deep learning network to generate a heat map, where the peak on the heat map corresponds to the center point of the detection target; wherein the deep learning network generates two heat maps; the peak on one heat map corresponds to the center point of the traffic light's outer light frame, and the peak on the other heat map corresponds to the center point of the illuminated light in the traffic light's outer light frame. The heat map of the traffic light's outer light frame is generated using an elliptical Gaussian kernel function, and the heat map of the illuminated light in the traffic light's outer light frame is generated using a circular Gaussian kernel function, and the radius of the circular Gaussian kernel is increased; S102: After obtaining the center point of the detection target, the coordinates and category of the detection target are regressed using a deep learning network; S103: Outputting traffic light information through a deep learning network model, where the traffic light information is the result of fusion of the two heat maps; S2: Acquire 2D images of traffic lights on a preset road section; S3: The acquired 2D image is fed into the built and trained deep learning network model for inference; S4: Output the coordinates of the traffic light in the 2D image and its category.

2. The method for detecting and recognizing traffic lights based on visual images according to claim 1, characterized in that: The coordinates are the coordinate positions of the external light frame of the traffic light in the 2D image; the category refers to the category of the light that is on in the traffic light at the output coordinate position, specifically including the color and shape information of the light.

3. The method for detecting and recognizing traffic lights based on visual images according to claim 2, characterized in that: The shape information includes one or more or all of a circle light, a number light, and an arrow light.

4. The method for detecting and recognizing traffic lights based on visual images according to claim 1, characterized in that: The detection targets include the external light frame of the traffic light and the lights on in the external light frame of the traffic light.

5. The method for detecting and recognizing traffic lights based on visual images according to claim 1, characterized in that: In S102, after obtaining the target point of the detection center as the outer frame of the traffic light, the width and height of the frame and the category information of the lights therein are regressed using a deep learning network; After obtaining the target point of the detection center as the light in the outer light frame of the traffic light, the deep learning network is used to regress the width and height of the light as well as its color and specific shape.

6. The method for detecting and recognizing traffic lights based on visual images according to claim 1, characterized in that: There are three forms of obtaining the target point of the detection center from the two heat maps: obtaining only the outer light frame of the traffic light, obtaining only the lights in the outer light frame of the traffic light, and obtaining both the outer light frame of the traffic light and the lights in the outer light frame of the traffic light.

7. The method for detecting and recognizing traffic lights based on visual images according to claim 6, characterized in that: Based on the three forms of obtaining the detection center target point, the output of the two heat map fusion results are: If only the outer frame of the traffic light is obtained, the output is the outer frame of the traffic light and the color and default category of the light in the outer frame of the traffic light; If only the lights in the outer frame of the traffic light are obtained, the output will be the position and color of the lights in the outer frame of the traffic light; At the same time, the external light frame of the traffic light and the light on in the external light frame of the traffic light are obtained, and the output is the position of the external light frame of the traffic light and the color of the light on in the external light frame of the traffic light.

Images