Traffic light detection method, device, vehicle and readable storage medium

The neural network model recognizes the feature points and other feature information of traffic lights, which solves the cumbersome problems of existing detection methods and realizes efficient identification of traffic light detection.

CN113989774BActive Publication Date: 2024-11-08GUANGZHOU XIAOPENG CONNECTIVITY TECH CO LTD
View PDF 9 Cites 0 Cited by

Patent Information

Application Number
CN202111258632.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-10-27
Publication Date
2024-11-08
Estimated Expiration
2041-10-27

Smart Images

  • Figure CN113989774B_ABST
    Figure CN113989774B_ABST
Patent Text Reader

Abstract

The embodiments of the present invention provide a traffic light detection method, device, vehicle and readable storage medium, the method comprising: acquiring an image to be detected; inputting the image to be detected into a preset neural network model, and determining the traffic light feature points from the output feature image; determining the image feature information located at the feature point in the feature image, and using the image feature information to determine other traffic light feature information of the traffic light to identify the indication signal of the traffic light. According to the embodiments of the present invention, the feature points of the traffic light can be determined first, and then the other feature information of the traffic light can be determined based on the image feature information at the feature points. By adopting the above method, there is no need to set the anchor in advance, which simplifies the processing process of the traffic light detection task, and can achieve a simple and efficient completion of the traffic light detection task.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of vehicle technology, and in particular to a traffic light detection method, a traffic light detection device, a vehicle and a computer-readable storage medium. Background Art

[0002] During the autonomous driving process, intelligent vehicles need to perform a variety of neural network-based detection tasks. Some detection tasks use anchor-based (also known as anchor box) methods to obtain the position of the object to be detected after image recognition, or use a two-stage detection method to regress the position of the detected object and regress and classify other feature information of the object from the image range in the two-dimensional box. The above detection methods have achieved certain results in generalized vehicle detection tasks, but for a specific detection task - traffic light detection tasks, the above detection methods are too cumbersome. Summary of the invention

[0003] In view of the above problems, embodiments of the present invention are proposed to provide a traffic light detection method and a corresponding traffic light detection device, a vehicle, and a computer-readable storage medium that overcome the above problems or at least partially solve the above problems.

[0004] An embodiment of the present invention discloses a traffic light detection method, the method comprising:

[0005] Acquire the image to be detected;

[0006] Inputting the image to be detected into a preset neural network model, and determining the traffic light feature points from the output feature image;

[0007] The image feature information at the feature point in the feature image is determined, and other traffic light feature information of the traffic light is determined using the image feature information to identify the indication signal of the traffic light.

[0008] Optionally, the preset neural network model includes a neural network, and the step of inputting the image to be detected into the preset neural network model and determining the traffic light feature points from the output feature image includes:

[0009] Inputting the image to be detected into the one neural network, and outputting a corresponding multi-channel thermal map;

[0010] Determining a first channel heat map for extracting position features from the multi-channel heat maps;

[0011] A position in the first channel heat map where the probability of the feature point existing is greater than a preset probability is determined as the position of the feature point.

[0012] Optionally, the step of inputting the image to be detected into a preset neural network model and determining the traffic light feature points from the output feature image includes:

[0013] Inputting the image to be detected into the preset neural network model, and outputting the feature image whose receptive field range matches the size of the traffic light back panel;

[0014] The feature points are determined from the feature image.

[0015] Optionally, the feature points include the center point of the back panel and / or the center point of each color light area and / or the corner points of the direction indication signs in each color light area; the other traffic light feature information includes the size feature information of the back panel and / or the color feature information of each color light area and / or the directional feature information of each color light area.

[0016] Optionally, the determining the image feature information at the feature point in the feature image, and using the image feature information to determine other traffic light feature information of the traffic light includes:

[0017] Determining a second channel heat map for extracting color features from the multi-channel heat map;

[0018] Determine first image feature information located at the feature point in the second channel heat map;

[0019] The first image feature information is used to determine the display color of each color light area of ​​the traffic light.

[0020] Optionally, the determining the image feature information at the feature point in the feature image, and using the image feature information to determine other traffic light feature information of the traffic light includes:

[0021] Determining a third channel heat map for extracting size features from the multi-channel heat map;

[0022] Determine second image feature information located at the feature point in the third channel heat map;

[0023] The absolute size of the traffic light back panel is determined using the second image feature information.

[0024] Optionally, the determining the image feature information at the feature point in the feature image, and using the image feature information to determine other traffic light feature information of the traffic light includes:

[0025] Determining a fourth channel heat map for extracting a directional feature from the multi-channel heat map;

[0026] Determine third image feature information located at the feature point in the fourth channel heat map;

[0027] The third image feature information is used to determine the indication directions of the various color light areas of the traffic light.

[0028] The embodiment of the present invention further discloses a traffic light detection device, the device comprising:

[0029] An acquisition module, used for acquiring an image to be detected;

[0030] A first determination module is used to input the image to be detected into a preset neural network model and determine the traffic light feature points from the output feature image;

[0031] The second determination module is used to determine the image feature information located at the feature point in the feature image, and use the image feature information to determine other traffic light feature information of the traffic light to identify the indication signal of the traffic light.

[0032] Optionally, the preset neural network model includes a neural network, and the first determination module includes:

[0033] A first output submodule, used for inputting the image to be detected into the one neural network and outputting a corresponding multi-channel thermal map;

[0034] A first determination submodule, used to determine a first channel heat map for extracting position features from the multi-channel heat map;

[0035] The second determination submodule is used to determine the position in the first channel heat map where the probability of the feature point existing is greater than a preset probability as the position of the feature point.

[0036] Optionally, the first determining module includes:

[0037] A second output submodule, used for inputting the image to be detected into the preset neural network model, and outputting the feature image whose receptive field range matches the size of the traffic light back panel;

[0038] The third determining submodule is used to determine the feature points from the feature image.

[0039] Optionally, the feature points include the center point of the back panel and / or the center point of each color light area and / or the corner points of the direction indication signs in each color light area; the other traffic light feature information includes the size feature information of the back panel and / or the color feature information of each color light area and / or the directional feature information of each color light area.

[0040] Optionally, the second determining module includes:

[0041] A fourth determination submodule determines, from the multi-channel thermal map, a second channel thermal map for extracting color features;

[0042] A fifth determination submodule determines first image feature information located at the feature point in the second channel heat map;

[0043] The sixth determination submodule uses the first image feature information to determine the display color of each color light area of ​​the traffic light.

[0044] Optionally, the second determining module includes:

[0045] A seventh determination submodule, used to determine a third channel heat map for extracting size features from the multi-channel heat map;

[0046] An eighth determination submodule, configured to determine second image feature information located at the feature point in the third channel thermal map;

[0047] The ninth determination submodule is used to determine the absolute size of the traffic light back panel by using the second image feature information.

[0048] Optionally, the second determining module includes:

[0049] a tenth determination submodule, configured to determine, from the multi-channel thermal maps, a fourth channel thermal map for extracting directional features;

[0050] An eleventh determination submodule is used to determine third image feature information located at the feature point in the fourth channel thermal map;

[0051] The twelfth determination submodule is used to determine the indication direction of each color light area of ​​the traffic light by using the third image feature information.

[0052] An embodiment of the present invention further discloses a vehicle, comprising: a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the steps of a traffic light detection method as described above when executed by the processor.

[0053] The embodiment of the present invention further discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the traffic light detection method described above are implemented.

[0054] The embodiments of the present invention include the following advantages:

[0055] In the embodiment of the present invention, the characteristic points of the traffic light can be determined first, and then other characteristic information of the traffic light can be determined according to the image characteristic information on the characteristic points. By adopting the above method, there is no need to set anchors in advance, which simplifies the processing process of the traffic light detection task. The above technical solution designed based on the physical characteristics of the traffic light can achieve simple and efficient completion of the traffic light detection task. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is a flow chart of the steps of a traffic light detection method provided by an embodiment of the present invention;

[0057] Figure 2 is a flowchart of another traffic light detection method provided by an embodiment of the present invention;

[0058] Figure 3 is a schematic diagram of a multi-channel thermal map of an embodiment of the present invention;

[0059] Figure 4 This is a schematic diagram of the model of the effect of the deformation characteristics of the imaging device, "larger near and smaller far", on the traffic light;

[0060] Figure 5 It is a structural block diagram of a traffic light detection device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0061] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0062] In the process of autonomous driving, it is necessary to perform a variety of neural network-based detection tasks. Some detection tasks use anchor-based (also known as anchor box) methods to obtain the position of the object to be detected after recognizing the image, or use a two-stage detection method to regress the position of the detected object separately, and regress and classify other feature information of the object from the image range in the two-dimensional box. The above detection methods have achieved certain results in generalized vehicle detection tasks, but for a specific detection task - traffic light detection tasks, the above detection methods are too cumbersome.

[0063] One of the core concepts of the embodiments of the present invention is that the characteristic points of the traffic light can be determined first, and then other characteristic information of the traffic light can be determined based on the image characteristic information at the characteristic points. By adopting the above method, there is no need to set anchors in advance, which simplifies the processing process of the traffic light detection task. The above technical solution designed based on the physical characteristics of the traffic light can achieve a simple and efficient completion of the traffic light detection task.

[0064] Reference Figure 1 , shows a flow chart of the steps of a traffic light detection method provided by an embodiment of the present invention, which may specifically include the following steps:

[0065] Step 101: Acquire an image to be detected.

[0066] Target detection in intelligent transportation systems uses image processing, sensor measurement and control, electronic information, and system engineering technologies to accurately identify, locate, and classify vehicles, roads, pedestrians, and other targets in traffic scenes, while providing accurate information input for intelligent transportation systems. The system collects information about the vehicle's surroundings through cameras installed at various locations on the vehicle, and analyzes the vehicle's current operating environment, thereby achieving assisted driving functions.

[0067] The embodiment of the present invention is to detect traffic lights. Traffic lights have fixed design features. Traffic lights are mainly composed of bright colored lights, usually round or arrow-shaped. The most common traffic light configuration is a round red, yellow and green light, each state of which represents stop, wait and pass. The basic component structure of a traffic light includes a pole, a backboard and a light area in the backboard. Traffic lights have obvious color characteristics and shape characteristics. Traffic lights are usually contained in a gray-black rectangular backboard, and one or more round lights are connected to the backboard. Although the climate, road environment, etc. will affect the acquisition of traffic light images to varying degrees, the shape and geometric dimensions of the traffic light will not change much.

[0068] During the automatic driving process of the intelligent vehicle, an image to be detected can be obtained through the vehicle transmitter, and the image to be detected includes a traffic light.

[0069] Step 102: input the image to be detected into a preset neural network model, and determine the traffic light feature points from the output feature image.

[0070] In the embodiment of the present invention, a neural network model is used to detect traffic lights on an image to be detected. Specifically, the acquired image to be detected is input into the neural network model for operation, and the position of the characteristic points of the traffic lights is determined according to the operation results. Compared with the prior art, in the embodiment of the present invention, the position of the characteristic points of the traffic lights is first identified, rather than identifying the features such as the color or geometric size of the traffic lights.

[0071] It should be noted that the traffic light feature point is used to subsequently identify other features of the traffic light, and the traffic light feature point can be set as a key point that is easy to identify other features of the traffic light. In a preferred example, the feature point can be the center point of the traffic light backboard (separated from the pole) or the center point of each color light area. In addition, those skilled in the art can also set different feature points according to actual needs, and the embodiment of the present invention does not impose specific limitations on this.

[0072] The neural network model is trained by using a large amount of traffic light image data as sample data. The training process of the neural network model can be performed in the vehicle, or it can also be performed on the PC. The training steps of the neural network model at least include: sending a large amount of traffic light image data as sample data to the neural network training system, wherein each traffic light image data is marked with the location of the traffic light feature point; using the neural network training system to train multiple sample data to obtain the corresponding neural network model.

[0073] Thus, in this step, the image to be detected can be input into the pre-trained neural network model, the corresponding feature image can be output, and the position of the feature point of the traffic light can be determined from the feature image. Specifically, the position of the feature point of the traffic light can be determined based on the pixel value of the pixel point in the output feature image.

[0074] The convolution-based object detection network is divided into Anchor-based algorithm and Anchor-free algorithm according to the preset anchor frame. The Anchor-based algorithm sets an anchor frame in the candidate area according to the features of the detected target, and traverses the feature map to achieve target detection. However, this detection method has technical defects such as imbalanced positive and negative samples, large amount of calculation, poor generalization ability and high subjectivity. To this end, an Anchor-free algorithm based on the key point idea is proposed, which uses the key points of the feature map to directly regress the target information without relying on the design of complex anchor frames. Typical networks include: CornerNet, ExtremeNet, CenterNet, etc.

[0075] Among them, CenterNet regards target detection as a center point detection problem, and directly regresses the relevant attributes of the target through the detected center point position, thus realizing anchor free detection. That is, in this step, if the feature point is the center point, the CenterNet algorithm can be used to determine the feature point position, and the feature image can be a heat map, and the peak point in the heat map corresponds to the center point (feature point).

[0076] Step 103: determining image feature information at the feature point in the feature image, and using the image feature information to determine other traffic light feature information of the traffic light, so as to identify the indication signal of the traffic light.

[0077] The feature image may be a plurality of thermal images output by the neural network containing various feature information of the traffic light. The feature image can not only determine the position of the feature point of the traffic light, but also determine other features of the traffic light, such as color features, size features or directional features, and the other features of the traffic light are determined by the image feature information at the feature point in the feature image output by the neural network model. In other words, other feature information of the traffic light can be determined according to the pixel value at the feature point.

[0078] The other traffic light characteristic information may include one or more of the size characteristic information of the back panel, the color characteristic information of each color light area, and the directional characteristic information of each color light area. A directional traffic light with a direction indicator has corresponding directional characteristic information.

[0079] The preset neural network model can be a convolutional neural network, which consists of a convolutional layer, a pooling layer, and a fully connected layer. The convolution operation on the image in the convolutional layer is that the convolution kernel slides on the image with a step size, and the convolution kernel and the pixel points on the image where it is located are inner-producted and summed to extract features. Different convolution kernels can extract different types of features. The main function of the pooling layer is to compress the feature map to extract the main features and eliminate irrelevant features. The fully connected layer connects all the features and passes the output value to the classifier.

[0080] In one embodiment of the present application, the training step of the neural network model at least includes: sending a large amount of traffic light image data as sample data to a neural network training system, wherein each traffic light image data is annotated with the location of the traffic light feature point, the display color of the traffic light area, and the backboard size; using the neural network training system to train multiple sample data to obtain a corresponding neural network model for determining the location of the traffic light feature point, determining the color characteristics of the traffic light area, and the backboard size characteristics. That is, all the above-mentioned feature information of the traffic light can be determined using one neural network model. And in the neural network model, other traffic light feature information is determined from the image feature information at the traffic light feature point. For example, after determining the location of the feature point, the pixel value at the feature point can be obtained, and other traffic light feature information can be determined based on the value, that is, the image feature information at other locations except the location of the traffic light feature point is not used to determine other features such as color features and size features. Specifically, the Anchor-free algorithm based on the key point concept can be used. If the feature point is the center point, the CenterNet algorithm can be used. In this algorithm, each object has only one positive sample point, and there is no need to use NMS (Non-Maximum Suppression). The local maximum value can be directly extracted through the output heat map, so that it can be known which point to use to generate the prediction box. In the detection task, the algorithm can directly predict the size of the target box, and the supervisory information of the target box size is only generated at the center point of the target box. In addition, additional attributes in other visual tasks can also be predicted. That is, the CenterNet algorithm can directly detect the center point and size of the object. Based on this feature, in an embodiment of the present invention, the CenterNet algorithm can be used to detect the center point of the traffic light backboard, and then the size of the backboard can be determined from the supervisory information at the center point.

[0081] Specifically, if the feature point is the center point of the back plate, after finding the feature point, the pixel value at the position of the feature point can be obtained from the thermal image from which the size feature of the back plate is extracted. The pixel value is the size value of the back plate.

[0082] In another example, if the feature point is the center point of each color light area, the pixel value at the feature point position can be obtained from the thermal image that extracts the color features of each color light area, and the pixel value is the confidence level value that the light area at that location is in a certain display color. For example, the three color categories of red, green, and yellow correspond to three thermal images, which respectively indicate the confidence level that the light area of ​​the traffic light is in each color. The three pixel values ​​at the center points of the three light areas of the traffic light can be determined from the thermal image corresponding to red. The three pixel values ​​are respectively used to indicate the confidence level that the display color of the three light areas is red, and can be converted into probability values ​​through the softmax function.

[0083] In another example, if the feature point is the center point of each color light area, the pixel value at the position of the feature point can be obtained from the thermal image that extracts the directional features of each color light area, and the pixel value is the confidence level value of the directional information of the light area at that location. For example, a thermal image is used to indicate the confidence level that the direction of a traffic light area is a left turn, and the three pixel values ​​at the center points of the three light areas of the traffic light can be determined from the thermal image. The three pixel values ​​are used to indicate the confidence level that the direction of the three light areas is a left turn, and are A, B, and C, respectively. Among them, if A is greater than B, greater than C, and A is greater than a preset confidence level threshold, it can be determined that the direction of the light area corresponding to A is a left turn.

[0084] In the embodiment of the present invention, if the color characteristic information of the traffic light is that the red light is on, the vehicle is controlled to stop; if the color characteristic information of the traffic light is that the green light is on and the directional characteristic information is to turn left, the vehicle can be controlled to drive left. In the embodiment of the present invention, the indication information of the traffic light in the detection image can be identified based on other acquired traffic light characteristic information to control the driving of the vehicle and determine whether to control the vehicle to pass through the intersection of the traffic light.

[0085] In addition, the other traffic light characteristic information may be used to draw a corresponding traffic light model and display it on the vehicle screen.

[0086] In summary, in the embodiment of the present invention, the characteristic points of the traffic light can be determined first, and then other characteristic information of the traffic light can be determined based on the image characteristic information at the characteristic points. By adopting the above method, there is no need to set the anchor in advance, which simplifies the processing process of the traffic light detection task. The above technical solution designed based on the physical characteristics of the traffic light can achieve a simple and efficient completion of the traffic light detection task.

[0087] Reference Figure 2 , shows a flowchart of another traffic light detection method provided by an embodiment of the present invention, which may specifically include the following steps:

[0088] Step 201: Acquire an image to be detected.

[0089] In an intelligent transportation system, vehicles need to detect targets such as vehicles, roads, pedestrians, etc. in a traffic scene. This application proposes a detection method for traffic lights based on the design features of traffic lights.

[0090] During the process of automatic driving of the vehicle, the image to be detected can be obtained through the vehicle sensor. The image to be detected can be a static image. The static image contains a traffic light.

[0091] In an embodiment of the present invention, the neural network model training method mentioned in the previous embodiment can be used to train the neural network model, and the acquired image to be detected is input into the neural network model for calculation, and the characteristic information of the traffic light is determined according to the calculation result.

[0092] In an optional example, the neural network model can be a neural network, and a neural network can be used to serially detect the image to be detected. Due to the design characteristics of the traffic light, the neural network model can only include one neural network, and the required traffic light feature information can be extracted by using this one neural network. Using a neural network can improve the processing speed of the traffic light detection task. Exemplarily, a one-stage detection method can be used to detect the image to be detected. The one-stage detection algorithm does not require the region proposal (generating candidate regions) stage, and directly generates the category probability and position coordinate value of the object. The final detection result can be directly obtained after a single detection, so it has a faster detection speed. Typical algorithms include YOLO, SSD, YOLOv2, YOLOv3, Retina-Net, etc.

[0093] Step 202: input the image to be detected into the one neural network, and output a corresponding multi-channel thermal map.

[0094] An RGB image has three channels, and the feature image output by the neural network model in the embodiment of the present invention is a heat map containing multiple channels. A heat map is a diagram that shows the page areas that visitors are interested in and the geographical areas where the visitors are located in a special highlighted form. In a neural network, it is mainly used to understand which part of a given detection image plays a role in the final classification decision of the neural network.

[0095] Class Activation Mapping (CAM) is a technique for generating heatmaps that uses a special convolutional neural network structure to generate visual heatmaps that can be used to highlight specific areas of an image class. Visualizing a neural network with a heatmap can help understand if the neural network is looking at the appropriate parts of the image or if the neural network is misleading.

[0096] If the neural network model is a neural network, after the image to be detected is input into the neural network, the corresponding multi-channel thermal map can be output. Different channels can respectively form corresponding thermal maps, and the thermal map corresponding to each channel is used to extract a category of traffic light feature information. The traffic light feature information may include one or more of the feature point position, the size feature information of the back panel, the color feature information of each color light area, and the directional feature information of each color light area. Exemplarily, the extraction of the feature point position can be performed in the first channel thermal map, while the extraction of the color feature information and the directional feature information can be extracted in the thermal map composed of other channels.

[0097] The role of each channel heat map is fixed. In order to facilitate searching, each channel in the output multi-channel heat map can be marked in advance, and the purpose of each channel can be determined. For example, the neural network model can be trained in advance to output the multi-channel heat map. The channel with the first mark is used to determine the position feature information of the traffic light feature point, the channel with the second mark is used to determine the color feature information of the traffic light, the channel with the third mark is used to determine the size feature information of the traffic light, and the channel with the fourth mark is used to determine the directional feature information of the traffic light, etc.

[0098] Step 203: determining a first channel heat map for extracting position features from the multi-channel heat maps.

[0099] In the embodiment of the present invention, a first channel heat map for extracting position features may be determined from the multi-channel heat maps.

[0100] Step 204 : determining a position in the first channel heat map where the probability of the feature point existing is greater than a preset probability as the position of the feature point.

[0101] Exemplarily, the feature points may include the center point of the back panel and the center point of each color light area. In the first channel heat map, the position where the probability of the feature point existing is greater than the preset probability can be determined as the feature point position. Preferably, the position where the probability of the feature point existing is the largest and greater than the preset probability can also be determined as the feature point position. Specifically, those skilled in the art hope that the regression result of the position corresponding to the feature point on the heat map approaches 1, that is, the probability of the existence of a certain feature point at the position approaches 1.

[0102] In an optional example, inputting the image to be detected into a preset neural network model, and determining the traffic light feature points from the output feature image includes:

[0103] The image to be detected is input into the preset neural network model, and the feature image whose receptive field range matches the size of the traffic light back panel is output; and the feature point is determined from the feature image.

[0104] In a neural network, the definition of the receptive field is the size of the area on the input image where the pixels on the feature map output by each layer of the neural network are mapped. A more popular explanation is that a point on the feature map corresponds to an area on the input image. The receptive field is used to limit the area within which a pixel in the input image can be linked to other pixels. Increasing the size of the receptive field can utilize contextual information in a larger image area for analysis and processing. The size of the receptive field is only related to the structural features of the neural network model. That is, in an embodiment of the present invention, the network structure of a neural network used for traffic light feature extraction can be set so that the receptive field range corresponding to the network structure matches the shape or size of the traffic light. Specifically, it can match the size of the traffic light backplane. Among them, the receptive field range matches the size of the traffic light backplane, and the receptive field range can be not less than the size of the traffic light backplane.

[0105] The neural network model in the present application can use convolution operations to extract features from the image to be detected, and each pixel of the feature image output by each layer of convolution can be set to correspond to a limited receptive field in the input image, and the receptive field range can be configured to match the size of the traffic light backboard. Using this method, under the premise that the image space structure is not destroyed by convolution, the receptive field can only include information that can indicate the characteristic information of the traffic light, and the receptive field range is limited to the size of the traffic light backboard, so that no additional interference information can be introduced without missing any information.

[0106] Step 205: determining image feature information at the feature point in the feature image, and using the image feature information to determine other traffic light feature information of the traffic light, so as to identify the indication signal of the traffic light.

[0107] After determining the position of the feature point, the image feature information located at the feature point in the output feature image (multi-channel thermal map) can be determined, and the image feature information is used to determine other traffic light feature information of the traffic light.

[0108] Reference Figure 3 The figure shows a schematic diagram of a multi-channel heat map of an embodiment of the present invention. The center point of the outer contour of the traffic light (the outer contour of the back plate) is used as a feature point (Keypoint), and the position of the feature point is highlighted in the first channel heat map. The image feature information at the position of the feature point in the heat map of other channels, such as the pixel value, is used to determine other traffic light feature information, including size feature information (Dim) and color feature information (Color).

[0109] In addition, with respect to step 205, determining other traffic light characteristic information may specifically include executing the following sub-steps:

[0110] Sub-step S11, determining a second channel heat map for extracting color features from the multi-channel heat map.

[0111] Sub-step S12, determining first image feature information located at the feature point in the second channel heat map.

[0112] Sub-step S13, using the first image feature information to determine the display color of each color light area of ​​the traffic light.

[0113] A second channel heat map for extracting color features can be determined from the multi-channel heat map, first image feature information located at a feature point in the second channel heat map can be determined, and the display colors of various color light areas of the traffic light can be determined using the first image feature information. The display color categories can include red light, yellow light, green light, and no light.

[0114] Since the image captured by the on-board imaging equipment and the change of traffic light color are independent events, and there are situations where traffic lights on the road are not lit or the switching interval is long, by adding the "not lit" category in the classification process, the learning process of the neural network can be made more in line with real cognition and the accuracy of neural network classification can be improved.

[0115] In a specific implementation, the first image feature information is used to determine the display color of each color light area of ​​the traffic light. Specifically, the first image feature information may be used to determine the classification probability of the light area, and the display color of the corresponding light area is determined according to the classification probability. For example, in a certain light area, the probability of a green light is calculated to be 0.0017; the probability of a red light is 0.6069; the probability of a yellow light is 0; the probability of an off light is 0.3914; then it can be determined that the display color of the light area is a red light.

[0116] In an optional example of the present invention, a corresponding traffic light model can be drawn based on the identified other traffic light characteristic information. Before identifying other traffic light characteristic information, the type of other traffic light characteristic information required to be obtained can be determined according to user needs.

[0117] With respect to step 205, determining other traffic light characteristic information may further include executing the following sub-steps:

[0118] Sub-step S21, determining a third channel heat map for extracting size features from the multi-channel heat map.

[0119] Sub-step S22, determining the second image feature information located at the position corresponding to the feature point in the third channel heat map.

[0120] Sub-step S23: using the second image feature information to determine the absolute size of the traffic light back panel.

[0121] A third channel heat map for extracting size features can be determined from the multi-channel heat map, second image feature information located at the feature point position in the third channel heat map can be determined, and the absolute size of the traffic light back panel can be determined using the second image feature information.

[0122] By regressing the size feature information of the traffic light back panel at the feature point position, the visual two-dimensional box (bounding box) can be determined based on the feature point position and size feature information. For the special task of traffic light detection, although the imaging device has the deformation characteristic of "larger near and smaller far", due to the relatively small size of the traffic light, its deformation on the image is extremely small and is still approximately rectangular. Figure 4 The figure shows a model diagram of the effect of the deformation characteristics of the imaging device "larger near and smaller far" on traffic lights. The traffic lights and buses in the image on the left are cut row by row and spliced ​​into the strip chart on the right. The subscript W in the right figure refers to the width of the corresponding block in each row in the left figure. The corresponding widths of the Traffic Light block are W1, W2 and W3 respectively; the corresponding widths of the Bus block are W a , W b , W c , W d and W e Since Traffic Light is a rectangle, the width of each block is almost the same, so Traffic Light block width W = average (W1, W2, W3). Bus is a trapezoid, so the width of each block is different, so Bus block width W ≠ average (W a , W b , W c , W d , W e ). That is, for the rectangular Traffic Light, the final W can be obtained by taking the average value of the W in each row, because their values ​​are almost the same; while for the trapezoidal Bus, the maximum value of the W in each row needs to be taken, which is complicated to calculate.

[0123] In addition, because the traffic lights in the same country or region are relatively uniform, with regular shapes, similar sizes, and fixed width-to-height ratios, the regression variance of the traffic light size is small, and no additional auxiliary means are needed, and the absolute length of the traffic light can be directly regressed.

[0124] In another optional example, the relative size of the traffic light can also be regressed. Exemplarily, instead of directly determining the absolute length of the traffic light backboard, the natural logarithm of the ratio of the absolute length to the standard length is regressed. The standard length can be the average length of the traffic light backboard size in the sample data set. In other words, in addition to training the neural network model to output the absolute size of the traffic light backboard, the neural network model can also be trained to output the relative size of the traffic light backboard. Compared with the absolute size, regressing the relative size can reduce the dynamic range of the neural network regression value, which can improve the regression performance of the neural network.

[0125] In an optional example, for step 205, the following steps may be performed:

[0126] Sub-step S31, determining a fourth channel heat map for extracting directional features from the multi-channel heat maps.

[0127] Sub-step S32, determining the third image feature information located at the feature point in the fourth channel heat map.

[0128] Sub-step S33, using the third image feature information to determine the indication direction of each color light area of ​​the traffic light.

[0129] A fourth channel heat map for extracting directional features can be determined from the multi-channel heat map, and the indication directions of the various color light areas of the traffic light can be determined using the third image feature information at the feature point position in the fourth channel heat map.

[0130] For the recognition of the direction indicated by traffic lights, the corner points of the direction indicator signs in the various color light areas can also be used as feature points to obtain better classification and recognition results of the direction indicated by traffic lights. For the use of center points as feature points, the target detection network of CenterNet can be used for detection, while for the use of corner points as feature points, other target detection networks can be used for detection.

[0131] It should be noted that the first image feature information, the second image feature information and the third image feature information are only used to distinguish different image feature information, and do not represent a sequence or other meanings.

[0132] After determining other traffic light characteristic information, the traffic light's indication signal can be identified, thereby determining whether to control the vehicle to pass through the intersection of the traffic light.

[0133] In order to enable those skilled in the art to better understand steps 201 to 206 of the embodiment of the present invention, an example is given below to illustrate:

[0134] The specific implementation process of the traffic light detection method is as follows:

[0135] 1. During the process of automatic driving of a vehicle, the vehicle sensor can obtain an image to be detected, and the image to be detected includes a traffic light; the automatic driving system of the vehicle has a pre-trained neural network model, and the neural network model is used for traffic light detection. And the neural network model is composed of a neural network.

[0136] 2. After the image to be detected is input into the neural network, multiple heat maps composed of different channels can be output in the neural network, which are used to identify various characteristic information of traffic lights, including characteristic point positions, light area color characteristic information, back panel size characteristic information, and light area directional characteristic information. Among them, the first thing to be identified is the position of the traffic light characteristic point, because the identification of other characteristic information depends on the position of the characteristic point. Therefore, the first channel heat map is used to determine the position of the characteristic point, while other features, such as size features, color features, and directional features, are identified in other heat maps. Specifically, they are determined based on the pixel value at the characteristic point position.

[0137] 3. Based on the identified traffic light features, determine the traffic light's indication signal and control the vehicle to perform corresponding driving behavior.

[0138] In summary, in the embodiment of the present invention, the characteristic points of the traffic light can be determined first, and then other characteristic information of the traffic light can be determined based on the image characteristic information at the characteristic points. By adopting the above method, there is no need to set anchors in advance, which simplifies the processing process of the traffic light detection task, and can increase the speed and reduce the overhead during the neural network training process. The above technical solution designed based on the physical characteristics of the traffic light can achieve a simple and efficient completion of the traffic light detection task.

[0139] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the described action sequence, because according to the embodiments of the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0140] Reference Figure 5 , shows a structural block diagram of a traffic light detection device provided by an embodiment of the present invention, which may specifically include the following modules:

[0141] An acquisition module 501 is used to acquire an image to be detected;

[0142] A first determination module 502, used for inputting the image to be detected into a preset neural network model, and determining the traffic light feature points from the output feature image;

[0143] The second determination module 503 is used to determine the image feature information located at the feature point in the feature image, and use the image feature information to determine other traffic light feature information of the traffic light to identify the indication signal of the traffic light.

[0144] In an optional embodiment, the preset neural network model includes a neural network, and the first determination module includes:

[0145] A first output submodule, used for inputting the image to be detected into the one neural network and outputting a corresponding multi-channel thermal map;

[0146] A first determination submodule, used to determine a first channel heat map for extracting position features from the multi-channel heat map;

[0147] The second determination submodule is used to determine the position in the first channel heat map where the probability of the feature point existing is greater than a preset probability as the position of the feature point.

[0148] In an optional embodiment, the first determining module includes:

[0149] A second output submodule, used for inputting the image to be detected into the preset neural network model, and outputting the feature image whose receptive field range matches the size of the traffic light back panel;

[0150] The third determining submodule is used to determine the feature points from the feature image.

[0151] In an optional embodiment, the feature points include the center point of the back panel and / or the center point of each color light area and / or the corner points of the direction indicator signs in each color light area; the other traffic light feature information includes the size feature information of the back panel and / or the color feature information of each color light area and / or the directional feature information of each color light area.

[0152] In an optional embodiment, the second determining module includes:

[0153] A fourth determination submodule determines, from the multi-channel thermal map, a second channel thermal map for extracting color features;

[0154] A fifth determination submodule determines first image feature information located at the feature point in the second channel heat map;

[0155] The sixth determination submodule uses the first image feature information to determine the display color of each color light area of ​​the traffic light.

[0156] In an optional embodiment, the second determining module includes:

[0157] A seventh determination submodule, used to determine a third channel heat map for extracting size features from the multi-channel heat map;

[0158] An eighth determination submodule, configured to determine second image feature information located at the feature point in the third channel thermal map;

[0159] The ninth determination submodule is used to determine the absolute size of the traffic light back panel by using the second image feature information.

[0160] In an optional embodiment, the second determining module includes:

[0161] a tenth determination submodule, configured to determine, from the multi-channel thermal maps, a fourth channel thermal map for extracting directional features;

[0162] An eleventh determination submodule is used to determine third image feature information located at the feature point in the fourth channel thermal map;

[0163] The twelfth determination submodule is used to determine the indication direction of each color light area of ​​the traffic light by using the third image feature information.

[0164] In summary, in the embodiment of the present invention, the characteristic points of the traffic light can be determined first, and then other characteristic information of the traffic light can be determined based on the image characteristic information at the characteristic points. By adopting the above method, there is no need to set anchors in advance, which simplifies the processing process of the traffic light detection task, and can increase the speed and reduce the overhead during the neural network training process. The above technical solution designed based on the physical characteristics of the traffic light can achieve a simple and efficient completion of the traffic light detection task.

[0165] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0166] An embodiment of the present invention also provides a vehicle, comprising: a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, the various processes of the above-mentioned traffic light detection method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0167] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned traffic light detection method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0168] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0169] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0170] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0171] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0172] These computer program instructions can also be loaded into a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable terminal device provide for implementing the process. Figure 1A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0173] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0174] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0175] The traffic light detection method and traffic light detection device, a vehicle and a computer-readable storage medium provided by the present invention are introduced in detail above. The principles and implementation methods of the present invention are explained in this article using specific examples. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A traffic light detection method, characterized in that: The method comprises: Acquire the image to be detected; Inputting the image to be detected into a preset neural network model, and determining the traffic light feature points from the output feature image; Determine the image feature information at the feature point in the feature image, and use the image feature information to determine other traffic light feature information of the traffic light, so as to identify the indication signal of the traffic light; wherein the other traffic light feature information includes one or more of the size feature information of the back panel, the color feature information of each color light area, and the directional feature information of each color light area; The preset neural network model includes a neural network, and the step of inputting the image to be detected into the preset neural network model and determining the traffic light feature points from the output feature image includes: Inputting the image to be detected into the one neural network, and outputting a corresponding multi-channel thermal map; Determining a first channel heat map for extracting position features from the multi-channel heat maps; A position in the first channel heat map where the probability of the feature point existing is greater than a preset probability is determined as the position of the feature point.

2. The method according to claim 1, characterized in that The step of inputting the image to be detected into a preset neural network model and determining the traffic light feature points from the output feature image comprises: Inputting the image to be detected into the preset neural network model, and outputting the feature image whose receptive field range matches the size of the traffic light back panel; The feature points are determined from the feature image.

3. The method according to claim 1, characterized in that The characteristic points include the center point of the back plate and / or the center point of each color light area and / or the corner points of the direction indication signs in each color light area.

4. The method according to claim 1, characterized in that: The determining of the image feature information at the feature point in the feature image, and using the image feature information to determine other traffic light feature information of the traffic light, includes: Determining a second channel heat map for extracting color features from the multi-channel heat map; Determine first image feature information located at the feature point in the second channel heat map; The first image feature information is used to determine the display color of each color light area of ​​the traffic light.

5. The method according to claim 4, characterized in that The determining of the image feature information at the feature point in the feature image, and using the image feature information to determine other traffic light feature information of the traffic light, includes: Determining a third channel heat map for extracting size features from the multi-channel heat map; Determine second image feature information located at the feature point in the third channel heat map; The absolute size of the traffic light back panel is determined using the second image feature information.

6. The method according to claim 4, characterized in that The determining of the image feature information at the feature point in the feature image, and using the image feature information to determine other traffic light feature information of the traffic light, includes: Determining a fourth channel heat map for extracting a directional feature from the multi-channel heat map; Determine third image feature information located at the feature point in the fourth channel heat map; The third image feature information is used to determine the indication directions of the various color light areas of the traffic light.

7. A traffic light detection device, characterized in that: The device comprises: An acquisition module, used for acquiring an image to be detected; A first determination module is used to input the image to be detected into a preset neural network model and determine the traffic light feature points from the output feature image; a second determination module, for determining image feature information at the feature point in the feature image, and using the image feature information to determine other traffic light feature information of the traffic light, so as to identify the indication signal of the traffic light; wherein the other traffic light feature information includes one or more of size feature information of a back panel, color feature information of each color light area, and directional feature information of each color light area; The preset neural network model includes a neural network, and the first determination module includes: A first output submodule, used for inputting the image to be detected into the one neural network and outputting a corresponding multi-channel thermal map; A first determination submodule, used to determine a first channel heat map for extracting position features from the multi-channel heat map; The second determination submodule is used to determine the position in the first channel heat map where the probability of the feature point existing is greater than a preset probability as the position of the feature point.

8. A vehicle, characterized in that: include: A processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the steps of a traffic light detection method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of a traffic light detection method according to any one of claims 1 to 6 are implemented.

Citation Information

Patent Citations

  • Traffic signal light state recognition method and device, vehicle-mounted control terminal and motor vehicle

    CN108804983A

  • Traffic light detection method, intelligent driving method and device, vehicle and electronic equipment

    CN110660254A

  • Traffic light state identification method and device, electronic equipment and storage medium

    CN111428663A

  • Traffic light image recognition processing method and device, auxiliary passage system and storage medium

    CN111723625A

  • Image traffic signal lamp detection method and device, electronic equipment and storage medium

    CN111832493A