Traffic signal light detection method, device and electronic equipment

By obtaining candidate detection areas from multiple scene images and determining the target detection area, inputting them into the detection model, the problem of insufficient view and resolution of the on-board camera image is solved, the accuracy of traffic light detection is improved, and the autonomous driving vehicle is ensured. Driving safety.

CN114863386BActive Publication Date: 2024-11-12GUANGZHOU WERIDE TECH LTD CO
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202210331586.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-30
Publication Date
2024-11-12
Estimated Expiration
2042-03-30

AI Technical Summary

Technical Problem

In the prior art, the traffic light images captured by the on-board camera have a large field of view and insufficient resolution, resulting in low accuracy in traffic light detection results, which is prone to missed detection or identification errors, affecting the driving safety and ride experience of driverless vehicles.

Method used

By acquiring candidate detection areas from multiple scene images containing traffic lights, determining the target detection area that meets the preset conditions, inputting it to the detection model to output the initial detection result, and determining the final detection result based on the image state, improving detection accuracy.

Benefits of technology

It improves the accuracy of traffic light detection results, reduces false alarms, missed reports and misreports, and enhances the driving safety and ride experience of autonomous vehicles at traffic lights.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114863386B_ABST
    Figure CN114863386B_ABST
Patent Text Reader

Abstract

The present invention provides a method, device and electronic device for detecting traffic lights. First, multiple candidate detection areas are obtained from multiple scene images containing traffic lights. The size of the obtained candidate detection areas is smaller than the size of the scene images. Then, a target detection area that meets preset conditions is determined from the candidate detection areas. The preset conditions are determined based on the shooting parameters of the scene images corresponding to the candidate detection areas and / or the regional content in the candidate detection areas. Then, the target detection area is input into a preset detection model, and the initial detection result of the traffic light and the image state of the target detection area are output. Finally, based on the initial detection result and the image state of the target detection area, the final detection result of the traffic light is determined. This method uses a detection area with a smaller size and shooting parameters and regional content that meet the preset conditions to detect traffic lights, thereby improving the accuracy of the detection results.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a method, device and electronic equipment for detecting a traffic light. Background Art

[0002] Unmanned vehicles need to detect traffic lights from images taken by on-board cameras, and then identify the traffic command information indicated by traffic lights, and control vehicle operation according to the traffic command information. In related technologies, images taken by on-board cameras are input into a neural network, and the detection results of traffic lights are output. In this method, if the image shooting field of view is large, the image area occupied by traffic lights is small, and if the image resolution is insufficient, it is easy to miss the detection; due to the constraints of the neural network, the image may need to be resized before being input into the neural network. The adjusted image may reduce the resolution or deform, further affecting the final detection results. These factors lead to low accuracy of traffic light detection results, which in turn leads to subsequent traffic command information recognition errors, which have an adverse impact on the driving safety and riding experience of unmanned vehicles. Summary of the invention

[0003] In view of this, an object of the present invention is to provide a method, device and electronic equipment for detecting a traffic light, so as to improve the accuracy of the detection result of the traffic light.

[0004] In a first aspect, an embodiment of the present invention provides a method for detecting a traffic light, the method comprising: obtaining a plurality of candidate detection areas from a plurality of scene images including traffic lights; wherein the candidate detection areas include at least a portion of the traffic light; the size of the candidate detection areas is smaller than the size of the scene image; determining a target detection area that meets preset conditions from the candidate detection areas; wherein the preset conditions are determined based on shooting parameters of the scene image corresponding to the candidate detection areas and / or the area content in the candidate detection areas; inputting the target detection area into a preset detection model, outputting an initial detection result of the traffic light and an image state of the target detection area; and determining a final detection result of the traffic light based on the initial detection result and the image state of the target detection area.

[0005] Optionally, in a first implementation method of the first aspect of the present invention, the above-mentioned step of obtaining multiple candidate detection areas from a scene image containing a traffic light includes: for each scene image, based on the position annotation information of the traffic light in the semantic map corresponding to the scene image, projecting the traffic light area indicated by the position annotation information onto the scene image to obtain the projection area in the scene image; and determining the projection area as the candidate detection area.

[0006] Optionally, in a second implementation method of the first aspect of the present invention, the above-mentioned position annotation information includes the vertex coordinates of the three-dimensional traffic light area in the semantic map; based on the position annotation information of the traffic light in the semantic map corresponding to the scene image, the traffic light area indicated by the position annotation information is projected onto the scene image to obtain the projection area in the scene image, comprising: based on the three-dimensional coordinate system of the semantic map and the camera coordinate system corresponding to the vehicle-mounted camera, converting the vertex coordinates of the three-dimensional traffic light area into position coordinates in the camera coordinate system; based on the camera coordinate system and the plane coordinate system corresponding to the scene image, transforming the position coordinates in the camera coordinate system into position coordinates in the plane coordinate system; and determining the circumscribed rectangle corresponding to the position coordinates in the plane coordinate system as the projection area in the scene image.

[0007] Optionally, in a third implementation manner of the first aspect of the present invention, the step of obtaining multiple candidate detection areas from a scene image containing a traffic light includes: inputting the scene image into a pre-trained target detection model, and outputting multiple target areas in the scene image; the target area includes a detection target whose category is a traffic light detected by the target detection model; for each target area, judging whether the detection target is a traffic light based on traffic light information in a related image of the scene image; the related image is captured by a vehicle-mounted camera for a scene corresponding to the scene image; the shooting time of the related image is earlier than the shooting time of the scene image; if so, determining the target area as a candidate detection area.

[0008] Optionally, in a fourth implementation manner of the first aspect of the present invention, the above-mentioned shooting parameters include a shooting angle of a scene image; the preset conditions specifically include: a detection state parameter determined based on the shooting parameters of the scene image corresponding to the candidate detection area and the regional content in the candidate detection area is greater than a preset parameter threshold; the detection state parameter characterizes the integrity and credibility of the detection result of the candidate detection area; the step of determining a target detection area that meets the preset conditions from the candidate detection area includes: for each candidate detection area, based on the shooting angle of the scene image corresponding to the candidate detection area, the display completeness of the traffic light indicated by the regional content, and the resolution of the candidate detection area, determining the detection state parameter of the candidate detection area; and determining the candidate detection area whose detection state parameter is greater than the preset parameter threshold as the target detection area.

[0009] Optionally, in a fifth implementation manner of the first aspect of the present invention, the above-mentioned step of inputting the target detection area into a preset detection model and outputting the initial detection result of the traffic light and the image state of the target detection area includes: inputting the target detection area into a preset detection model, extracting the attribute characteristics of the traffic light in the target detection area and the image characteristics of the target detection area through the detection model, determining the initial detection result of the traffic light based on the attribute characteristics of the traffic light, and determining the image state of the target detection area based on the image characteristics; the detection model is established based on a feature graph pyramid network.

[0010] Optionally, in a sixth implementation manner of the first aspect of the present invention, the above-mentioned initial detection results include attribute information of multiple traffic lights; the step of determining the final detection result of the traffic light based on the initial detection results and the image state of the target detection area includes: determining the confidence of each attribute information based on the attribute information of multiple traffic lights and the image state corresponding to the attribute information; determining the final detection result of the traffic light with the attribute information having a confidence higher than a preset confidence threshold.

[0011] Optionally, in a seventh implementation manner of the first aspect of the present invention, the attribute information includes multiple groups of attribute sets of traffic lights; each group of attribute sets includes multiple sub-attributes corresponding to the same traffic light; the image state includes the lighting conditions of the scene image corresponding to the target detection area and the image quality of the target detection area; based on the image state and the attribute information of multiple traffic lights, the step of determining the confidence of each attribute information includes: for each group of attribute sets, judging whether the image quality of the target detection area corresponding to the attribute set is higher than a preset quality threshold; if so, judging whether the state information of the traffic lights indicated by each sub-attribute in the attribute set is consistent; if consistent, determining that the confidence of the sub-attribute in the attribute set is higher than the preset confidence threshold; if inconsistent, correcting the sub-attributes in the attribute set based on the lighting conditions of the scene image corresponding to the target detection area; if after correction, the state information of the traffic lights indicated by each sub-attribute in the attribute set is consistent, determining that the confidence of the sub-attribute in the attribute set is higher than the preset confidence threshold.

[0012] In a second aspect, an embodiment of the present invention provides a traffic light detection device, which includes: a candidate detection area acquisition module, used to acquire multiple candidate detection areas from multiple scene images containing traffic lights; wherein the candidate detection areas include at least part of the traffic lights; the size of the candidate detection areas is smaller than the size of the scene image; a target detection area determination module, used to determine a target detection area that meets preset conditions from the candidate detection areas; wherein the preset conditions are determined based on the shooting parameters of the scene image corresponding to the candidate detection areas and / or the area content in the candidate detection areas; a detection module, used to input the target detection area into a preset detection model, and output the initial detection result of the traffic light and the image status of the target detection area; a final detection result determination module, used to determine the final detection result of the traffic light based on the initial detection result and the image status of the target detection area.

[0013] In a third aspect, an embodiment of the present invention provides an electronic device, including a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the above-mentioned traffic light detection method.

[0014] In a fourth aspect, an embodiment of the present invention provides a machine-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the above-mentioned traffic light detection method.

[0015] The embodiments of the present invention bring the following beneficial effects:

[0016] The embodiment of the present invention provides a method, device and electronic device for detecting a traffic light. First, multiple candidate detection areas are obtained from multiple scene images containing traffic lights. The size of the obtained candidate detection areas is smaller than the size of the scene image. Then, a target detection area that meets preset conditions is determined from the candidate detection areas. The preset conditions are determined based on the shooting parameters of the scene image corresponding to the candidate detection area and / or the area content in the candidate detection area. Then, the target detection area is input into a preset detection model, and the initial detection result of the traffic light and the image state of the target detection area are output. Finally, based on the initial detection result and the image state of the target detection area, the final detection result of the traffic light is determined. This method uses a detection area with a smaller size and shooting parameters and area content that meet the preset conditions to detect traffic lights, thereby improving the accuracy of the detection results.

[0017] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0018] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 A flow chart of a method for detecting a traffic light provided by an embodiment of the present invention;

[0021] Figure 2 A flow chart of another method for detecting a traffic light provided by an embodiment of the present invention;

[0022] Figure 3 A schematic diagram of the structure of a traffic signal light detection device provided by an embodiment of the present invention;

[0023] Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0025] Traffic light detection is a very important step in the traffic light recognition process. Especially for vehicles with a high degree of autonomous driving or unmanned vehicles, if the detection results of traffic lights are not accurate enough, it will cause problems such as false alarms, omissions, and wrong alarms of traffic light attribute information, leading to subsequent recognition errors, and causing vehicles to run red lights, brake suddenly, get stuck, etc. at intersections with traffic lights, seriously threatening driving safety and damaging the riding experience.

[0026] Based on this, the embodiments of the present invention provide a method, device and electronic device for detecting a traffic light. The technology can be applied to the detection process of traffic lights in various scenarios.

[0027] To facilitate understanding of this embodiment, a method for detecting a traffic light is first introduced. Figure 1 As shown, the method comprises the following steps:

[0028] Step S102, obtaining a plurality of candidate detection areas from a plurality of scene images including traffic lights; wherein the candidate detection areas include at least a portion of the traffic lights; and the size of the candidate detection areas is smaller than the size of the scene images.

[0029] The above-mentioned multiple scene images can be obtained by photographing the traffic intersection in front of the vehicle by multiple forward-facing cameras installed on the vehicle. Since the camera's shooting field of view is limited, multiple forward-facing cameras are usually distributed on the vehicle, and the positions and shooting angles of each forward-facing camera are different. Multiple images are captured by these forward-facing cameras to obtain a relatively complete scene picture of the intersection in front of the vehicle during the driving process. These forward-facing cameras can capture scene pictures at the same time, or they can capture scene images in sequence in a certain order.

[0030] When obtaining the candidate detection area, the scene image can be detected by using a pre-trained traffic light detection network to obtain an image area containing the detection target. Since the detection network may have detection errors, such as detecting an obstacle similar to a traffic light as a traffic light, the traffic light can be tracked based on scene images taken by the same forward camera at different times. If it can be tracked, it is determined that the detection target in the detected image area is a traffic light, and then the detected image area is determined to be a candidate detection area.

[0031] In addition, some vehicles are also equipped with a system capable of creating a semantic map, which can project the annotated area of ​​the traffic light in the semantic map corresponding to the scene image into the scene image to obtain a candidate detection area containing the traffic light in the scene image.

[0032] Step S104, determining a target detection area that meets a preset condition from the candidate detection areas; wherein the preset condition is determined based on shooting parameters of the scene image corresponding to the candidate detection area and / or area content in the candidate detection area.

[0033] Since the candidate detection areas include multiple ones, and they are usually captured by different cameras, the traffic lights contained in the area content of the candidate detection areas may be repeated. When the integrity of the traffic lights in the candidate detection areas is different, the detection results of the same traffic light may be different, resulting in conflicting detection results; and repeatedly detecting the same traffic light will result in low detection efficiency. Therefore, it is necessary to select a candidate detection area with relatively clear and complete traffic lights contained in the area content.

[0034] The shooting angles and focal lengths of the forward cameras distributed on the vehicle may be different. Some cameras mainly capture images of obstacles that are lower on the road where the vehicle is traveling; some cameras mainly capture images of obstacles that are higher and diagonally above the vehicle. The images captured by such cameras usually include traffic lights, and the traffic lights are relatively complete.

[0035] Therefore, it is necessary to combine the shooting parameters of the scene image corresponding to the candidate detection area or the regional content in the candidate detection area, or consider both at the same time, and select the candidate detection area with a relatively complete traffic light and clear image as the target detection area. One or more target detection areas can be selected according to needs.

[0036] Step S106, inputting the target detection area into a preset detection model, and outputting the initial detection result of the traffic light and the image status of the target detection area.

[0037] The above detection model can be established based on neural network and / or deep learning theory, and is obtained by training with a large amount of sample data labeled with traffic light information. The detection model needs to extract the attribute characteristics of the traffic light and the image characteristics of the target detection area, and then determine the detection result of the traffic light and the image state of the target detection area based on the attribute characteristics of the traffic light and the image characteristics of the target detection area.

[0038] The above initial detection result may include various attribute information of the traffic lights, such as the location of the traffic lights, the number of traffic lights, the composition of each traffic light (such as the light frame and the light bulb), the color, the display content, etc. The above image status may include the image quality of the target detection area, such as the resolution and clarity, and the lighting conditions when the target detection area is photographed, such as daytime, night, cloudy, etc.

[0039] Step S108, determining a final detection result of the traffic light based on the initial detection result and the image status of the target detection area.

[0040] The above-mentioned image state can represent the shooting conditions and image quality of the scene image corresponding to the captured target detection area. Generally speaking, when the image quality is high and the shooting conditions are daytime and the lighting is sufficient, the image of the traffic light in the target detection area has more details and is relatively clear, so the detection result is credible. Therefore, the initial detection results can be screened based on the image state of the target detection area; when the image state of the target detection area corresponding to the initial detection result is poor, it can be considered that the confidence of the initial detection result is low.

[0041] In addition, the initial detection results usually include multiple attribute information of traffic lights. The attribute information of the same traffic light should be consistent, and the attribute information of the same group of traffic lights should indicate the same traffic information. For example, for the same traffic light, the color of its light frame and bulb should be the same. If they are different, the confidence of the detection result is low. In the detection results of the same group of traffic lights, when the red light is on and the green light is also on, the confidence of the detection result is also low.

[0042] Based on the image state of the target detection area and the consistency of the initial detection results, the confidence of each detection result in the initial detection results can be determined. Finally, the initial detection result with a confidence higher than a certain threshold can be used as the final detection result, or the initial detection result with the highest confidence can be used as the final detection result, thereby obtaining a detection result of a traffic light with higher accuracy.

[0043] The embodiment of the present invention provides a method for detecting a traffic light. First, multiple candidate detection areas are obtained from multiple scene images containing traffic lights. The size of the obtained candidate detection areas is smaller than the size of the scene image. Then, a target detection area that meets preset conditions is determined from the candidate detection areas. The preset conditions are determined based on the shooting parameters of the scene image corresponding to the candidate detection area and / or the area content in the candidate detection area. Then, the target detection area is input into a preset detection model, and the initial detection result of the traffic light and the image state of the target detection area are output. Finally, based on the initial detection result and the image state of the target detection area, the final detection result of the traffic light is determined. This method uses a detection area with a smaller size and shooting parameters and area content that meet the preset conditions to detect traffic lights, thereby improving the accuracy of the detection results.

[0044] The following embodiment provides a specific implementation process of obtaining multiple candidate detection areas from a scene image containing a traffic light.

[0045] When the degree of automation of the vehicle is high, a semantic map of the vehicle during driving can be obtained; the semantic map marks the types and location information of obstacles in the vehicle's environment, and has high accuracy. For each scene image, based on the location annotation information of the traffic light in the semantic map corresponding to the scene image, the traffic light area indicated by the location annotation information can be projected onto the scene image to obtain the projection area in the scene image, and the projection area is determined as the candidate detection area. Among them, the semantic map corresponding to the scene image can be determined based on the location of the vehicle when the scene image is taken.

[0046] Specifically, the above-mentioned position annotation information includes the vertex coordinates of the three-dimensional traffic light area in the semantic map; the semantic map usually uses a rectangular parallelepiped to annotate traffic lights, so the position annotation information includes the coordinates of the eight vertices of the rectangular parallelepiped in the three-dimensional world. During the projection process, based on the three-dimensional coordinate system of the semantic map and the camera coordinate system corresponding to the on-board camera, the vertex coordinates of the three-dimensional traffic light area can be converted into position coordinates in the camera coordinate system, and then based on the camera coordinate system and the plane coordinate system corresponding to the scene image, the position coordinates in the camera coordinate system are transformed into position coordinates in the plane coordinate system, and finally the circumscribed rectangle corresponding to the position coordinates in the plane coordinate system is determined as the projection area in the scene image.

[0047] When the vehicle's autonomous driving level is low, the scene image can be subjected to target detection, and a candidate detection area can be determined based on the target detection result. In a specific implementation, the scene image can be input into a pre-trained target detection model, and multiple target areas in the scene image can be output, and the output target area includes the detection target of the category of traffic lights detected by the target detection model; wherein the target detection model can be established based on a neural network or deep learning theory. Then, for each target area, based on the traffic light information in the relevant image of the scene image, it is determined whether the detection target is a traffic light; the relevant image is taken by the on-board camera for the scene corresponding to the scene image; the shooting time of the relevant image is earlier than the shooting time of the scene image; if so, the target area is determined as a candidate detection area. Wherein, the relevant image can be a plurality of images of the traffic intersection taken before the scene image is taken, and the traffic light information of the relevant image includes the location information of the traffic light. When the traffic light detected in the scene image can be tracked from the relevant image based on the location information of the traffic light in the relevant image, the detection target in the target area is considered to be a traffic light, and the target area is determined as a candidate detection area.

[0048] The embodiment of the present invention also provides another method for detecting a traffic signal light. Figure 1This method mainly introduces the specific implementation process of determining the target detection area that meets the preset conditions from the candidate detection area, the specific implementation process of obtaining the initial detection result of the traffic light and the image state of the target detection area, and the specific implementation process of determining the final detection result of the traffic light. Figure 2 As shown, the method comprises the following steps:

[0049] Step S202, obtaining a plurality of candidate detection areas from a plurality of scene images including traffic lights; wherein the candidate detection areas include at least a portion of the traffic lights; and the size of the candidate detection areas is smaller than the size of the scene images.

[0050] Step S204, for each candidate detection area, based on the shooting angle of the scene image corresponding to the candidate detection area, the display completeness of the traffic light indicated by the area content, and the resolution of the candidate detection area, determine the detection state parameters of the candidate detection area.

[0051] Different forward cameras may have different shooting angles. Some forward cameras are suitable for shooting traffic lights located above and in front of the vehicle, such as forward cameras with telephoto lenses (30 degrees). Therefore, scores and weights can be set for different shooting angles as part of the parameters for calculating the detection state parameters.

[0052] In addition, since the display completeness of the traffic light and the resolution of the candidate detection area will affect the detection result of the traffic light, scores and weights can be set for different display completeness and different resolutions. Based on the scores and weights of the three, the detection state parameters of the candidate detection area are calculated. The detection state parameters can characterize the completeness and credibility of the detection results of the candidate detection area.

[0053] Step S206: determine the candidate detection area whose detection state parameter is greater than a preset parameter threshold as the target detection area.

[0054] Step S208, input the target detection area into a preset detection model, extract the attribute characteristics of the traffic lights in the target detection area and the image characteristics of the target detection area through the detection model, determine the initial detection result of the traffic lights based on the attribute characteristics of the traffic lights, and determine the image state of the target detection area based on the image characteristics; the detection model is established based on the feature graph pyramid network.

[0055] The above detection model can be built based on the Feature Pyramid Networks (FPN) of the two-stage detector. The detection model can simultaneously output the light frame and bulb of the traffic light, where the bulb is completely included in the light frame, and supports the output of multi-dimensional attributes of the detection results, including color / shape / special state, etc. In addition to detection, the image quality and lighting condition prediction of the entire input image can also be performed to distinguish special states such as normal / blurred / face light, and lighting conditions such as day / night / dim.

[0056] Step S210 , based on the attribute information of the plurality of traffic lights and the image states corresponding to the attribute information, the confidence level of each attribute information is determined.

[0057] The above initial detection result includes attribute information of multiple traffic lights; the above attribute information includes multiple attribute sets of traffic lights; each attribute set includes multiple sub-attributes corresponding to the same traffic light; the image state includes the lighting conditions of the scene image corresponding to the target detection area and the image quality of the target detection area; the above step S210 can be implemented in the following way:

[0058] (1) For each attribute set, determine whether the image quality of the target detection area corresponding to the attribute set is higher than a preset quality threshold. Since the detection results of the target detection area with poor image quality are unreliable, the confidence of the attribute set below the quality threshold can be directly set to a lower value.

[0059] (2) If yes, determine whether the status information of the traffic lights indicated by each sub-attribute in the attribute set is consistent.

[0060] For example, the shapes of the light frame and the light bulb of the same traffic light are usually similar. If the shape of the light frame detected is rectangular and the shape of the light bulb is round, then the two are inconsistent.

[0061] (2) If they are consistent, determine that the confidence of the sub-attribute in the attribute set is higher than a preset confidence threshold.

[0062] (4) If they are inconsistent, the sub-attributes in the attribute set are corrected based on the lighting conditions of the scene image corresponding to the target detection area.

[0063] For example, if the color of the light frame is inconsistent with that of the light bulb, if the scene image is shot under poor lighting conditions, such as shooting at night, since the light frame does not emit light, when the light bulb is lit, the color of the light frame is corrected based on the color of the light bulb.

[0064] (5) If, after correction, the state information of the traffic light indicated by each sub-attribute in the attribute set is consistent, it is determined that the confidence of the sub-attribute in the attribute set is higher than the preset confidence threshold. For example, if the color of the light frame and the light bulb is consistent after correction, and other sub-attributes of the traffic light are also consistent, it can be considered that the sub-attribute in the attribute set is relatively credible, and its confidence is determined to be higher than the preset confidence threshold.

[0065] Step S212: determining the final detection result of the traffic light based on the attribute information whose confidence level is higher than a preset confidence level threshold.

[0066] The above method uses a smaller and higher quality detection area to detect traffic lights, and screens the initial detection results based on the image status of the detection area, and determines the initial detection results with higher confidence as the final detection results, thereby improving the accuracy of the detection results, thereby improving the driving safety of autonomous driving vehicles to a certain extent.

[0067] The embodiment of the present invention also provides another method for detecting a traffic signal light. Figure 1 The method is implemented based on the method shown. The method aims to solve the following technical problems:

[0068] 1. When detecting traffic lights based on the entire vehicle image, the size of the input detection model is too large, which consumes a lot of computing power and video memory resources. In addition, since the original image is too large, the image resolution used for detection is only half of the original image, so the detection accuracy is also low.

[0069] 2. Solve the problem of poor model universality caused by differences in resolution, size, and aspect ratio of different cameras (i.e. the "forward-facing camera" mentioned above). In order to speed up, detection based on the entire image needs to be adjusted to a uniform size by interpolation, which will destroy the original aspect ratio and distortion pattern of the image, making the universality of the data and model worse.

[0070] 3. When the driving area and lighting conditions vary greatly, the brightness and reflection of the observed traffic lights vary greatly, resulting in the detection model being unable to be universally applied.

[0071] This method uses semantic map information to reduce the detection range of traffic lights, improve model accuracy and versatility, and reduce resource usage. When the semantic map is already labeled, the whole image detection can be changed to detection around the semantic map annotation, skipping meaningless areas and avoiding the loss of accuracy caused by image scaling. At the same time, reducing the detection range can also provide more accurate classification of image quality and lighting conditions in small areas.

[0072] The method is mainly implemented through the following steps:

[0073] Step 1: Project the traffic light annotations on the semantic map to the scene image and generate candidate detection areas.

[0074] Traffic lights (also called "traffic lights") are marked as cuboids on the semantic map. By querying the semantic map, the coordinates of the eight vertices of the traffic light in the three-dimensional world can be obtained. The traffic light can be projected onto the image through the following steps and parameters:

[0075] 1. The vertex coordinates of the three-dimensional world are transformed into the camera coordinate system.

[0076] 2. Transform the coordinates in the camera coordinate system to a two-dimensional image.

[0077] 3. Correct the projection through the distortion model and remove points that are not completely within the scene image.

[0078] 4. Take the circumscribed rectangle of the eight vertices of each traffic light after projection.

[0079] When the size of the scene image is 1920*1600, a 400*400 area can be taken as a candidate detection area with the center of the obtained circumscribed rectangle as the center, which greatly reduces the size of the detection area.

[0080] Step 2: Filter candidate detection areas according to on-board resources (equivalent to the above-mentioned "shooting parameters" and "area content") and pre-process the candidate areas.

[0081] Some complex intersections have more than a dozen traffic lights (left / center / right / U-turn / brt / pedestrian lights), which may generate dozens of candidate detection areas when projected onto the six forward-facing cameras. Due to limited computing resources on the vehicle, the number of candidate areas needs to be limited.

[0082] There are many factors that affect the selection of candidate regions, such as:

[0083] 1. Try not to miss any light.

[0084] 2. Prioritize candidate areas with telephoto lenses (30 degrees).

[0085] 3. Prioritize candidate areas with more traffic lights.

[0086] 4. Try to avoid selecting ambiguous candidate areas.

[0087] 5. Avoid selecting areas that will cause traffic lights to be truncated.

[0088] 6. Avoid choosing areas where lights are blocked.

[0089] 7. Under the same conditions, choose areas with more redundant lights.

[0090] 8. Under the same conditions, try to select an area that is similar to the previous frame to avoid state jumps.

[0091] By comprehensively considering the above conditions and scoring the areas according to the weights, and then sorting them from high to low according to the scores, the optimal detection area (equivalent to the above-mentioned "target detection area") can be selected, and then the optimal detection area can be used as the model input.

[0092] Step 3: Use a neural network to detect the traffic light frame and light bulb in the preprocessed candidate detection area, and give a regional image quality classification (equivalent to the above-mentioned "image status").

[0093] This module uses a neural network to detect the input area. Specifically, a two-stage detector feature map pyramid network can be used for detection.

[0094] Step 4: Finally, based on the image quality of the candidate area, the detection results are post-processed and fused.

[0095] In the previous step, a variety of results and attributes are output, which need to be processed comprehensively in this step, such as:

[0096] 1. Some are exclusive, for example, the detection results of blurred areas are often unreliable.

[0097] 2. Sometimes the results are contradictory, such as the color of the bulb and the light frame are inconsistent.

[0098] 3. For uncertain detection results, other information can be combined to make judgments. For example, it is often impossible to detect lights that are not turned on at night.

[0099] The above method has the following beneficial effects:

[0100] 1. Use semantic map information to reduce the detection range, save resources, and speed up detection.

[0101] 2. Output information such as light frame, light bulb, image quality, etc. at the same time, and give detailed details such as color, shape, number, etc. for each traffic light.

[0102] 3. Comprehensively process multiple information to solve the problem of poor detection effect when the image quality is low, and the problem of being unable to detect black (unlit) traffic lights at night.

[0103] Corresponding to the above method embodiment, the embodiment of the present invention provides a detection device for a traffic light, such as Figure 3 As shown, the device comprises:

[0104] The candidate detection region acquisition module 302 is used to acquire multiple candidate detection regions from multiple scene images including traffic lights; wherein the candidate detection regions include at least part of the traffic lights; and the size of the candidate detection regions is smaller than the size of the scene images;

[0105] The target detection region determination module 304 is used to determine a target detection region that meets a preset condition from among the candidate detection regions; wherein the preset condition is determined based on a shooting parameter of a scene image corresponding to the candidate detection region and / or a region content in the candidate detection region;

[0106] The detection module 306 is used to input the target detection area into a preset detection model and output the initial detection result of the traffic light and the image status of the target detection area;

[0107] The final detection result determination module 308 is used to determine the final detection result of the traffic light based on the initial detection result and the image status of the target detection area.

[0108] Furthermore, the above-mentioned candidate detection area acquisition module is also used to: for each scene image, based on the position annotation information of the traffic light in the semantic map corresponding to the scene image, project the traffic light area indicated by the position annotation information onto the scene image to obtain the projection area in the scene image; and determine the projection area as the candidate detection area.

[0109] Furthermore, the above-mentioned position annotation information includes the vertex coordinates of the three-dimensional traffic light area in the semantic map; the above-mentioned candidate detection area acquisition module is also used to: based on the three-dimensional coordinate system of the semantic map and the camera coordinate system corresponding to the vehicle-mounted camera, convert the vertex coordinates of the three-dimensional traffic light area into position coordinates in the camera coordinate system; based on the camera coordinate system and the plane coordinate system corresponding to the scene image, transform the position coordinates in the camera coordinate system into position coordinates in the plane coordinate system; determine the circumscribed rectangle corresponding to the position coordinates in the plane coordinate system as the projection area in the scene image.

[0110] Furthermore, the above-mentioned candidate detection area acquisition module is also used to: input the scene image into a pre-trained target detection model, and output multiple target areas in the scene image; the target area includes the detection target whose category is traffic light detected by the target detection model; for each target area, based on the traffic light information in the related image of the scene image, it is judged whether the detection target is a traffic light; the related image is taken by the on-board camera for the scene corresponding to the scene image; the shooting time of the related image is earlier than the shooting time of the scene image; if so, the target area is determined as a candidate detection area.

[0111] Furthermore, the above-mentioned shooting parameters include a shooting angle of a scene image; the preset conditions specifically include: a detection state parameter determined based on the shooting parameters of the scene image corresponding to the candidate detection area and the regional content in the candidate detection area is greater than a preset parameter threshold; the detection state parameter characterizes the integrity and credibility of the detection result of the candidate detection area; the above-mentioned target detection area determination module is also used to: for each candidate detection area, determine the detection state parameter of the candidate detection area based on the shooting angle of the scene image corresponding to the candidate detection area, the display completeness of the traffic light indicated by the regional content, and the resolution of the candidate detection area; and determine the candidate detection area whose detection state parameter is greater than the preset parameter threshold as the target detection area.

[0112] Furthermore, the above-mentioned detection module is also used to: input the target detection area into a preset detection model, extract the attribute characteristics of the traffic lights in the target detection area and the image characteristics of the target detection area through the detection model, determine the initial detection result of the traffic lights based on the attribute characteristics of the traffic lights, and determine the image state of the target detection area based on the image characteristics; the detection model is established based on the feature graph pyramid network.

[0113] Furthermore, the above-mentioned initial detection results include attribute information of multiple traffic lights; the above-mentioned final detection result determination module is also used to: determine the confidence of each attribute information based on the attribute information of multiple traffic lights and the image state corresponding to the attribute information; determine the final detection result of the traffic light based on the attribute information whose confidence is higher than a preset confidence threshold.

[0114] Furthermore, the above-mentioned attribute information includes multiple groups of attribute sets of traffic lights; each group of attribute sets includes multiple sub-attributes corresponding to the same traffic light; the image state includes the lighting conditions of the scene image corresponding to the target detection area and the image quality of the target detection area; the above-mentioned final detection result determination module is also used to: for each group of attribute sets, determine whether the image quality of the target detection area corresponding to the attribute set is higher than a preset quality threshold; if so, determine whether the state information of the traffic lights indicated by each sub-attribute in the attribute set is consistent; if consistent, determine that the confidence of the sub-attribute in the attribute set is higher than the preset confidence threshold; if inconsistent, correct the sub-attributes in the attribute set based on the lighting conditions of the scene image corresponding to the target detection area; if after correction, the state information of the traffic lights indicated by each sub-attribute in the attribute set is consistent, determine that the confidence of the sub-attribute in the attribute set is higher than the preset confidence threshold.

[0115] This embodiment also provides an electronic device, including a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the above-mentioned traffic light detection method.

[0116] See also Figure 4 As shown, the electronic device includes a processor 100 and a memory 101. The memory 101 stores machine executable instructions that can be executed by the processor 100. The processor 100 executes the machine executable instructions to implement the above-mentioned traffic light detection method.

[0117] Further, Figure 4 The electronic device shown further includes a bus 102 and a communication interface 103 , and the processor 100 , the communication interface 103 and the memory 101 are connected via the bus 102 .

[0118] The memory 101 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 103 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used. The bus 102 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0119] The processor 100 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 100. The above processor 100 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present invention can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 101, and the processor 100 reads the information in the memory 101 and completes the steps of the method of the above embodiment in combination with its hardware.

[0120] This embodiment further provides a machine-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the above-mentioned traffic light detection method.

[0121] A traffic light detection method, device and electronic device provided in an embodiment of the present invention include a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the method described in the previous method embodiment. The specific implementation can be found in the method embodiment and will not be repeated here.

[0122] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0123] In addition, in the description of the embodiments of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, and it can be the internal communication of two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0124] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0125] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0126] Finally, it should be noted that the above embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above embodiments, those skilled in the art should understand that any person skilled in the art can still modify the technical solutions recorded in the above embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for detecting a traffic light, characterized in that: The method comprises: Acquire multiple candidate detection areas from multiple scene images including traffic lights; wherein the candidate detection areas include at least part of the traffic lights; and the size of the candidate detection areas is smaller than the size of the scene images; Determine a target detection area that meets a preset condition from the candidate detection areas; wherein the preset condition is determined based on a shooting parameter of a scene image corresponding to the candidate detection area and / or an area content in the candidate detection area; Inputting the target detection area into a preset detection model, and outputting an initial detection result of the traffic light and an image state of the target detection area; Determining a final detection result of the traffic light based on the initial detection result and the image state of the target detection area; The initial detection result includes attribute information of multiple traffic lights; the attribute information includes multiple attribute sets of traffic lights; each attribute set includes multiple sub-attributes corresponding to the same traffic light; the image state includes the lighting conditions for shooting the scene image corresponding to the target detection area and the image quality of the target detection area; The step of determining the final detection result of the traffic light based on the initial detection result and the image state of the target detection area includes: For each attribute set, determining whether the image quality of the target detection area corresponding to the attribute set is higher than a preset quality threshold; If yes, determining whether the state information of the traffic lights indicated by each of the sub-attributes in the attribute set is consistent; If they are consistent, determining that the confidence of the sub-attribute in the attribute set is higher than a preset confidence threshold; If they are inconsistent, correcting the sub-attributes in the attribute set based on the lighting conditions of the scene image corresponding to the target detection area; If, after correction, the state information of the traffic lights indicated by each of the sub-attributes in the attribute set is consistent, determining that the confidence of the sub-attribute in the attribute set is higher than a preset confidence threshold; Determining a final detection result of the traffic light based on attribute information having a confidence level higher than a preset confidence level threshold; The shooting parameters include the shooting angle of the scene image; the preset conditions specifically include: the detection state parameter determined based on the shooting parameters of the scene image corresponding to the candidate detection area and the area content in the candidate detection area is greater than a preset parameter threshold; the detection state parameter represents the integrity and credibility of the detection result of the candidate detection area; The step of determining a target detection area that meets a preset condition from the candidate detection areas comprises: For each of the candidate detection areas, determining a detection state parameter of the candidate detection area based on a shooting angle of a scene image corresponding to the candidate detection area, a display completeness of a traffic light indicated by content of the area, and a resolution of the candidate detection area; The candidate detection area whose detection state parameter is greater than a preset parameter threshold is determined as the target detection area.

2. The method according to claim 1, characterized in that The step of obtaining a plurality of candidate detection regions from a scene image including a traffic light comprises: For each scene image, based on the position annotation information of the traffic light in the semantic map corresponding to the scene image, project the traffic light area indicated by the position annotation information onto the scene image to obtain the projection area in the scene image; The projection area is determined as a candidate detection area.

3. The method according to claim 2, characterized in that The position annotation information includes vertex coordinates of a three-dimensional traffic light area in a semantic map; The step of projecting the traffic light area indicated by the position annotation information onto the scene image based on the position annotation information of the traffic light in the semantic map corresponding to the scene image to obtain the projection area in the scene image comprises: Based on the three-dimensional coordinate system of the semantic map and the camera coordinate system corresponding to the vehicle-mounted camera, converting the vertex coordinates of the three-dimensional traffic light area into position coordinates in the camera coordinate system; Based on the camera coordinate system and the plane coordinate system corresponding to the scene image, transforming the position coordinates in the camera coordinate system into the position coordinates in the plane coordinate system; The circumscribed rectangle corresponding to the position coordinates in the plane coordinate system is determined as the projection area in the scene image.

4. The method according to claim 1, characterized in that: The step of obtaining a plurality of candidate detection regions from a scene image including a traffic light comprises: Inputting the scene image into a pre-trained target detection model, and outputting a plurality of target regions in the scene image; the target regions include detection targets of the traffic light category detected by the target detection model; For each of the target areas, judging whether the detection target is a traffic light based on traffic light information in a related image of the scene image; the related image is captured by a vehicle-mounted camera for a scene corresponding to the scene image; and the shooting time of the related image is earlier than the shooting time of the scene image; If yes, the target area is determined as a candidate detection area.

5. The method according to claim 1, characterized in that The step of inputting the target detection area into a preset detection model and outputting the initial detection result of the traffic light and the image state of the target detection area comprises: The target detection area is input into a preset detection model, and the attribute characteristics of the traffic light in the target detection area and the image characteristics of the target detection area are extracted through the detection model. The initial detection result of the traffic light is determined based on the attribute characteristics of the traffic light, and the image state of the target detection area is determined based on the image characteristics. The detection model is established based on a feature graph pyramid network.

6. A detection device for a traffic light, characterized in that: The device comprises: A candidate detection region acquisition module, used to acquire a plurality of candidate detection regions from a plurality of scene images including traffic lights; wherein the candidate detection regions include at least a portion of the traffic lights; and the size of the candidate detection regions is smaller than the size of the scene images; A target detection area determination module, used to determine a target detection area that meets a preset condition from the candidate detection areas; wherein the preset condition is determined based on a shooting parameter of a scene image corresponding to the candidate detection area and / or an area content in the candidate detection area; A detection module, used for inputting the target detection area into a preset detection model, and outputting an initial detection result of the traffic light and an image state of the target detection area; A final detection result determination module, used to determine the final detection result of the traffic light based on the initial detection result and the image state of the target detection area; The initial detection result includes attribute information of multiple traffic lights; the attribute information includes multiple attribute sets of traffic lights; each attribute set includes multiple sub-attributes corresponding to the same traffic light; the image state includes the lighting conditions for shooting the scene image corresponding to the target detection area and the image quality of the target detection area; The final detection result determination module is also used for: For each attribute set, determining whether the image quality of the target detection area corresponding to the attribute set is higher than a preset quality threshold; If yes, determining whether the state information of the traffic lights indicated by each of the sub-attributes in the attribute set is consistent; If they are consistent, determining that the confidence of the sub-attribute in the attribute set is higher than a preset confidence threshold; If they are inconsistent, correcting the sub-attributes in the attribute set based on the lighting conditions of the scene image corresponding to the target detection area; If, after correction, the state information of the traffic lights indicated by each of the sub-attributes in the attribute set is consistent, determining that the confidence of the sub-attribute in the attribute set is higher than a preset confidence threshold; Determining a final detection result of the traffic light based on attribute information having a confidence level higher than a preset confidence level threshold; The shooting parameters include the shooting angle of the scene image; the preset conditions specifically include: the detection state parameter determined based on the shooting parameters of the scene image corresponding to the candidate detection area and the area content in the candidate detection area is greater than a preset parameter threshold; the detection state parameter represents the integrity and credibility of the detection result of the candidate detection area; The target detection area determination module is also used for: For each of the candidate detection areas, determining a detection state parameter of the candidate detection area based on a shooting angle of a scene image corresponding to the candidate detection area, a display completeness of a traffic light indicated by content of the area, and a resolution of the candidate detection area; The candidate detection area whose detection state parameter is greater than a preset parameter threshold is determined as the target detection area.

7. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the traffic signal light detection method according to any one of claims 1 to 5.

8. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the traffic signal light detection method according to any one of claims 1 to 5.

Citation Information

Patent Citations

  • Traffic light state identification method and device, electronic equipment and storage medium

    CN111428663A

  • Countdown signal lamp detection method and device, electronic equipment and storage medium

    CN111507204A

  • Traffic signal lamp detection method and device, electronic equipment and storage medium

    CN112288031A

  • Traffic signal lamp detection method and device, and automatic driving automobile

    CN112289021A

  • Traffic signal lamp identification method and system

    CN112380973A