Method for tracking traffic lights, driving system and vehicle

Abstract

The invention relates to a method (4) for tracking traffic lights for a driving system (21). According to the method, first images are acquired by camera sensors of vehicle cameras (22) at a first time, first traffic light panels (1) are recognized on the acquired first images, static features of each of the recognized first traffic light panels (1) are determined; positions, in particular three-dimensional positions, of each of the recognized first traffic light panels (1) are determined and dynamic features of each of the recognized first traffic light panels (1) are determined. The same is repeated for second images acquired by the camera sensors at a second time, wherein the second time is later than the first time. Finally, the second traffic light panels (1) are associated with the first traffic light panels (1) based on the static features, the positions, and the dynamic features of the first traffic light panels (1) and the second traffic light panels (1).

Description

[0001] METHOD FOR TRACKING TRAFFIC LIGHTS, DRIVING SYSTEM AND VEHICLE

[0002] TECHNICAL FIELD

[0003] The invention relates to computer vision for driver assistance systems and / or for autonomous driving, in particular to a method for tracking traffic lights for a driving system, to a driving system that is adapted to track traffic lights and to a vehicle with said driving system.

[0004] BACKGROUND

[0005] Detection and recognition of traffic lights, in particular of those that are relevant to an ego vehicle, is a crucial part of any advanced driver assistance system and / or autonomous driving system. When there are multiple traffic lights present at various locations in an observed scene, like at a traffic junction, the detection, recognition and tracking of said traffic lights is a challenging perception task.

[0006] However, in such scenarios, tracking traffic lights is of utmost importance to derive stable output and to find the most relevant traffic lights for the ego vehicle. The detection and recognition of traffic lights is further complicated due to their small size, which makes it even more difficult to detect the traffic lights as their distance from the sensor increases. Further difficulties in the detection of traffic lights arise from fluctuations in the detection, from noisy images, and from blinking and / or changing of the lights of the traffic light.

[0007] SUMMARY

[0008] It is therefore an object of the present invention to provide a method for tracking traffic lights, a related driving system and a vehicle comprising such a driving system that overcome the above-mentioned problem, in particular that improve the tracking of traffic lights. The object of the present invention is solved by the subject-matter of the independent claims, wherein further embodiments are incorporated in the dependent claims.

[0009] According to an aspect of the invention, a method for tracking traffic lights for a driving system is provided.

[0010] The method may be a computer-implemented method. In particular, the method may be performed by a computing unit, in particular the computing unit of the driving system. The computing unit may comprise one or more processors, and the method may be performed by several processors by way of parallel computing and / or distributed computing.

[0011] Traffic lights may comprise one or more traffic signals and are signaling devices that may be positioned, e.g., at road intersections, pedestrian crossings, and other locations, in particular to control the flow of traffic. Traffic lights may have standard lights that are red, amber (or yellow), and / or green, but may also have other colors.

[0012] The driving system may be a driver assistance system, in particular an advanced driver assistance system, and / or an autonomous driving system. The driving system may use the information on the tracked traffic lights to determine to determine the color of the traffic light relevant for the ego vehicle and to take the appropriate actions. In this context, the ego vehicle is the vehicle to which the driving system is allocated. As an example, if the traffic light relevant for the ego vehicle is green, the driving system may keep the vehicle going, whereas, if the traffic light for the ego vehicle is red, the driving system will slow down the vehicle (in the case of an autonomous driving system) or perform an emergency brake (in the case of an advanced driver assistance system, given that the driver has not slowed the vehicle down). In the context of this application, the vehicle may be a road vehicle, in particular a car, a bus, a truck or a motorbike.

[0013] According to the method, first images are acquired by camera sensors of vehicle cameras. In particular, the vehicle may comprise at least one vehicle camera, and the sensor(s) of said camera(s) capture images. These images are then transferred, via a wired and / or wireless connection, to the computing unit that performs the method. The images may be directly transferred from the vehicle camera to the computing unit that performs the method, or may be first transferred to another computing unit, where some pre-processing of the images may be performed, and then transferred to the computing that performs the method. Said first images correspond to a first time. This first time may be the time at which the first images have been taken or at which the first images have been acquired - the latter being a delay time later than the time at which the images have been taken.

[0014] Then, first traffic light panels are recognized on the acquired first images. In this context, traffic light panels refer to a group of at least one traffic signal, wherein all traffic signals belonging to one group of traffic signals have, at least approximately, the same orientation, and are arranged adjacent to one another. In particular, the traffic signals within a traffic light panel are arranged vertically and / or horizontally next to one another. The traffic light panel may comprise a board that groups the traffic signals together, however also traffic signals that are adjacent to one another without such a board may be considered a traffic light panel.

[0015] The recognition of the traffic light panels on the images may be performed using computer vision algorithms. As an example, the recognition may be performed using feature detection algorithms such as edge detection algorithms with a subsequent analysis of the detected features, and / or using artificial intelligence, such as a trained deep neural network.

[0016] Static features of each of the recognized first traffic light panels are determined. In this context, static features refer to features of the traffic light panels that do not change over time.

[0017] Further, positions of each of the first traffic light panels are determined. In particular, said positions may be three-dimensional positions. The positions of the first traffic light panels may be given in a vehicle-fixed coordinate system, i.e., the positions of the first traffic light panels may change as the vehicle moves. Alternatively, the positions of the first traffic light panels may be given in a global coordinate system, i.e., the positions of the first traffic light panels do not change as the vehicle moves. The positions of the first traffic light panels may be determined based on their location in the first images. For three-dimensional positions, stereo vision cameras may be used, and / or the information in the first images may be combined with information obtained from other sensors such as radar sensors or lidar sensors.

[0018] Further, dynamic features of each of the recognized first traffic light panels are determined. In this context, dynamic features refer to features of the traffic light panels that change over time.

[0019] The method further comprises acquiring second images by the camera sensors of the vehicle cameras at a second time. Here, the second time is later than the first time. Like for the first time, the second time may be the time at which the second images have been taken or at which the second images have been acquired. The second time being later than the first time means that the time at which the second images have been taken is later than the time at which the first images have been taken or that the time at which the second images have been acquired is later than the time at which the first images have been acquired. One example is that the camera sensors capture an image stream, i.e., a video, and that the second time is one frame length of the video later than the first time, i.e., that the images taken at the second time are one frame later than the images taken at the first time.

[0020] The steps indicated above for the first images are repeated for the second images. In particular, second traffic light panels are recognized on the acquired second images; static features of each of the recognized second traffic light panels are determined; positions, in particular three-dimensional positions, of each of the recognized second traffic light panels are determined; and dynamic features of each of the recognized second traffic light panels are determined.

[0021] Finally, the second traffic light panels are associated with the first traffic light panels based on the static features, the positions, and the dynamic features of the first traffic light panels and the second traffic light panels. In this context, associating one of the second traffic light panels with one of the first traffic light panels means determining that said one of the second traffic light panels and said one of the first traffic light panels correspond to the same real-world traffic light panel. The above-described association allows for an intelligent, efficient and robust tracking of the traffic light panels. In comparison with tracking of active traffic signals, it provides better performance, e.g., in multiple complex scenarios. In particular, a better tracking stability is provided by maintaining a constant track identification for a longer duration. Further, tracking is easier for a traffic light panel when compared to single traffic signals, especially blinking traffic signals, since the number of features that can be derived from the traffic light panel is greater than that of active traffic signals. Moreover, it is easier to re-associate a track back to its original track with traffic light panel tracking. Also, the overall system complexity is reduced as the number of tracks gets reduced: instead of separate tracks for the single traffic signals, only one combined track for the traffic light panel is used. Further, subsequent modules of the driver system, e.g., traffic light assist, receive more consolidated data about the traffic lights which makes it easier to identify the most relevant traffic light for the ego vehicle. Moreover, many features of the traffic light panels, including the static features, positions and dynamic features may be provided to the subsequent modules of the driver system.

[0022] According to an embodiment which may be combined with the above-described embodiment or with any below described further embodiment, the static features of the first traffic light panels and the second traffic light panels comprise a type and / or an orientation of the traffic light panels. The type of the traffic light panel may comprise the number of traffic signals within the traffic light panel. The type of the traffic light panel may further comprise the arrangement of the traffic signals within the traffic light panel. As examples, there may be a single traffic signal in the traffic light panel, there may be two, three or four traffic signals in a row, or five traffic signals may be arranged in a so-called doghouse type, i.e., four of them as a square and one centered above said square. The type of the traffic light panel may further comprise whether or not there is a board grouping the traffic signals together. The orientation of the traffic light panels refers to whether the traffic signals in the traffic light panel are arranged vertically or horizontally. This applies, in particular, to traffic light panels where the traffic signals are arranged in a row. Further static features may include the outer shape of the traffic lights within the traffic light panel, in particular whether they are square or round. Other static features may comprise the color of the frame of the traffic signal and / or the color of the board. Since these static features do not change over time, an association using the static features is robust and easy.

[0023] According to an embodiment which may be combined with any above-described embodiment or with any below described further embodiment, the dynamic features of the first traffic light panels and the second traffic light panels comprise colors of active lights in the traffic light panels and / or shapes of the active lights in the traffic light panels. In this context, the active lights in the traffic light panels are the active traffic signals in the traffic light panels. In particular, the colors may be the standard colors red, amber (or yellow), and / or green, but also other colors may be observed. The shapes may include round traffic signals, arrow-shaped traffic signals and pedestrian and / or bicycle shaped traffic signals. While these dynamic features may change over time, they can still provide useful information for the traffic light panel association and further improve said association.

[0024] According to an embodiment which may be combined with any above-described embodiment or with any below described further embodiment, associating the second traffic light panels with the first traffic light panels comprises determining whether the static features of one of the second traffic light panels agree with the static features of one of the first traffic light panels. For this association, every one of the second traffic light panels may be compared to every one of the first traffic light panels. If it is determined that the static features of the one of the second traffic light panels disagree with the static features of the one of the first traffic light panels, it is inferred that the one of the second traffic light panels is not associated with the one of the first traffic light panels. On the other, if it is determined that the static features of the one of the second traffic light panels agree with the static features of the one of the first traffic light panels, it is inferred that said one of the second traffic light panels is a candidate for being associated with said one of the first traffic light panels. That is, the static features are used as a strong key for association between tracks over frames. As an example, a panel detected as vertical panel will be associated only with another vertical panel. The static features are more reliable than the dynamic features, hence they act as primary key in the track association. Since the static features do not change over time and are easy to detect, they provide for a very robust step in the association of the traffic light panels. Further, once it has been determined that the one of the second traffic light panels is not associated with the one of the first traffic light panels, further association steps for the pair of these two traffic light panels do not have to be performed, which saves computing time.

[0025] According to an embodiment which may be combined with any above-described embodiment or with any below described further embodiment, associating the second traffic light panels with the first traffic light panels comprises determining an association value based on the positions and on the dynamic features of the one of the second traffic light panels and on the positions and on the dynamic features of the one of the first traffic light panels. In particular, for the association value, a measure of agreement between the position of the one of the second traffic light panels and the one of the first traffic light panels may be computed, e.g., as a probability that - given these two positions - the second traffic light panel and the first traffic light panel correspond to the same real-world traffic light panel. Further, a dynamic association value may be determined that corresponds to a likelihood that the dynamic features of the first traffic light panel have changed to the dynamic features of the second traffic light panel. Then, the association value may be a function of the measure of agreement and the dynamic association value. Further, it is determined whether the association value fulfils an association condition. In particular, if the association value is a numerical value, the association condition may be exceeding or being below a predetermined threshold. If it is determined that the association value does not fulfil the association condition, it is inferred that the one of the second traffic light panels is not associated with the one of the first traffic light panels. On the other hand, if it is determined that the association value fulfils the association condition, it is inferred that the one of the second traffic light panels is associated with the one of the first traffic light panels. It is understood that the step in this embodiment may only be performed if it has been determined that the one of the second traffic light panels does not disagree with the one of the first traffic light panels. The association value then combines the information on the positions and the dynamic features of the traffic light panels. If there is high agreement between the positions, low agreement on the dynamic features may be sufficient to associate the traffic light panels. However, if there is a larger uncertainty between the positions, a higher agreement on the dynamic features may be necessary to associate the traffic light panels. In conclusion, said determination of the association is based on many features of the traffic light panels, leading to a robust association of said traffic light panels.

[0026] According to an embodiment which may be combined with any above-described embodiment or with any below described further embodiment, associating the second traffic light panels with the first traffic light panels comprises determining a measure of agreement between the position of one of the second traffic light panels and the position of one of the first traffic light panels. For example, said measure of agreement may be a likelihood that the one of the second traffic light panels and the one of the first traffic light panels correspond to the same real- world traffic light panel based on their positions. As a specific example, for both of the positions, a global coordinate and an uncertainty in said global coordinate may be provided. For example, the uncertainty in the global coordinate may be defined as a function of vehicle speed, vehicle acceleration and / or the location of the traffic light panel in the image frame, e.g., the closer the traffic light panel is to the edge of the frame, the higher the uncertainty. The above-mentioned likelihood then corresponds to whether or not the global coordinates of the two traffic light panels agree within their uncertainties. It is then determined whether the measure of agreement fulfils a predetermined condition. In the above example, the predetermined condition may be a position agreement threshold, and fulfilling the predetermined condition means that the likelihood that the one of the second traffic light panels and the one of the first traffic light panels correspond to the same real-world traffic light panel is greater than said position agreement threshold. If it is determined that the measure of agreement does not fulfil the predetermined condition, it is inferred that the one of the second traffic light panels is not associated with the one of the first traffic light panels. Hence, in this step, the information on the positions of the traffic light panels is used to determine the association between the second traffic light panels and the first traffic light panels. This position information is another good indicator on whether or not the traffic light panels are associated with one another. It is to be understood that this step may be only performed if it has been determined that the one of the second traffic light panels does not disagree with the one of the first traffic light panels, based on the static features. Further, if it is determined, based on the positions of the traffic light panels, that the one of the second traffic light panels is not associated with the one of the first traffic light panels, further steps relating these two traffic light panels may be skipped, saving computing time and resources.

[0027] According to an embodiment which may be combined with the above-described embodiment or with any below described further embodiment, associating the second traffic light panels with the first traffic light panels further comprises that, if it is determined that the measure of agreement fulfils the predetermined condition, a dynamic association threshold is set based on the measure of agreement. That is, the dynamic association threshold depends on the measure of agreement. In particular, if it is determined that there is a large likelihood that the one of the second traffic light panels is associated with the one of the first traffic light panels, the dynamic association threshold may be lower than if it is determined that there is only a small likelihood that the one of the second traffic light panels is associated with the one of the first traffic light panels. Then, a dynamic association value is determined based on the dynamic features of the one of the second traffic light panels and the dynamic features of the one of the first traffic light panels and is compared to the dynamic association threshold. If it is determined that the dynamic association value is below the dynamic association threshold, it is inferred that the one of the second traffic light panels is not associated with the one of the first traffic light panels. On the other hand, if it is determined that the dynamic association value exceeds the dynamic association threshold, it is inferred that the one of the second traffic light panels is associated with the one of the first traffic light panels. Hence, for inferring that the one of the second traffic light panels is associated with the one of the first traffic light panels, all of the information on the traffic light panels has been taken into account, providing a very robust and reliable association result.

[0028] According to an embodiment which may be combined with the above-described embodiment or with any below described further embodiment, the dynamic association threshold is set as a monotonic function of the measure of agreement. That is, if the position-based association gives good confidence, the further association logic using the dynamic features is evaluated with a lower threshold and if the position-based association gives poor confidence, the further association logic using the dynamic features is evaluated with a higher threshold. As a particularly simple monotonic function, a step function may be chosen. This step provides an easy-to-implement way of combining the position information and the dynamic features of the traffic light panels.

[0029] According to an embodiment which may be combined with the above-described embodiment or with any below described further embodiment, the dynamic association value is calculated based on a number of colors of active lights that match between the one of the second traffic light panels and the one of the first traffic light panels and / or a number of shapes of active lights that match between the one of the second traffic light panels and the one of the first traffic light panels. As an example, the dynamic association value may be calculated as the following function: dav = (num_col_match * col_w + num_shape_match * shape_w) / num_active, wherein dav is the dynamic association value, num_col_match is the number of active lights for which the color of the active light matches between the first traffic light panel and the second traffic light panel, col_w is a weight for said color matching, num_shape_match is the number of active lights for which the shape of the active light matches between the first traffic light panel and the second traffic light panel, shape_w is a weight for said shape matching, and num_active is the number of active lights in the traffic light panel. num_active may be chosen for either the first traffic light panel or the second traffic light panel, col w and shape w may be, e.g., each 0.5. If col w + shape w = 1 , dav will be 1 when all the colors and shapes agree, i.e. , when the traffic signals have not changed between the first time and the second time. This calculation of the dynamic association value is easy and very fast to perform, reducing complexity of the algorithm and saving computing time.

[0030] According to an embodiment which may be combined with any above-described embodiment or with any below described further embodiment, inferring that the one of the second traffic light panels is not associated with any one of the first traffic light panels comprises assigning a new tracking identification to the one of the second traffic light panels. In other words, if it is determined that the one of the second traffic light panels does not correspond to any one of the previously detected first traffic light panels, it is considered to be a newly detected traffic light panel, to which a new tracking identification is assigned. This is an efficient way of assigning tracking identifications.

[0031] According to an embodiment which may be combined with any above-described embodiment or with any below described further embodiment, inferring that the one of the second traffic light panels is associated with the one of the first traffic light panels comprises assigning a tracking identification of the one of the first traffic light panels to the one of the second traffic light panels. Since only one tracking identification is assigned to the full traffic light panel, the number of tracking identifications is reduced compared to tracking each traffic signal separately, improving the efficiency of the tracking routines.

[0032] According to an embodiment which may be combined with any above-described embodiment or with any below described further embodiment, the method further comprises providing the static features, positions, dynamic features and / or tracking identification of the second traffic light panels to the driving system. Hence, the driving system receives robust tracking information on the traffic light panels, including many information needed for determining whether the traffic light panel is relevant for the ego vehicle and the status of the traffic light panel.

[0033] According to another aspect of the invention, a driving system is provided. The driving system comprises at least one vehicle camera and a computing unit and is configured to be operated according to the above description. The advantages and further embodiments correspond to those given in the above description.

[0034] According to an embodiment which may be combined with the above-described embodiment, the driving system is a driver assistance system and / or an autonomous driving system. In either case, the operation of the driving system is improved by tracking traffic light panels with a robust algorithm. According to yet another aspect of the invention, a vehicle is provided. The vehicle comprises a driving system according to the above description. Hence, the advantages and further embodiments correspond to those given in the above description.

[0035] BRIEF DESCRIPTION OF THE DRAWINGS

[0036] These and other aspects of the invention will be apparent from and elucidated further with reference to the embodiments described by way of examples in the following description and with reference to the accompanying drawings, in which

[0037] Figs. 1a to 1d show examples of traffic light panels with different static features and dynamic features;

[0038] Fig. 2 shows a flowchart of an embodiment of a method for tracking traffic lights; and

[0039] Fig. 3 show a schematic top view of an embodiment of a vehicle with a driving system.

[0040] In the figures, elements which correspond to elements already described may have the same reference numerals. Examples, embodiments or optional features, whether indicated as non-limiting or not, are not to be understood as limiting the invention as claimed.

[0041] DESCRIPTION OF EMBODIMENTS

[0042] Figures 1a to 1d show examples of traffic light panels 1 with different static features and dynamic features.

[0043] The traffic light panel 1 shown in Figure 1a comprises three traffic signals 2 that are arranged in row. Further, the traffic light panel 1 shown in Figure 1a is arranged vertically. Hence, static features of the traffic light panel 1 shown in Figure 1a are “three-in-a-row” and “vertical”. Further, the red light and the amber light are active and are both circular lights. Therefore, the dynamic features of the traffic light panel 1 in Figure 1a may be “red, amber” and “circular, circular”. Alternatively, the dynamic features may be labeled “red - circular, amber - circular”.

[0044] The traffic light panel 1 shown in Figure 1b also comprises three traffic signals 2 that are arranged in a row, however, this time the traffic light panel 1 is arranged horizontally. Hence, static features of the traffic light panel 1 shown in Figure 1 b are “three-in-a-row” and “horizontal”. Further, the green light is active and is a circular light. Therefore, the dynamic features of the traffic light panel 1 in Figure 1 b may be “green” and “circular”. Alternatively, the dynamic features may be labeled “green - circular”.

[0045] The traffic light panel 1 shown in Figure 1c is of a so-called doghouse type, i.e., four traffic signals 2 arranged in a square and one centered above the square. The traffic light panel 1 of the doghouse type may be defined to be arranged vertically. Hence, static features of the traffic light panel 1 shown in Figure 1c are “doghouse” and “vertical”. Further, the amber light on the left having the shape of a left arrow and the green light on the right having a circular shape are active. Therefore, the dynamic features of the traffic light panel 1 in Figure 1c may be “amber, green” and “left arrow, circular”. Alternatively, the dynamic features may be labeled “amber - left arrow, green - circular”.

[0046] The traffic light panel 1 shown in Figure 1d again has the static features “three-in- a-row” and “vertical” and the dynamic features “red” and “circular” or “red - circular”. The traffic light panel 1 shown in Figure 1d has a further static feature, namely, it comprises a board 3. Hence, the static feature “board: yes” may be added. For the traffic light panels 1 shown in Figures 1a - 1c, this static feature would be “board: no”.

[0047] Using the static features and dynamic features of the traffic light panels as well as their position, in particular their three-dimensional position, a method 4 for tracking traffic lights may be as shown in Figure 2.

[0048] First, a track list 5 is received. It is then determined 6 whether or not the track list 5 is empty. If the track list 5 is empty, newly recognized traffic light panels 1 cannot be associated with previously recognized traffic light panels 1 , hence new tracks will be created 7. For the creation 7 of said new tracks, a list 8 of detected traffic panels along with their static features, positions, and dynamic features will be used. If the track list 5 is not empty, an association 9 using static features of the traffic light panels 1 will be performed. The static features are features that do not change over time, e.g., the type and orientation of the traffic light panel. In this association, the static features of all the currently detected traffic light panels 1 with the traffic light panels 1 from the track list 5 are compared. Since the traffic light panels 1 from the track list 5 have been derived from previously detected traffic light panels 1 , the comparison is a comparison of the static features of currently detected traffic light panels 1 with previously detected traffic light panels. It is then determined 10 whether or not the static features match, i.e., whether the currently detected traffic light panel 1 has the same static features as one of the previously detected traffic light panels 2. If none of the traffic light panels 1 from the track list 5 match the currently detected traffic light panel 1 , the currently detected traffic light panel 1 is not associated with any of the previously detected traffic light panels 1 and a new track will be created 7. If, on the other hand, the static features of the currently detected traffic light panel 1 match with at least one of the traffic light panels 1 from the track list 5, an association 11 based on the positions, in particular the three-dimensional positions, of the traffic light panels 1 is performed. If it is determined 12 that the positions of the currently detected and previously detected traffic light panels 1 are incompatible, it is inferred that the currently detected traffic light panel 1 is not associated with any of the previously detected traffic light panels 1 and new tracks are created 7. If, on the other hand, it is determined that the position of the currently detected traffic light panel 1 is compatible with at least one of the previously detected traffic light panels 1 , the confidence of said compatibility, i.e., a measure of agreement, is determined and compared 13 to a threshold. If the confidence is higher than the threshold, a low dynamic association threshold 14, such as a value between 0.2 and 0.4, is set. If, on the other hand, the confidence is lower than the threshold, a high dynamic association threshold 15, such as a value between 0.4 and 0.7, is set. In the next step, a dynamic association value is calculated 16. As an example, the dynamic association value may be calculated as the following function: dav = (num_col_match * col_w + num_shape_match * shape_w) / num_active, wherein dav is the dynamic association value, num col match is the number of active lights for which the color of the active light matches between the currently detected traffic light panel and the previously detected traffic light panel, col_w is a weight for said color matching, num_shape_match is the number of active lights for which the shape of the active light matches between the currently detected traffic light panel and the previously detected traffic light panel, shape_w is a weight for said shape matching, and num_active is the number of active lights in the traffic light panel. num_active may be chosen for either the currently or previously detected traffic light panel. col_w and shape_w may be, e.g., each 0.5. If col_w + shape_w = 1 , dav will be 1 when all the colors and shapes agree, i.e., when the active traffic signals have not changed. Then, the dynamic association value is compared 17 to the dynamic association threshold. If the dynamic association value is less than the dynamic association threshold, it is determined that the currently detected traffic light panel 1 is not associated with the previously detected traffic light panel 1 , and new tracks have to be created 7. If, on the other hand, the dynamic association value is greater than the dynamic association threshold, it is determined that the currently detected traffic light panel 1 is associated with the previously detected traffic light panel 1 and the corresponding track is updated 18. For said update 18, further information from the list 8 of detected traffic panels may be used. The resulting updated track list 19 may then be sent to functions of a driving system, such as a driver assistance system or an autonomous driving system. With the above-described method, a robust and efficient way of tracking traffic lights is provided.

[0049] More generally, but not shown here, an association value may be computed based on the positions and on the dynamic features of currently detected and previously detected traffic light panels 1. Said association value may, e.g., be calculated as assoc val = dav / thresh (pos conf), wherein assoc val is the association value, dav is the dynamic association value described above, pos conf is the position confidence and thresh is a function of the position confidence, providing a dynamic association threshold. If thresh is a step function of the position confidence, and an association condition is chosen such that the association value is compared to one, this method is equivalent to the previously described method 4 of Figure 2. However, it is understood that many different functions may lead to a desirable result.

[0050] Finally, Figure 3 show a schematic top view of an embodiment of a vehicle 20 with a driving system 21 . Said driving system 21 may be a driver assistance system, in particular an advanced driver assistance system, and / or an autonomous driving system. The driving system 21 comprises at least one camera 22 that is connected to a computing unit 23. While the connection is shown as a wired connection, a wireless connection is also possible. The driving system 21 is configured to perform the method for tracking traffic lights described above and hence provides an improved tracking of traffic lights.

[0051] Other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from the study of the drawings, the disclosure, and the appended claims. In the claims, the word “comprising” does not exclude other elements or steps and the indefinite article “a” or “an” does not exclude a plurality. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. Any reference signs in the claims should not be construed as limiting the scope of the claims.

[0052] List of reference signs

[0053] 1 traffic light panel

[0054] 2 traffic signal

[0055] 3 board

[0056] 4 method

[0057] 5 track list

[0058] 6 determination

[0059] 7 creation

[0060] 8 list

[0061] 9 association

[0062] 10 determination

[0063] 11 association

[0064] 12 determination

[0065] 13 comparison

[0066] 14 low dynamic association threshold

[0067] 15 high dynamic association threshold

[0068] 16 creation

[0069] 17 comparison

[0070] 18 update

[0071] 19 updated track list

[0072] 20 vehicle

[0073] 21 driving system

[0074] 22 camera

[0075] 23 computing unit

Claims

Patent claims1 . Method (4) for tracking traffic lights for a driving system (21), comprising: acquiring first images by camera sensors of vehicle cameras (22) at a first time; recognizing first traffic light panels (1) on the acquired first images; determining static features of each of the recognized first traffic light panels (1 ); determining positions, in particular three-dimensional positions, of each of the recognized first traffic light panels (1); determining dynamic features of each of the recognized first traffic light panels (1); acquiring second images by the camera sensors at a second time, wherein the second time is later than the first time; recognizing second traffic light panels (1) on the acquired second images; determining static features of each of the recognized second traffic light panels (1); determining positions, in particular three-dimensional positions, of each of the recognized second traffic light panels (1); determining dynamic features of each of the recognized second traffic light panels (1); associating the second traffic light panels (1 ) with the first traffic light panels (1 ) based on the static features, the positions, and the dynamic features of the first traffic light panels (1) and the second traffic light panels (1 ).

2. Method (4) according to claim 1 , wherein the static features of the first traffic light panels (1) and the second traffic light panels (1) comprise a type and / or an orientation of the traffic light panels (1).

3. Method (4) according to claim 1 or 2, wherein the dynamic features of the first traffic light panels (1) and the second traffic light panels (1 ) comprise colors of active lights in the traffic light panels (1) and / or shapes of the active lights in the traffic light panels (1).

4. Method (4) according to any one of claims 1 to 3, wherein associating the second traffic light panels (1) with the first traffic light panels (1) comprises: determining whether the static features of one of the second traffic light panels (1 ) agree with the static features of one of the first traffic light panels (1 ); and based on determining that the static features of the one of the second traffic light panels (1 ) disagree with the static features of the one of the first traffic light panels (1), inferring that the one of the second traffic light panels (1 ) is not associated with the one of the first traffic light panels (1 ).

5. Method (4) according to any one of claims 1 to 4, wherein associating the second traffic light panels (1) with the first traffic light panels (1) comprises: determining an association value based on the positions and on the dynamic features of the one of the second traffic light panels (1 ) and on the positions and on the dynamic features of the one of the first traffic light panels (1 ); determining whether the association value fulfils an association condition; based on determining that the association value does not fulfil the association condition, inferring that the one of the second traffic light panels (1 ) is not associated with the one of the first traffic light panels (1 ); and based on determining that the association value fulfils the association condition, inferring that the one of the second traffic light panels (1 ) is associated with the one of the first traffic light panels (1).

6. Method (4) according to any one of claims 1 to 4, wherein associating the second traffic light panels (1) with the first traffic light panels (1) comprises: determining a measure of agreement between the position of one of the second traffic light panels (1) and the position of one of the first traffic light panels (1 ); determining whether the measure of agreement fulfils a predetermined condition; and based on determining that the measure of agreement does not fulfil the predetermined condition, inferring that the one of the second traffic light panels (1) is not associated with the one of the first traffic light panels (1 ).

7. Method (4) according to claim 6, wherein associating the second traffic light panels (1) with the first traffic light panels (1) further comprises: based on determining that the measure of agreement fulfils the predetermined condition: setting a dynamic association threshold based on the measure of agreement; determining a dynamic association value based on the dynamic features of the one of the second traffic light panels (1) and the dynamic features of the one of the first traffic light panels (1 ); comparing the dynamic association value to the dynamic association threshold; based on determining that the dynamic association value is below the dynamic association threshold, inferring that the one of the second traffic light panels (1 ) is not associated with the one of the first traffic light panels (1 ); and based on determining that the dynamic association value exceeds the dynamic association threshold, inferring that the one of the second traffic light panels (1) is associated with the one of the first traffic light panels (1).

8. Method (4) according to claim 7, wherein the dynamic association threshold is set as a monotonic function, more particularly a step function, of the measure of agreement.

9. Method (4) according to claim 7 or 8, wherein the dynamic association value is calculated based on a number of colors of active lights that match between the one of the second traffic light panels (1 ) and the one of the first traffic light panels (1 ) and / or a number of shapes of active lights that match between the one of the second traffic light panels (1 ) and the one of the first traffic light panels (1)-10. Method (4) according to any one of claims 4 to 9, wherein inferring that the one of the second traffic light panels (1 ) is not associated with any one of the first traffic light panels (1) comprises assigning a new tracking identification to the one of the second traffic light panels (1 ).11 . Method (4) according to claim 5 or any one of claims 7 to 10, wherein inferring that the one of the second traffic light panels (1 ) is associated with the one of the first traffic light panels (1 ) comprises assigning a tracking identification of the one of the first traffic light panels (1 ) to the one of the second traffic light panels (1).

12. Method (4) according to any one of claims 1 to 11 , wherein the method (4) further comprises providing the static features, positions, dynamic features and / or tracking identification of the second traffic light panels (1 ) to the driving system (21 ).

13. Driving system (21), comprising at least one vehicle camera (22) and a computing unit (23), wherein the driving system (21) is adapted to be operated according to the method (4) according to any one of claims 1 to 12.

14. Driving system (21) according to claim 13, wherein the driving system (21) is a driver assistance system and / or an autonomous driving system.

15. Vehicle (20), comprising a driving system (21) according to claim 13 or 14.

Images