Procedures for recognizing traffic signs

Abstract

The invention relates to a method for recognizing traffic signs, wherein - image capture is carried out using cameras arranged on a large number of vehicles, in which images of the respective vehicle environment are captured, - a capture time is determined for each image capture, - in image processing, traffic signs and their regulatory content that may be present in the images can be recognized, - the recognized traffic signs and their regulatory content are aggregated as traffic sign information over a defined aggregation period, - for each geographical position where a traffic sign was detected during the aggregation period, a chronological sequence of image captures is determined based on the capture times, - an analysis is performed based on the temporal sequence for each geographical position to determine whether traffic sign information has changed within the aggregation period and which traffic sign with which associated regulatory content is located at a respective geographical position at the end of the aggregation period, and - valid traffic sign information is merged with map information already present in a digital road map.

Description

[0001] The invention relates to a method for recognizing traffic signs.

[0002] It is generally known from the state of the art that it is not always possible to reliably recognize traffic signs using camera systems in a vehicle. Therefore, it is known that camera detection of traffic signs is compared with map data and combined from all sources to create an assumption of currently valid traffic signs, such as speed limits. For this purpose, the map data is kept up-to-date by evaluating numerous individual observations from vehicle sensors in various vehicles.

[0003] From DE 10 2021 005 438 A1, a method for determining route attributes in a road map of a vehicle system from sign observations is known, in which a traffic sign detected at a specific point is transmitted to a navigation device and assigned as a route attribute to a predefined section of the road map. The road sign is detected by a detection unit positioned in a vehicle moving along the road section, whereby all traffic signs detected by the vehicle's detection units in a specific area within a predefined period are collected. The points determined by the traffic signs, including the direction of travel of the vehicles, are assigned to a road network of the existing road map.From a collection of point-source information, road sections influenced by the respective traffic signs are selected and assigned the route attributes resulting from the sign observations. These route attributes are then made available to the vehicle's systems for use.

[0004] Furthermore, a method for camera-based recognition and verification of traffic signs is known from DE 10 2023 000 356 A1. In this method, images of the surroundings of numerous vehicles are captured using cameras mounted on them. Traffic signs in the respective vehicle environment are then identified through image analysis, and the identified traffic signs are statistically evaluated to determine the applicable speed limit for vehicles on a given road section. The speed limit determined by the camera-based traffic sign recognition for a given road section is then compared with the actual speeds driven by vehicles on that section.The comparison identifies plausible and implausible speed limits, whereby implausible speed limits are not provided to other vehicle systems, remain unused when digital road maps are updated, and are suppressed in camera-based traffic sign recognition.

[0005] The present invention is based on the objective of providing a novel method for the recognition of traffic signs.

[0006] The problem is solved according to the invention by a method which has the features specified in claim 1.

[0007] Advantageous embodiments of the invention are the subject of the dependent claims.

[0008] In the inventive method for the recognition of traffic signs - Image capture is carried out using a large number of cameras arranged on vehicles, in which images of the respective vehicle environment are captured, - A capture time is determined for each image capture, - In image processing, traffic signs and their regulatory content, if present in the images, are recognized, - the recognized traffic signs and their regulatory content are aggregated as traffic sign information over a defined aggregation period, - For each geographical position where a traffic sign was detected during the aggregation period, a chronological sequence of image captures is determined based on the capture times. - An analysis is performed based on the temporal sequence for each geographical position to determine whether traffic sign information has changed within the aggregation period and which traffic sign with which associated regulatory content is located at a respective geographical position at the end of the aggregation period, and - Valid traffic sign information is merged with map information already present in a digital road map.

[0009] In contrast to previously described, prior art methods that aggregate recorded data over a defined period, the present method differentiates between individual traffic sign recognitions occurring at the beginning or end of the aggregation period by considering the respective recording time and determining the temporal sequence of image acquisitions. Particularly in geographical areas with a relatively low number of vehicles, the aggregation period is chosen to be correspondingly long, for example, four weeks, to obtain statistically significant data. Even with such long aggregation periods, the present method can reliably detect situations where changes occur within the aggregation period, such as the removal or replacement of existing traffic signs or the installation of new ones.The method enables particularly reliable and fast recognition of traffic signs and route attributes from measurements or image captures carried out using the cameras.

[0010] In one possible implementation of the procedure, all traffic sign information collected during the aggregation period is grouped, with a group being created for each geographic location on the road map and the traffic sign information collected at that location being assigned to the corresponding group. Such grouping enables efficient storage and processing of the traffic sign information.

[0011] In another possible iteration of the procedure, the analysis is carried out separately for each group. This allows for a particularly efficient analysis of the traffic sign information.

[0012] In another possible embodiment of the procedure, the analysis is carried out using at least one trained artificial neural network and / or at least one statistical method, so that the analysis can be carried out automatically, with little effort and at the same time with particular reliability.

[0013] In another possible embodiment of the method, a test drive is conducted at the end of the aggregation period during the training of at least one artificial neural network. During this drive, actual traffic signs present at the geographical locations are manually recorded and used as ground truth. This allows for particularly accurate and reliable training, enabling the artificial neural network to quickly recognize changes in traffic signs.

[0014] In another possible embodiment of the method, traffic sign information from an immediately preceding aggregation period is used as ground truth in the training of at least one artificial neural network for a current aggregation period, provided that this information was recorded in both the current and the immediately preceding aggregation periods. Such an embodiment enables automated training with high reliability, allowing the artificial neural network to quickly detect changes to traffic signs.

[0015] In another possible implementation of the procedure, the length of the aggregation period is variably defined. This allows, for example, the aggregation period to be adjusted to the number of surveys of a given geographical area to achieve a sufficient number of image captures of traffic signs. Conversely, the acquisition of an exceptionally large number of traffic sign image captures can be avoided. Thus, on the one hand, a sufficient amount of traffic sign information is always available for the procedure to be carried out, and on the other hand, the procedure can be executed particularly efficiently by limiting the amount of traffic sign information.

[0016] In another possible embodiment of the process, a quality measure is determined for each image capture and for each traffic sign information derived from it. If a predefined target quality measure is not met, the respective traffic sign information is rejected as implausible. Thus, traffic sign information classified as unreliable is disregarded during traffic sign recognition.

[0017] In another possible embodiment of the procedure, a reliability score is determined for each image capture and taken into account when verifying the plausibility of traffic sign information derived from that image capture. This allows for an increase in the reliability of traffic sign recognition.

[0018] In another possible embodiment of the procedure, the reliability measure is determined based on attributes, where the following are considered attributes: - a sensor generation of the respective camera used for image capture and / or - lighting conditions present during image capture and / or - weather conditions present during image capture and / or - Quality measures from image processing These attributes have proven to be particularly suitable for determining the reliability measure with exceptional accuracy and reliability.

[0019] Exemplary embodiments of the invention are explained in more detail below with reference to a drawing.

[0020] This shows: Fig. 1. Schematically, a sequence of steps in a procedure for recognizing traffic signs.

[0021] In the only Fig. Figure 1 shows a possible embodiment of a method for recognizing traffic signs.

[0022] To reliably recognize traffic signs using camera systems in a vehicle, the present method is designed to compare camera detections of traffic signs with map data and combine them from all sources to create a fused assumption of currently valid traffic signs, such as speed limits. For this purpose, the map data is kept up-to-date by evaluating numerous individual observations from vehicle sensors in various vehicles.

[0023] In a process step S1, image acquisitions are first carried out using a large number of cameras mounted on vehicles, capturing images of the respective vehicle's surroundings. A specific time is determined for each image acquisition.

[0024] These captured images, along with their corresponding capture timestamps, are fed to a processing device, which may be a central computing unit, such as a backend server. In a process step S2, the processing device identifies any traffic signs and their regulatory content that may be present in the images.

[0025] Furthermore, in a process step S3, the processing unit aggregates the recognized traffic signs and their regulatory content as traffic sign information over a defined aggregation period.

[0026] In a process step S4, for each geographical position where a traffic sign was detected during the aggregation period, a temporal sequence of image captures is determined based on the capture times.

[0027] In a process step S5, an analysis is performed based on the temporal sequence for each geographical position to determine whether traffic sign information has changed within the aggregation period and which traffic sign with which associated regulatory content is located at a respective geographical position at the end of the aggregation period.

[0028] This means that for each geographical location in a road network where a traffic sign was detected during the aggregation period, the temporal sequence of individual measurements taken by the cameras at that location is evaluated. This allows for an estimation of whether and which traffic sign will be present at the corresponding location at the end of the aggregation period. To achieve this, for each vehicle passing the location, the time of the measurement and information on whether and which traffic signs were detected there are determined.

[0029] In particular, this is carried out in two stages. First, traffic sign recordings from the entire aggregation period—that is, the traffic sign information derived from the recognized traffic signs and their regulatory content—are grouped into traffic sign recordings that are assigned to the same traffic sign at the corresponding geographical location. In a second step, all data collected at the geographical location are then analyzed over time to estimate the state of the corresponding traffic sign at the end of the aggregation period.

[0030] This analysis of the temporal sequence of image acquisitions primarily utilizes artificial intelligence methods, such as artificial neural networks trained in supervised or unsupervised learning. Alternatively or additionally, statistical methods can also be used for this analysis.

[0031] For example, an artificial neural network is trained by conducting a precise measurement drive at the end of an aggregation period, during which actual traffic signs are identified manually, i.e., by at least one occupant of at least one vehicle. These signs are then used as ground truth for training the network, which learns an expected result from an existing sequence of measurement data, i.e., image acquisitions.

[0032] It is also possible to use results obtained from a previously conducted procedure to train the network. If the same traffic sign was detected at a geographical location in both an initial aggregation period and an immediately subsequent aggregation period by the previous procedure, it can be assumed with a high degree of probability that the traffic sign was already present at the end of the first aggregation period and can therefore be used as ground truth.

[0033] If, on the other hand, a traffic sign was not recognized in the first aggregation period but was recognized in the following aggregation period, the traffic sign was most likely installed in the first aggregation period or in the following aggregation period.

[0034] If it is determined that something has changed on a sign, i.e., a traffic sign and thus the associated traffic sign information, a time at which the traffic sign was installed can subsequently be determined relatively accurately through statistical analysis of the individual image captures.

[0035] A single detection of a traffic sign may be a false positive. However, from the moment the traffic sign is actually installed, it should be detected by the camera in most cases when passing by.

[0036] To train the network, individual observations, i.e., single image captures, can be used within an aggregation period that ends at a start time plus a specified length. The length of the aggregation period can be a fixed time value or, for example, chosen such that a certain number of vehicles pass over a geographical location or through a geographical area occur within the aggregation period. The correct result derived by the artificial neural network from the image captures is then the corresponding traffic sign. Accordingly, the process can also be carried out if a traffic sign has been removed or replaced by another traffic sign at a given time.

[0037] The length of the aggregation period can also be varied to find out how much time or how many image captures are needed to reliably detect the change of a traffic sign.

[0038] If, during process step S5, the analysis determines that a traffic sign is located at the relevant position at the end of the aggregation period and is therefore valid, the corresponding valid traffic sign information is merged with map information already present in a digital road map in process step S6, and the road map is updated. Similarly, the digital road map is updated if a sign previously present at a relevant position no longer exists at the end of the aggregation period.

[0039] In addition to estimating the traffic signs valid at the end of the aggregation period, one possible implementation of the process generates a quality score for the recognized traffic signs, i.e., for each image capture and for each derived piece of traffic sign information. If the quality score falls below a predefined target level, the respective traffic sign information can be discarded as implausible and not used to update the road map. One reason for a traffic sign being deemed implausible is, for example, that it was not recognized in the image captures performed during the last few passes, but it cannot yet be definitively stated that it has been removed.

[0040] Additional attributes can be evaluated from each image capture to assess its reliability. Depending on the reliability, a decision can be made after a varying number of image captures as to whether a change to the traffic signs is necessary. Possible attributes include, for example: - a sensor generation used in the respective camera used for image acquisition, with newer sensors generally considered more reliable, and / or - lighting conditions present during image capture, for example during the day or at night, and / or - weather conditions present during image capture, such as fog, rain or snow, and / or - Quality measurements from the image processing performed. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] DE 10 2021 005 438 A1

[0003] DE 10 2023 000 356 A1

[0004]

Claims

[1] Methods for the recognition of traffic signs, wherein - image capture is carried out using cameras arranged on a large number of vehicles, in which images of the respective vehicle environment are captured, - a capture time is determined for each image capture, - in image processing, traffic signs and their regulatory content that may be present in the images can be recognized, - the recognized traffic signs and their regulatory content are aggregated as traffic sign information over a defined aggregation period, - for each geographical position where a traffic sign was detected during the aggregation period, a chronological sequence of image captures is determined based on the capture times, - an analysis is performed based on the temporal sequence for each geographical position to determine whether traffic sign information has changed within the aggregation period and which traffic sign with which associated regulatory content is located at a respective geographical position at the end of the aggregation period, and - valid traffic sign information is merged with map information already present in a digital road map. [2] Method according to claim 1, wherein all traffic sign information determined during the aggregation period is grouped, wherein a group is created for each geographical position in the road map and traffic sign information determined at the respective geographical position is assigned to the associated group. [3] Method according to claim 2, wherein the analysis is carried out separately for each group. [4] Method according to any of the preceding claims, wherein the analysis is carried out using at least one trained artificial neural network and / or at least one statistical method. [5] Method according to one of the preceding claims, wherein in a training of the at least one artificial neural network a measurement drive is carried out at the end of the aggregation period, in which traffic signs actually present at the geographical positions are manually recorded and used as ground truth. [6] Method according to one of the preceding claims, wherein in a training of the at least one artificial neural network for a current aggregation period, traffic sign information from an immediately preceding aggregation period is used as ground truth if it was recorded in the current aggregation period and the immediately preceding aggregation period. [7] Method according to one of the preceding claims, wherein the length of the aggregation period is variably specified. [8] Method according to one of the preceding claims, wherein a quality measure is determined for each image acquisition and for each traffic sign information derived therefrom, wherein if a predetermined target quality measure is not met, the respective traffic sign information is rejected as implausible. [9] Method according to one of the preceding claims, wherein a reliability measure is determined for each image acquisition and is taken into account when verifying the plausibility of traffic sign information obtained from the respective image acquisition. [10] Method according to claim 9, wherein the reliability measure is determined using attributes, wherein the attributes are - a sensor generation of the respective camera used for image capture and / or - lighting conditions present during image capture and / or - weather conditions present during image capture and / or - Quality measures from image processing are used.

Images