Method, system, vehicle and computer program product for classifying traffic signs

Abstract

The invention relates to a method, system, vehicle, and computer program product for classifying traffic signs, such as those for road, rail, water, or air traffic. The method according to the invention comprises the steps (S10) receiving a digital representation of a traffic sign, (S20) determining a traffic sign based on the received digital representation, and (S30) classifying a traffic sign or determining that a traffic sign cannot be determined or cannot be determined with sufficient certainty.

Description

[0001] The invention relates to a method, system, vehicle and computer program product for classifying traffic signs for a vehicle.

[0002] Methods, systems, vehicles and computer program products for classifying traffic signs are known from the state of the art.

[0003] The method according to the invention has the features of claim 1, the system according to the invention has the features of claim 7, the vehicle according to the invention has the features of claim 8, and the computer program product according to the invention has the features of claim 9. The dependent claims describe advantageous embodiments.

[0004] A first aspect of the invention relates to a method for classifying a traffic sign, such as for road, rail, water, or air traffic. The method according to the invention comprises the steps (S10) receiving a digital representation of a traffic sign, (S20) determining a traffic sign based on the received digital representation, and (S30) classifying a traffic sign or determining that a traffic sign cannot be determined or cannot be determined with sufficient certainty.

[0005] Step (S20), "Determining a traffic sign," comprises the sub-steps (S20a) "Determining a probability for the presence of a characteristic of a traffic sign" and (S20b) "Determining a probability for the presence of a traffic sign." Step (S30), "Classifying a traffic sign," is carried out based on the characteristics and probabilities determined in step (S20), "Determining a traffic sign."

[0006] In one embodiment of the method according to the invention, in the sub-step Determining Features (S20a) probabilities for symbols, shapes, colors, alphanumeric characters and / or their arrangement contained in the depicted traffic sign are determined, and in the sub-step Determining a Traffic Sign (S20b) the features determined in this way are assigned to predefined combinations of features.

[0007] In one embodiment of the method according to the invention, in the sub-step "Determining Features" (S20a), a probability is determined for a digital representation belonging to a predefined subgroup of traffic signs, and in the sub-step "Determining a Traffic Sign" (S20b), a probability is determined for a digital representation being assigned to a traffic sign contained in a predefined subgroup. Alternatively, it is determined that the subgroup determined in the preceding step (S20a) does not include the traffic sign depicted in the digital representation. This has the advantage that the training of a neural network, such as a CNN, can be specifically optimized.

[0008] In one embodiment of the method according to the invention, in the sub-step "Determining Features" (S20a), interfering factors within the digital representation are determined. Interfering factors are those that impair the determination of features and traffic signs in a digital representation, such as low resolution, distortion, shadows, low contrast or color saturation due to weather or lighting conditions, partial obscuration due to objects, blurring of the representation, damage or weathering of the traffic sign.

[0009] In one embodiment of the method according to the invention, in the step classifying a traffic sign (S30), if a digital representation is determined to be unclassifiable or not classifiable with certainty, a message is generated indicating that a digital representation cannot be classified or cannot be classified with certainty.

[0010] In one embodiment of the method according to the invention, in one step checking a traffic sign (S40) depending on characteristics, probabilities and thresholds determined in previous steps determining a traffic sign (S20) and classifying a traffic sign (S30), one or more of the following sub-steps are carried out: In sub-step S40a, the step "Determine a traffic sign" (S20) is repeated. In sub-step S20a, characteristics are determined that differ from those determined in a previous iteration of sub-step S20. Sub-step S40b compares the results with a database containing traffic signs expected at a given location. Sub-step S40c defines the environment at a given location, such as a highway, country road, city, non-public area like a parking lot, parking garage, or company or port premises, intersection, roundabout, or construction zone, and identifies the traffic signs likely to be found there. In a sub-step (S40d), a comparison and / or combination with another digital representation of a traffic sign takes place, e.g., another representation that was captured from a different distance, at a different speed, or from a different angle than the previously analyzed digital representation. In a sub-step (S40e), the quality of a digital representation is improved by means of digital image processing and / or combination with another digital representation.

[0011] A second aspect of the invention relates to a system comprising a sensor, a processing unit, and a data communication interface. This system is configured to initiate the execution of a method according to the first aspect of the invention.

[0012] A third aspect of the invention relates to a vehicle. This vehicle comprises a system according to the second aspect of the invention.

[0013] A fourth aspect of the invention relates to a computer program product. This computer program product comprises computer-readable instructions which, when executed by one or more data processing units, cause one or more data processing units to execute a method according to the first aspect of the invention.

[0014] Exemplary embodiments of the invention are explained in more detail below. Fig. 1 an embodiment of the method according to the invention; Fig. Figure 2 shows an embodiment of the method according to the invention by means of parallel determination of features; Fig. 3. An example of a classification of traffic signs into subgroups; Fig. Figure 4 shows an embodiment of the method according to the invention by means of assignment to subgroups; Fig.5 an example of an assignment of safety levels to misclassifications of traffic signs; Fig. 6 a digital representation of a traffic sign “Mandatory direction of travel” straight ahead and Fig. 7 an embodiment of the method according to the invention.

[0015] Fig.Figure 1 shows an embodiment of the method according to the invention. In a first step S10 of the method according to the invention, a digital representation of a traffic sign is received. In a second step S20, a traffic sign is identified based on this received digital representation. In a first sub-step S20a, probabilities for the presence of features are determined. In a subsequent sub-step S20b, an overall probability for the presence of a traffic sign is determined. In a subsequent step S30, a final classification is carried out, or further actions are initiated, particularly if a final classification is not possible with sufficient certainty.

[0016] Fig.Figure 2 shows an embodiment of the method according to the invention using the example of a digital representation of a stop sign. In substep S20a, probabilities for the presence of predefined symbols, shapes, colors, alphanumeric characters, and their arrangement are determined using parallel, specialized feature determination modules, such as specialized CNNs. In the example shown, the probabilities for the presence of shape, frame (color), background color, type of symbols, and their meaning are determined.

[0017] A CNN (Convolutional Neural Network) is an artificial neural network. It is a concept in machine learning inspired by biological processes. The structure of a classic Convolutional Neural Network generally consists of one or more convolutional layers followed by a pooling layer. This unit can, in principle, be repeated any number of times.

[0018] In a subsequent step, S20b, the likely characteristics are compared with predefined combinations of characteristics. An index representing the overall probability of the presence of the traffic sign corresponding to the combination is then determined; in the example shown, this is a stop sign. This can be done, for example, by summing all the individual characteristic probabilities. In the example shown, this results in an index value of 4.39. Other calculation methods are also possible, such as averaging, multiplication, or determining the minimum. In the example shown, for instance, the overall probability of a stop sign being present is 87.8% (average), 79% (minimum), or 51.2% (multiplication).

[0019] Traffic signs can be divided into subgroups of similar traffic signs. Fig.Figure 3 shows an example of such a classification of road traffic signs into subgroups. In the example shown, these are the subgroups speed limits, other prohibitions, cancellation of prohibitions, direction of travel, hazard warnings, and signs with unique designs.

[0020] In a further embodiment of the method according to the invention, which is described in Fig.As shown in Figure 4, this method is used in substep S20a to determine the probability of a traffic sign belonging to a subgroup. In a subsequent substep S20b, a sign within the subgroup determined in the preceding substep S20a is identified using modules specialized for the respective subgroup, in particular specialized CNNs. An index of the overall probability for the presence of a stop sign is determined using the probabilities determined in substeps S20a and S20b, for example, by summation, averaging, multiplication, or finding the minimum.

[0021] Research on the present invention has shown that difficulties in classification are largely attributable to interfering factors in the digital representation of a traffic sign. Therefore, in one embodiment of the method according to the invention, the probabilities for the presence of interfering factors in the received digital representation are determined in substep S20a. Relevant interfering factors include, for example, difficult lighting conditions, harsh shadows, soiling, damage, obscuration, blurriness / shaky image, and low resolution.

[0022] In one embodiment of the invention, in the subsequent step S20b, the probabilities determined for the disturbance factors are directly included in the determination of an overall probability, e.g., by subtracting them from the previously described index by summation or by multiplication with the inverse probability (1-probability). In a further embodiment, a separate index is determined for the presence of disturbance factors.

[0023] As in Fig.As shown in Figure 1, a final classification takes place in a subsequent step S30. In the embodiment of the method according to the invention described here, it is checked whether the determined overall probability of a traffic sign is above a first predefined threshold value. In addition, it is checked whether the overall probability for other possible traffic signs is below a second predefined absolute or relative threshold value. In one embodiment of the method according to the invention, the selection of the first and second predefined threshold values ​​depends on a predefined safety level. As shown in Fig. As shown in Figure 5, in the embodiment shown here the respective safety level is determined, for example, by means of a predefined hazard risk resulting from a misclassification of traffic signs.

[0024] If both tests are successful, a final classification of the traffic sign in the received digital representation is performed. If one or both tests are unsuccessful, no classification is performed. Instead, it is determined that classification cannot be performed or cannot be performed with sufficient probability. If it is determined that classification cannot be performed or cannot be performed with sufficient probability, a message is output in one embodiment of the method according to the invention, which informs, for example, a person in the vehicle and / or a technical system that classification is not possible for a given digital representation. This message may contain further information about the representation and individual results of the preceding steps for determination and classification.

[0025] This will be illustrated using the example of a [missing information] in Fig.The digital representation of a traffic sign shown in section 6 will be explained in more detail. In sub-step S20a, the following can be explained for the... Fig. 6. The digital representation shown, for example, shows that the features and probabilities shown in Tables 1.1) to 1.5) below are determined by specialized CNNs. 1.1) Form Circle 1,00 Inverse triangle 0,00 triangle 0,00 rhombus 0,00 octagon 0,00 1.2) Text No text 1,00 Stop 0,00 60 0,00 50 0,00 30 0,00 1.3) Symbols roundabout 0,47 Direction of travel straight ahead 0,31 Passing on the left 0,13 Driving direction left 0,12 Direction of travel straight ahead or left 0,08 1.4) Background Blue 1,00 Yellow 0,00 White 0,00 Red 0,00 1.5) Frame (color) No blue frame 1,00 No red frame 0,00 No frame white 0,00 Red 0,00 White 0,00

[0026] Based on the results from sub-step S20a, in sub-step S20b, as shown in the following tables 2.1) to 2.5), a probability of 47% is determined for a traffic sign “roundabout”, 31% for “Mandatory direction straight ahead”, 13% for “Passing on the left”, 12% for “Mandatory direction left” and 8% for “Mandatory direction straight ahead and right”. 2.1) Roundabout Index Total Probability Multiplication 47% Index Total Probability Sum 4,47 2.2) Prescribed direction of travel: straight ahead Index Total Probability Multiplication 31% Index Total Probability Sum 4,31 2.3) Passing on the left Index Total Probability Multiplication 13% Index Total Probability Sum 4,13 2.4) Prescribed direction of travel: left Index Total Probability Multiplication 12% Index Total Probability Sum 4,12 2.5) Direction of travel straight ahead and to the right Index Total Probability Multiplication 8% Index Total Probability Sum 4,08

[0027] The in Fig.In the matrix shown, a misclassification as "roundabout" instead of the second-placed "straight ahead" results in a safety level of ASIL B. Based on the determined safety level ASIL B, the first threshold is, for example, set at 50%, and the second threshold is, for example, the relative value of 30% for the sign with the higher probability, i.e., 30% of 50% = 15%.

[0028] Thus, the test in step S30 shows that the probability for "roundabout" is not above the first threshold (47% < 50%) and the probability for the second-placed traffic sign is not below the second relative threshold (31% > 15% = 50% of 30%). Therefore, it is determined that at least one traffic sign cannot be identified with sufficient probability.

[0029] In one embodiment, a probability for interfering factors can be determined in step 20a. In the example of the Fig. For example, "Difficult lighting conditions" or "Drop shadows" can be defined, and the isolated white dot that cannot be assigned to any predefined symbol can be defined as "Dirt" or "Damage." This can result, for example, in a reduction of the overall probabilities or an increase in the first threshold, e.g., from 50% to 60%, and a reduction in the second relative threshold, e.g., from 30% to 20%.

[0030] The parallel analysis of different aspects and the multi-stage nature of the process can help identify uncertainty in a classification. Furthermore, they can help to distinguish between certain and uncertain sub-aspects. This can then facilitate the implementation of appropriate corrective measures.

[0031] As in Fig. As shown in Figure 7, the above circumstance is used in an embodiment of the method according to the invention to correct an incorrect classification in subsequent sub-steps S40a to S40e, to increase the certainty of a classification, to confirm a lack of possibility of a safe classification and / or, if necessary, to specify the reasons and causes for this.

[0032] In substep S40a, the step "Determine a traffic sign" (S20) can be repeated. In substep S20a, "Determine features," features can be identified that differ from those identified in a previous iteration. For example, a traffic sign might be assigned to a different subgroup of traffic signs that was previously excluded due to a lower probability of assignment. In such a case, the subsequent substep S20b attempts to identify the traffic sign using recognition modules or CNNs specialized for the traffic signs of the other subgroup.

[0033] In sub-step S40b, a comparison can be made with a database containing traffic signs that are expected to be present at the location where the digital representation of a traffic sign was captured. Such a database can be stored, for example, in a cloud and / or locally in the vehicle. This database can also include digital representations of traffic signs that were captured by the vehicle itself or by other vehicles during previous passes over the same location under more favorable conditions, such as better lighting.

[0034] In sub-step S40c, a comparison can be made with the type of environment at the location where the digital representation of a traffic sign is captured. This allows for the determination of the probability of the presence of a traffic sign in question. For example, the probability of a traffic sign differs depending on whether a vehicle is on a highway or rural road, in an urban area, or in a non-public area such as a parking lot, parking garage, company premises, or port area. The same applies if an intersection is detected in front of the vehicle instead of a roundabout, or vice versa. Such information can be provided, for example, by a navigation database, a positioning system, a vehicle assistance system, or other sensors and evaluation systems within the vehicle.

[0035] In sub-step S40d, the capture or evaluation of a further digital representation of the traffic sign in question can be initiated. This can be done, for example, using another camera on a vehicle that captures the traffic sign from a different angle or with a higher resolution. Similarly, a traffic sign can be captured at a later time, after a vehicle has approached the traffic sign further or is traveling at a lower speed.

[0036] In sub-step S40e, the quality of a given digital representation can be improved using image processing. This can be done, for example, with AI-supported image enhancement methods, and in a preceding sub-step S20a, certain interference factors can be taken into account. Likewise, multiple representations of the same traffic sign can be combined.

[0037] As a result, steps S20 and S30 may be repeated to determine and classify a traffic sign.

Claims

[1] A method for classifying a traffic sign, such as for road, rail, water or air traffic, comprising the following steps: (S10) receiving a digital representation of a traffic sign; (S20) Determining a traffic sign based on the digital representation by means of the following sub-steps: (S20a) Determining a probability for the presence of a feature of a traffic sign and (S20b) Determining a probability for the presence of a traffic sign or determining; (S30) Depending on the features and probabilities determined in the step of determining a traffic sign (S20), classifying a traffic sign or determining that a traffic sign cannot be determined or cannot be determined with sufficient certainty. [2] Method according to the preceding claim, wherein in the sub-step determining features (S20a), probabilities for symbols, shapes, colours, alphanumeric characters and / or their arrangement contained in the displayed traffic sign are determined and in the sub-step of determining a traffic sign (S20b), the specific features are assigned to predefined combinations of features. [3] Method according to one of the preceding claims, wherein in the sub-step of determining features (S20a), a probability for a digital representation belonging to a predefined subgroup of traffic signs is determined and in the sub-step of determining a traffic sign (S20b), a probability for an assignment of a digital representation to a traffic sign contained in the subgroup determined in the previous step (S20a) is determined, or determining that the subgroup determined in the previous step (S20a) does not include the traffic sign shown in the digital representation. [4] Method according to one of the preceding claims, wherein in the sub-step of determining features (S20a) disturbing factors within the digital representation are determined, such as low resolution, distortion, shadows, low contrast or color saturation due to weather or lighting conditions, partial obscuration due to objects, blurring of the representation, damage or weathering of the traffic sign. [5] Method according to one of the preceding claims, wherein in the step of classifying a traffic sign (S30), if a digital representation is determined to be not or not reliably classifiable, a message is generated that a digital representation cannot be or cannot be safely classified. [6] Method according to one of the preceding claims, wherein in a step of checking a traffic sign (S40) depending on features, probabilities and threshold values ​​determined in preceding steps of determining a traffic sign (S20) and classifying a traffic sign (S30), one or more of the following sub-steps (S40a to S40e) are carried out: (S40a) Repeated execution of the step of determining a traffic sign (S20), wherein in the step of determining features (20a) features are determined which are different from features that were determined in a previous execution of the sub-step of determining features (20a); (S40b) comparison with a database containing traffic signs expected at a given position; (S40c) Determining an environment such as a motorway, country road, city, non-public area such as a parking lot, multi-storey car park or company or port area, intersection, roundabout or construction site area, at a given position and the traffic signs likely to be present at the position; (S40d) comparison and / or combination with another digital representation of a traffic sign, e.g. which was captured from a different distance, at a different speed or from a different angle than before; (S40e) Improvement of the quality of the representation by means of digital image processing and / or combination with another digital representation. [7] System comprising a sensor, a computing unit and an interface for data communication, wherein the system is arranged to cause the execution of a method according to one of the preceding claims. [8] A vehicle comprising a system according to the preceding claim. [9] A computer program product comprising computer-readable instructions which, when executed by one or more data processing units, cause the one or more data processing units to carry out a method according to any one of claims 1 to 6.

Images