Method and computer program product for providing a neural network and system for classifying objects for a vehicle

Abstract

The invention relates to a method and computer program product for providing a neural network and system for classifying objects for a vehicle. The method according to the invention comprises the steps of providing a hazard and risk matrix for objects to be classified (S10), determining and assigning a safety weight to a misclassification depending on a risk rating (S20), determining and assigning a training weight to a misclassification (S30), and training a neural network by means of a plurality of iterations (S30). In this process, assigned training weights are dynamically adjusted to optimize a safety index, and the safety index is determined depending on misclassifications and assigned safety weights.

Description

[0001] The invention relates to a method and computer program product for providing a neural network and system for classifying objects for a vehicle.

[0002] Methods and computer program products for providing a neural network and systems for classifying objects for vehicles are known from the state of the art.

[0003] The method according to the invention has the features of claim 1, the system according to the invention has the features of claim 7, the vehicle according to the invention has the features of claim 8, and the computer program product according to the invention has the features of claim 9. The dependent claims describe advantageous embodiments.

[0004] A first aspect of the invention relates to a method for providing a neural network for a vehicle, such as a road, rail, water or air vehicle, for classifying objects.

[0005] The method according to the invention comprises a step of providing a hazard and risk matrix for objects to be classified. The hazard and risk matrix includes assignments of risk classifications to misclassifications of objects.

[0006] In further steps of the procedure, safety weights are determined and assigned to misclassifications of objects depending on the associated risk classifications, and training weights are determined and assigned to misclassifications of objects with the same risk classification.

[0007] In a subsequent step of the method according to the invention, a neural network is trained for object classification by means of a plurality of iterations over a predefined training dataset. The training is carried out using a loss function and an objective function.

[0008] During a given iteration, the loss function determines a value for each misclassification, depending on the training weight assigned to that misclassification. After completion of a given iteration, the objective function determines a value for each misclassification in that iteration, depending on the safety weight assigned to that misclassification. Based on these values, the objective function calculates a safety index (SI). The training weights for the given iteration are then adjusted for the subsequent iteration based on this safety index.

[0009] In an advantageous embodiment of the method according to the invention, a risk classification of a given misclassification is carried out depending on a hazard to a user or an environment of a target vehicle due to the given misclassification, a probability of the occurrence of an object to be classified in an environment of a target vehicle and / or a controllability of a given misclassification by a target vehicle.

[0010] The term "target vehicle" refers to the vehicle for which the system or neural network is intended to be used for object classification. Various likely operating environments may be known or assumed for a target vehicle. This can be taken into account during the development or training of the neural network using the method according to the invention, for example, by providing different hazard and risk matrices that reflect the respective operating environments.

[0011] In a further advantageous embodiment of the method according to the invention, a risk classification of a given misclassification is carried out in accordance with a standard or regulation that a target vehicle must meet. For road vehicles, this can, for example, be an ASIL classification according to the ISO 26262 standard.

[0012] In a further advantageous embodiment of the method according to the invention, the determination and assignment of a safety weight to a given misclassification with a given risk classification is carried out depending on an expected value of a damage assessment for the given misclassification with the given risk classification.

[0013] In a further advantageous embodiment of the method according to the invention, training weights of a given iteration are adjusted for a subsequent iteration by means of Bayesian optimization.

[0014] In yet another advantageous embodiment of the method according to the invention, the determination of the safety index by the objective function is additionally dependent on the probability of an occurrence of an object to be classified in an operating environment of a target vehicle.

[0015] A second aspect of the invention relates to a system for a vehicle for classifying objects. The system according to the invention comprises a neural network for classifying objects, which was provided by a method according to the first aspect of the invention, and / or is configured to initiate an execution of such a neural network.

[0016] A third aspect of the invention relates to a vehicle. The vehicle according to the invention comprises a system according to the second aspect of the invention.

[0017] A fourth aspect of the invention relates to a computer program product. The computer program product according to the invention comprises computer-readable instructions which, when executed by one or more data processing units, cause one or more data processing units to execute a method according to the first aspect of the invention.

[0018] Exemplary embodiments of the invention are explained in more detail below. Fig. 1 an embodiment of a method according to the invention; Fig. 2 examples of misclassifications of objects with varying degrees of severity; Fig. 3. An example of a hazard and risk matrix for traffic signs to be classified; Fig. 4 misclassifications per risk assessment after completion of 500 training iterations with dynamically optimized training weights; Fig. 5 and Fig. 6 misclassifications per risk assessment without dynamic optimization of training weights and Fig. 7. The development of the safety index SI over the course of 500 iterations with dynamic optimization compared to the course without dynamic optimization of the training weights.

[0019] AI is essential for the development of autonomous vehicles, such as highly automated "Level 4" or fully automated "Level 5" vehicles for road use. A crucial task in this context is the complete and accurate classification of objects in a vehicle's environment, as misclassifications can lead to critical situations and fatal accidents.

[0020] A classic approach to reducing misclassifications is to provide more training data, improved sensors, and larger neural networks. This means increased effort for creating, maintaining, and monitoring the training data and the trained neural networks, increased demands on the vehicle's hardware and connectivity, and increased energy consumption.

[0021] Another aspect of classical approaches is that misclassifications are detected and corrected by parallel or downstream stages, which also entails increased effort. Furthermore, the use of parallel or downstream stages often already poses a risk to the surroundings, such as an emergency stop that could endanger following traffic. Since parallel or downstream stages can also fail without further safety mechanisms taking effect, the frequent fallback to parallel or downstream stages represents an increased safety risk.

[0022] Furthermore, the registration or homologation of vehicles for road use regularly requires proof that the vehicle or vehicle system to be registered complies with the requirements of relevant regulations. For example, depending on the required ASIL safety levels according to ISO 26262, different failure probabilities are suggested during the operation of a vehicle intended for road use. For example, a failure probability of 10 is suggested for ASIL A. -6 / hour, for ASIL B of 10 -7 / hour, for ASIL C of 10 -8 / hour and for ASIL D of 10 -9per hour is suggested. Assuming an average speed of 50 km / h, this would correspond to a failure every 50 million kilometers (ASIL A) up to a failure every 50 billion kilometers (ASIL D). Such proof can be required both for initial type approval and for updates, and therefore poses a particular challenge in the case of AI-based systems, potentially involving a very high level of effort.

[0023] The following example will illustrate how the invention can help to better solve the disadvantages and challenges described above.

[0024] Fig. Figure 1 shows an embodiment of a method according to the invention for providing a neural network for classifying objects for vehicles. In step S10, providing a hazard and risk matrix, a hazard and risk matrix for misclassifications F is generated. i,j of an object Oi as object O j provided.

[0025] As in Fig. As illustrated by examples, the misclassification of objects can have varying degrees of severity for a vehicle. For instance, misclassifying a stop sign as a speed limit can cause an autonomous vehicle to accelerate, leading to critical situations. In contrast, misclassifying a 30 km / h speed limit as a 20 km / h speed limit is not safety-relevant but merely represents a quality management (QM) defect.

[0026] According to the invention, this circumstance is taken into account when providing the hazard and risk matrix. Fig.Figure 3 shows an example hazard and risk matrix for 42 road traffic signs to be classified. The hazard and risk matrix in the example shown assigns misclassifications of traffic signs to risk levels ASIL QM and ASIL-A to ASIL D according to the factor "Severity of the error or hazard to the user or the environment" as defined in ISO 26262. The hazard and risk matrix can also consider further factors, such as "Probability of occurrence" and "Controllability of the error," according to the ASIL classification in ISO 26262.

[0027] According to the invention, these factors are determined depending on the intended degree of automation of a target vehicle, in the illustrated example for highly or fully automated vehicles in road traffic. Furthermore, according to the invention, the factors can be determined depending on different operating environments (Operational Design Domain, ODD), such as highways, rural roads, urban traffic, or parking garages. The probability of encountering a speed limit of 120 km / h on a highway is significantly higher than the probability of encountering a stop sign. In urban traffic, the opposite is true.

[0028] As illustrated in the table below, in a further step (S20) of the procedure, safety weights S are assigned to each misclassification according to its ASIL rating. In the traffic sign classification example shown, these are the safety weights SQM =1 to S D =32. The safety weights in the example are chosen depending on the expected damage. ASIL risk classification QM A B C D Safety weight S QM S A S B S C S D Value 1 4 8 16 32

[0029] As shown in the table below, in a further step (S30) each misclassification F i,j according to their ASIL risk rating R k a training weights T QM are D with a starting value and limits assigned. In the example shown here, the starting values ​​correspond to the training weights T. QM are D the values ​​of the corresponding safety weights S QM to S D The allowed value range is set to 1 and 128. ASIL risk classification QM A B C D Safety weight S QM S A S B S C S D Value 1 4 8 16 32 Training weight T QM T A T B T C T D Starting value 1 4 8 16 32 Permitted value range 1-128 1-128 1-128 1-128 1-128

[0030] In the application example described here, a convolutional network is chosen as the neural network to be trained in a subsequent step (S40). According to the invention, a small, readily available network can be selected. The network chosen in the embodiment described here comprises only about 123,000 trainable parameters and, as shown in the following table, has a widely known and commonly used network architecture. # layers Layer type Kernel size filter (adjusted) Stride Padding Output size 1 Conv 3 × 3 21 1 1 50 × 50 2 Conv 3 × 3 44 1 1 50 × 50 3 Max-pool 2 × 2 1 2 0 25 × 25 4 Conv 3 × 3 54 1 1 25 × 25 5 Conv 3 × 3 29 1 1 25 × 25 6 Max-pool 2 × 2 1 2 0 12 × 12 7 Conv 3 × 3 43 1 1 12 × 12 8 Avg-pool 2 × 2 1 2 0 6 × 6

[0031] In the example shown, the training for the selected neural network takes place over 500 iterations I1 to I. 500 over the entire training dataset of 31,367 records. A loss function V f assessed misclassifications F i,j , in the first iteration I1 depending on the initial values ​​of the training weights T QM are DIn the example shown, an ADAM optimization is used as the learning algorithm and a decreasing learning rate of 0.01 is used.

[0032] At the end of a given iteration I l is based on the misclassifications F i,j of iteration I l using an objective function Z f depending on the assigned safety weights S QM to S D A safety index (SI) is determined. According to the invention, the safety weights (S) can be QM to S D The safety index (SI) should be chosen in such a way that it represents a direct measure of compliance with the requirements of a standard or regulatory requirement relevant to a target vehicle and / or operating environment, e.g., safety requirements according to ISO 26262 for highly automated “Level 4” vehicles for road traffic, for driving on motorways and / or driving in parking garages.

[0033] The training weights T are then determined using an optimization algorithm, such as Differential Evolution, Nelder Mead, Bayes or Random Search. QM are D for a subsequent iteration I l+1 adapted. In the example shown here, a cross-entropy loss function is used and the Bayes algorithm is selected as the optimization algorithm, which has proven to be particularly suitable in the course of work on the invention.

[0034] Fig. Figure 4 and the following table show the result after completion of 500 training iterations with the optimized training weights T. QM are D , the number of misclassifications F i,j per risk classification R k and the resulting Safety Index (SI). The result shows that ASIL D misclassifications are completely avoided. The Safety Index is 43. ASIL risk classification QM A B C D Security Index SI Safety weights 1 4 8 16 32 Training weights 3.6 16.4 1.0 9.5 28.0 # Misclassifications 3 2 2 1 0 summands of the safety index SI 3 8 16 16 0 43

[0035] Fig. 5 and the following table show the result without dynamic optimization of the training weights T QM are D , with a starting value of 1 for all training weights T QM are D The safety index (SI) in this case is 100. ASIL risk classification QM A B C D Security Index SI Safety weights 1 4 8 16 32 Training weights 1 1 1 1 1 # Misclassifications 4 2 1 3 1 summands of the safety index SI 4 8 8 48 32 100

[0036] Fig. 6 and the following table show the result without dynamic optimization of the training weights T QM are D , with starting values ​​of the training weights T QM are D , which correspond to the safety weights S QM to S D The safety index (SI) in this case is 94. ASIL risk classification QM A B C D Security Index SI Safety weights 1 4 8 16 32 Training weights 1 4 8 16 32 # Misclassifications 6 2 2 0 2 summands of the 6 8 16 0 64 94 Security Index SI

[0037] Fig. Figure 7 shows the development of the safety index SI of the method according to the invention over the course of 500 iterations I1 to I 500 compared to classic training approaches without dynamic optimization of separate training weights T.

[0038] The described application example demonstrates how the Safety Index (SI) can directly measure the safety of the trained neural network with respect to object classification, for example, in relation to meeting the safety requirements of ISO 26262 for autonomous vehicles. Distinguishing between safety weights and training weights, and dynamically optimizing the training weights, can help ensure optimized training regardless of the size and architecture of the neural network, the classification task, or the training datasets. Experience has shown that neural networks with 5 to 10 times fewer parameters than classically trained neural networks can be used in this way.

Claims

Method for providing a neural network for classifying objects for a vehicle, wherein the method comprises the following steps: (S10) providing a hazard and risk matrix for N objects (O1...N) to be classified, which assigns a risk rating (Rk) to a misclassification (Fi,j) of an object i (Oi) as an object j (Oj) with 1 ≤ i, j ≤ N and i ≤ j; (S20) determining and assigning a safety weight (Sk) to a misclassification (Fi,j) depending on a risk rating (Rk); (S30) determining and assigning a training weight (Tk) to a misclassification (Fi,j); (S40) training a neural network by means of a plurality of iterations (I1...M) over a predefined training dataset to classify objects (O1...N) using a loss function (Vf) and an objective function (Zf), wherein during the execution of an iteration (Il) the loss function (Vf) determines a value for each misclassification (Fi,j) depending on a training weight (Tk) assigned to the misclassification (Fi,j), and wherein after completion of an iteration (Il), the objective function (Zf) determines a value for each misclassification (Fi,j) of the iteration (Il) depending on a safety weight (Sk) assigned to the misclassification (Fi,j), determines a safety index SI depending on these determined values ​​and adjusts the training weights (Tk) for the subsequent iteration (Il+1) depending on the determined safety index SI. Method according to claim 1, wherein a risk classification (Rk) of a misclassification (Fi,j) is performed depending on a hazard to a user or an environment of a target vehicle due to a misclassification (Fi,j), a probability of the occurrence of an object (Oi) in an environment of a target vehicle and / or a controllability of a misclassification (Fi,j) by a target vehicle. Method according to one of the preceding claims, wherein a risk classification (Rk) of a misclassification (Fi,j) is carried out in accordance with a standard or regulation to be met by a target vehicle, such as an ASIL classification in accordance with the standard ISO 26262. Method according to one of the preceding claims, wherein the determination and allocation of a safety weight (Sk) to a misclassification (Fi,j) with a risk rating (Rk) is carried out depending on an expected value of a damage assessment for a misclassification (Fi,j) with a risk rating (Rk). Method according to one of the preceding claims, wherein the adjustment of the training weights (Tk) for a subsequent iteration (Il+1) is carried out by means of Bayesian optimization. Method according to one of the preceding claims, wherein the determination of a safety index SI by the objective function (Zf) is additionally carried out depending on a probability of an object (Oi) occurring in an operating environment of a target vehicle. System for classifying objects, comprising a neural network and / or configured to classify objects using a neural network provided by a method according to one of the preceding claims. Vehicle comprising a system according to the preceding claim. Computer program product comprising computer-readable instructions which, when executed by one or more data processing units, cause one or more data processing units to execute a method according to any one of claims 1 to 6.

Images