Method for object classification

Abstract

The invention relates to a method, in particular a computer-implemented method, for classifying at least one object (O), to a computer program and to a computer program. The method comprises the following method steps: providing input data (Eo(t)), which contains at least one item of information about the at least one object (O), as a function of time (t), classifying the at least one object (O) by means of a classifier (CL) and determining an item of classification information (I(t)) comprising a probability (clx) that the at least one object (O) belongs to at least one predefined class (CLx), providing the item of classification information (I(t)) as input for a trained neural network (NN), said neural network (NN) being designed to determine an updated item of classification information (Iac(t)) on the basis of the input, the updated item of classification information (Iac(t)) being a superposition of the item of classification information (I(t)) and at least one item of classification information (Iac(t-n)) determined at a prior time, and outputting the updated item of classification information (Iac(t)). The invention also relates to a method for training a corresponding neural network.

Description

[0001] Description

[0002] Methods for object classification

[0003] The invention relates to a method, in particular a computer-implemented method, for classifying at least one object by means of a neural network, a method, in particular a computer-implemented method, for training a neural network, in particular a neural network for use in the method according to the invention for object classification, a computer program for carrying out the method according to the invention and a computer program product on which the method is stored.

[0004] For object classification, input data from sensors, for example, input data containing information about the objects to be detected and classified, is examined for the presence of various predefined features and, if necessary, these features are assigned to specific object classes. In this context, many different methods and, in particular, numerous approaches from the field of machine learning for object recognition have become known. Assignment to an object class is achieved, for example, with the help of an activation function, which can be used to define specific value ranges for the detected features.

[0005] For example, object classification is often performed in connection with advanced driver assistance systems (ADAS), which use neural networks to process input data from various sensors or environmental sensors, such as radar, lidar, or ultrasonic sensors, as well as cameras. ADAS functions typically provided by ADAS systems are often based on previous object classifications and can either support the driver while maintaining control of the vehicle, or, depending on the level of automation, fully automated driving can be realized.Examples of ADAS functions include various methods for detecting objects or obstacles on the road, methods for detecting lane boundaries and / or keeping the vehicle in a lane, methods for detecting rain on the windshield, or methods for assisting or performing a parking maneuver.

[0006] For numerous such applications, but also for classification tasks independent of driver assistance systems, certain objects must be classified over a specific period of time, especially continuously. The classification information determined in each case can then vary as a function of time. Examples of time-dependent classification information include classifications regarding a vehicle's current location, such as a stop in a tunnel or parking garage, or the presence of a specific category for a vehicle environment, such as a highway, a country road, city traffic, or a construction site.

[0007] Typically, the objects under consideration are always individually classified at a specific point in time based on the corresponding input data at that time, thus classifying them in a cycle-based manner. The sensors used to capture the input data also typically operate in a cycle-based manner. If the classification information obtained in this way for different object classes is viewed as a function of time, fluctuations and / or oscillations in the classification results often occur in such cases. The reason for this can include, among other things, misclassifications that occur, particularly temporarily, i.e., only for individual input data.

[0008] However, such classification errors can be problematic when using the classification results for more advanced functions, for example, for the provision of different ADAS functions. Temporal stabilization of the classification information obtained at different times would therefore be desirable. For temporal stabilization, different filters, particularly low-pass filters or median filters, are often used, which are manually parameterized and adjusted. However, the use of such filters is also fraught with various problems. In the case of a low-pass filter, for example, the filter effect must, on the one hand, be strong enough to effectively reduce fluctuations or oscillations. On the other hand, an excessively strong filter effect will result in a real change in the classification information being filtered out.

[0009] The present invention is therefore based on the object of enabling an improved time-dependent object classification.

[0010] This object is achieved by the inventive methods according to claims 1 and 7, the computer program according to claim 14 and the computer program product according to claim 15.

[0011] In a first aspect, the object underlying the invention is achieved by a method, in particular a computer-implemented method, for classifying at least one object, comprising the following method steps:

[0012] - Providing input data containing at least one piece of information about the at least one object as a function of time,

[0013] - Classifying the at least one object by means of a classifier and determining classification information comprising a probability for the at least one object to belong to at least one predetermined class,

[0014] - Providing the classification information as input for a trained neural network, which neural network is designed to determine updated classification information based on the input, wherein the updated classification information is a superposition of the classification information and at least one classification information determined at a previous point in time, and

[0015] - Outputting the updated classification information. The at least one object to be classified can be a wide variety of objects that are detectable in the input data. In the event that the method is used in a driver assistance system, it is, for example, a wide variety of objects in the environment of a vehicle into which the driver assistance system is integrated. The objects are preferably, on the one hand, fixed or moving external objects, such as pedestrians, other road users, in particular vehicles, buildings, trees, road markings or the like. However, it can also be background information, for example information about whether the respective input data belongs to a specific environment category, for example a presence in a tunnel or parking garage, or to a specific scene category, e.g.Highway, country road, or city traffic. However, the method is by no means limited to implementation in a driver assistance system; rather, it can be used for a wide variety of object classification tasks.

[0016] Providing input data as a function of time can, for example, include input data acquired at different points in time using a suitable sensor, in particular an environmental detection sensor. The input data can be acquired periodically or continuously at specific, in particular predeterminable, points in time.

[0017] Any object classifier known from the state of the art can be used for object classification. In particular, classifiers based on one or more neural networks can be used, especially methods from the field of machine learning or deep learning, e.g., based on convolutional neural networks (CNNs), so-called region proposal methods, single-shot detectors, or the YOLO detector. However, classical methods can also be used.

[0018] For classification, for example, a probability vector whose dimensions correspond to a number of object classes can be used. The corresponding classification information can then be used to determine the object class with the highest probability. This is preferably determined using an arg-max operator.

[0019] The classifier determines a piece of classification information as a function of time, i.e., based on different input data acquired at different points in time. According to the invention, this classification information is provided to a trained neural network, which is designed to determine updated classification information based on the input. The updated classification information is a superposition of the classification information used as input and at least one piece of classification information determined, in particular updated, at a previous point in time. Multiple pieces of classification information determined, in particular updated, at different previous points in time can also be used to update current classification information.

[0020] The method according to the invention enables efficient and precise consideration of a temporal component in object classification. This avoids temporary misclassifications, such as fluctuations and / or oscillations. At the same time, however, it ensures that real changes in the respective classification are taken into account. The neural network according to the invention thus has the effect of a highly selective, highly specific, and individually tuned low-pass filter while simultaneously avoiding its disadvantages.

[0021] Furthermore, the method according to the invention also increases the performance of multi-cycle object classification by considering the temporal component when determining the respective classification information. Accumulating classification information as a function of time and superposing it to determine updated classification information can also lead to a reduction in computational effort. Instead of accumulating a large number of features in all images, classification information for specific objects can be considered as a function of time.

[0022] To initialize the determination of the updated classification information at a predeterminable start time, it can be assumed, for example, that the first determined updated classification information corresponds to the current classification information determined at the start point, or that an updated classification information determined at a predeterminable time before the start time is zero.

[0023] It is therefore possible, for example, to store or save the classification information as a function of time and to use the stored classification information to determine the updated classification information.

[0024] Another advantageous embodiment of the method according to the invention includes the neural network being a recurrent neural network (RNN). Using an RNN, temporal information in input data can be taken into account. Thus, the use of an RNN within the scope of the present invention allows temporal filtering and class linking. There are, in turn, various options for implementing an RNN. On the one hand, the RNN can be a closed recurrent unit (Gated Recurrent Unit (GRU)) or a so-called long short-term memory network (LSTM). However, it can also be an individually configured RNN in which, in particular, a number of nodes and connections is suitably minimized in order to reduce the number of network parameters and enable computationally and memory-efficient operation.

[0025] In a further, particularly preferred, embodiment, the neural network is configured to consider a transition probability for a transition of a detected object from a first predefined class to a second predefined class when determining the updated classification information. In this way, mutual influences between different classes, and in particular time-varying classification information, can be taken into account particularly efficiently. A network configured in this way is thus particularly well-suited to detecting temporary misclassifications and time-varying classification information and to process this information appropriately.

[0026] In a further embodiment, an output of the neural network, in particular the updated classification information, is limited to a predeterminable range of values, in particular to an interval with values ​​between zero and one.

[0027] A further embodiment includes that the neural network is designed to take into account an activation function, in particular a sigmoid function, when determining the updated classification information.

[0028] Alternatively or additionally, a normalization of the respectively determined probability values ​​for membership in a specific predefined class can also be carried out.

[0029] Preferably, the input data also comprises sensor data from a sensor, in particular a radar sensor or a camera. The input data may also include sensor data acquired by multiple sensors, in particular environmental detection sensors.

[0030] The object underlying the invention is achieved according to a second aspect of the present invention by a method, in particular a computer-implemented method, for training a neural network, which neural network is designed to determine updated classification information based on at least one item of classification information, wherein the updated classification information is a superposition of the classification information and at least one item of classification information determined at a previous point in time, in particular a neural network for use in a method according to the invention according to one of the previously described embodiments. The training method comprises the following method steps:

[0031] - Providing training data which contains at least one piece of information about the at least one object,

[0032] - Providing an indication regarding a class for the at least one object for all training data,

[0033] - Generating an output of the neural network for all training data (DT(O)) by processing the training data using the neural network in accordance with parameters for the neural network,

[0034] - comparing the generated output with the indication using at least one error function, and

[0035] - Optimize the parameters based on the comparison.

[0036] The training data is preferably sensor data, in particular data from one or more environmental detection sensors, for which at least one piece of information about at least one object is specified in the form of an indication or a label. The object can be an external object or information about a background of the recorded sensor data. An external object is, for example, a fixed or moving object in the vicinity of the sensor used in each case, for example a passerby, a road user, an infrastructure object, e.g. a building, a tree, or the like. Background information relates, for example, to information about where and under what circumstances the sensor data was recorded, for example in a tunnel or parking garage, or in a specific traffic situation, for example on a motorway, on a country road, or in city traffic.

[0037] Depending on the classification information determined, the indication can, on the one hand, be constant over time. However, the indication can also be a function of time and thus change over time. In particular, the indication can change as a function of a specific classification cycle. In this case, the indication is, for example, a vector whose dimension corresponds to the number of possible classes. For each classification cycle, i.e. for each classification, the vector component corresponding to the correct class then takes on a predeterminable value, in particular the value one, while all other vector components take on the value zero. By means of a time-variable indication, information about the current environment of the sensors, by means of which the respective input images are generated, can be appropriately taken into account during training.This improves the subsequent accuracy of the neural network in object classification, in particular it leads to improved accuracy, consideration of a temporal component with regard to classification information and stabilization with regard to temporary misclassifications.

[0038] The training process can be started or executed multiple times. To initialize the training process, suitable initial parameters for the neural network can be selected.

[0039] It is also advantageous to optimize the parameters using a gradient method or a steepest descent method. Random initialization can be advantageously chosen.

[0040] With regard to the training process, it is also advantageous to perform regularization. For example, a regularization term can be added to the error function. Regularization can ensure that the output of the neural network, in particular the updated classification information, is always greater than zero and yet as small as possible in magnitude. This results in low values ​​or probabilities being output by default, for example, for scenarios for which only a small amount of training data is available.

[0041] In a preferred embodiment of the training procedure, the

[0042] Error function is chosen in such a way that the error function can be used to

[0043] Classification error can be optimized. The classification error is based on a difference between the respective indication and the determined updated classification information.

[0044] In a further embodiment of the training method according to the invention, the error function is selected such that a probability error is optimized using the error function. The probability error, in turn, is based on a difference between the respective indication and the probabilities output by the neural network for the presence of the at least one predeterminable class.

[0045] The different error functions have different advantages and disadvantages. Optimizing for the probability error offers the mathematical advantage of a suitable gradient, while optimizing for the classification error corresponds precisely to the desired training result: finding the correct classification information.

[0046] A further embodiment of the training method according to the invention includes selecting the error function such that a classification error and a probability error are taken into account. The error function is thus a combination, in particular a weighted sum, of an error function that can be used to optimize a classification error and an error function that can be used to optimize a probability error. The weights for the summands can be determined, for example, as part of hyperparameter optimization.

[0047] The method according to the invention for classifying at least one object and / or the training method according to the invention is / are preferably used for object classification, in particular within a driver assistance system. The method according to the invention is particularly preferably used for object classification using a radar sensor or a camera. It is also advantageous if the method is used for classifying time-varying object classes.

[0048] The present invention also relates to a driver assistance system configured to implement the method according to the invention for classifying at least one object. In the case of a driver assistance system, the objects are, for example, other road users, in particular vehicles or pedestrians, road barriers, or other objects within the respective detection range of the sensors used.

[0049] The object underlying the invention is further achieved by a computer program with instructions which, when the computer program is executed by a computer, cause the computer to carry out the method according to the invention for classification according to one of the described embodiments or the training method according to the invention.

[0050] The object underlying the invention is also achieved by a computer program product on which a computer program according to the invention is stored.

[0051] It should be noted that the methods used in connection with the method according to the invention for classifying at least one object are mutatis mutandis also applicable in connection with the training method according to the invention, the computer program according to the invention, and the computer program product according to the invention, and vice versa.

[0052] The invention and its advantageous embodiments are explained in more detail with reference to the following figures. They show:

[0053] Fig.1 is a flow chart illustrating the method for object classification according to the invention; Fig. 2 is a diagram illustrating the consideration of

[0054] transition probabilities; and

[0055] Fig. 3 is a flow chart illustrating the training method according to the invention.

[0056] In the figures, identical elements are always provided with the same reference symbol.

[0057] Fig. 1 illustrates the method according to the invention for classifying at least one object 0. First, input data E0(t) is acquired as a function of time t, which contains at least one piece of information about the at least one object 0. The input data E0(t) is, for example, sensor data acquired by one or more sensors, for example, data acquired by a camera or a radar sensor. Such sensor data is typically acquired as a function of time t and thus cycle-based.

[0058] The input data E0(t) are made available to a classifier CL. The classifier can be based either on a conventional classification method or on a method from the field of artificial intelligence, in particular a method from the field of machine learning, for example using a neural network. The classification of the at least one object 0 typically also comprises a detection of the object 0 in the input data Eo(t), which is integrated into the classifier CI or precedes the classification, followed by a classification into one or more predefinable object classes or classes CL. This classification can be carried out, for example, using a probability vector by determining a probability for the at least one object 0 to belong to at least one predefined class CL or to each of the predefined classes CL.

[0059] Based on the input data E0(t), the classifier CL thus determines classification information l(t) as a function of time. This classification information could be, for example, a probability for the presence of the class CL with the highest probability. In this case, I(t) corresponds to a class probability.

[0060] In a next step, the determined classification information l(t) is provided as input to a trained neural network (NN). The neural network (NN) is designed to generate updated classification information l based on the input. ac (t) to be determined and output.

[0061] The updated classification information l ac(t) is a superposition of the classification information l(t) at the current time t and at least one updated classification information l determined at a previous time t-1 , t-2, ... ac (t-1 ), l ac (t-2), ... ,l ac (tn). The superposition can, for example, be a weighted sum of the currently determined classification information l(t) and one or more n updated classification information l determined in the past. ac (tn). In principle, classification information l(tn) from the past, which has not been updated by means of the trained neural network NN, can also be included in the determination of the updated classification information l(t); however, the updating according to the invention results in a stabilization of the classification and increased accuracy, so that the consideration of the updated classification information l ac(tn) from the past is preferred.

[0062] Optionally, the neural network NN can also be designed to determine the updated classification information l ac (t) to consider a transition probability p for a transition of a detected object 0 from a first predetermined class CL1 to a second predetermined class CL2, as indicated in Fig. 1 by the dashed lines. In this way, a temporal change in the classification of an object 0 can be considered in a particularly efficient manner. A real classification change can be precisely distinguished from a fluctuation or oscillation in the classification as a function of time, i.e., from individual erroneous classifications occurring at specific points in time. This is explained in more detail with reference to Fig. 2.

[0063] Fig. 2 shows three different predeterminable classes CL1 - CL3. For the embodiment shown, it is further assumed, without loss of generality, that the object classification is carried out in conjunction with a driver assistance system of a vehicle. In this case, one or more environment detection sensors are arranged on the vehicle and serve to detect objects 0 in the vehicle's surroundings. Here, it is assumed that the first predeterminable class CL1 is a motorway as the vehicle environment, the second predeterminable class CL2 is a country road as the vehicle environment, and the third predeterminable class CL3 is a city traffic as the vehicle environment.

[0064] During a longer journey, the vehicle's environment can change multiple times, so an object class for the vehicle's environment can also change as a function of time. Thus, the classification into one of the three classes CL1 - CL3 also varies over time.

[0065] Other examples of time-varying classifications could include classes to distinguish between being in or outside a tunnel, in or outside a parking garage, or in or outside a construction site.

[0066] For the embodiment shown in Fig. 2, the updated classification information l ac(t) for example, transition probabilities p for transitions between the first CL1 and second predeterminable class CL2 pi2 and p2i , the first CL1 and third predeterminable class CL3 p and psi as well as the second CL2 and third predeterminable class CL3 P23 and p32 are taken into account. The individual probabilities p can be the same or different. During the inference of the trained neural network NN, ie during the determination of the updated classification information l ac (t) based on the previously determined classification information l(t), these transition probabilities are in particular constant parameters of the neural network NN, which are optimized in advance during training.

[0067] It should be noted that, within the scope of the present invention, it is by no means mandatory that each object 0 be classified into different classes CL as a function of time t. Rather, objects 0 whose classification is constant over time can also be considered. In this case, the respective determined classification information l(t)=const. Furthermore, objects 0 with temporally constant and temporally variable classifications can also be considered or processed in parallel. In the case of temporally variable classification information l(t), these can be periodic changes or individual changes at specific times t.

[0068] To further explain the functioning of the neural network NN according to the invention, a classification task with N predeterminable classes CL is assumed. A probability vector CL=[cl1, cl2, ... cIN] can then be used to classify object 0 into one of the predeterminable classes CL. From this vector, the respective classification information l=clx regarding object 0's membership in the predeterminable class clx can be determined, for example, using an arg-max operator applied to the probability vector CL. A temporal component t can also be taken into account for the vector CL: CL(t)=[cl1(t), cl2(t), ... clN(t)].

[0069] In the context of the present invention, based on the classification information l(t), an updated classification information l ac (t). When considering multiple classes CL, the underlying probability vectors are:

[0070] CLac(t)=CLac(t-1 )T + w CL(t), i.e. a superposition of the vector with the currently determined classification information CL(t) and a probability vector with classification information CL determined at a previous point in time, in particular updated ac (t-1 ). As already mentioned, several classification information CLac(tn) determined at different previous times t can also be taken into account.

[0071] In the given equation, T represents a matrix for the transition probabilities p and w is a weight vector.

[0072] Finally, Fig. 3 illustrates the inventive training method for training a neural network NN according to the invention, as described in connection with Fig. 1. Training data To, which contain at least one piece of information about at least one object 0 as well as associated indications or labels in the form of the classification information lref(t) associated with the object 0, are provided to the neural network NN as input. The indications lref(t) can be functions of time t, as indicated here, but they can also be temporally constant classification information l re f=const. It goes without saying that the training data To, regardless of any possible time dependence of the indication lref(t), is a series of input data acquired at different times t.

[0073] Based on the training data To, an output of the neural network NN is then generated in the form of updated classification information l ac (t) is generated and compared with the respective indication lref(t). Subsequently, the parameters of the neural network NN are optimized using a suitable error function LF, particularly iteratively. The error function LF can be chosen to optimize a classification error, e.g., LF=(I ac (t)-lref(t)) 2 . This is therefore in particular a difference between the issued updated classification information l ac (t) and the corresponding indication lref(t). However, it is also conceivable to optimize a probability error using the error function LF, e.g., LF=(lref(t)-CLx(t)) 2This, in turn, is the difference between the indication lref(t) and the determined probability CLx(t). An error function LF that takes both the classification error and the probability error into account is also conceivable. Various other measures can also be carried out during training. For example, regularization can occur. The transition matrix T can be regularized using an I_2 norm, for example. In addition or alternatively, values ​​of the matrix T or the vector w can be chosen so that the neural network NN acts like a low-pass filter. Furthermore, the vector w can be normalized or restricted with regard to a range of values. Similar measures can be taken for certain matrix elements, for example the diagonal elements of the matrix T.

Claims

Patent claims 1. Method, in particular computer-implemented method, for classifying at least one object (0), comprising the following method steps: - providing input data (E0(t)) containing at least one piece of information about the at least one object (0) as a function of time (t), - Classifying the at least one object (0) by means of a classifier (CL) and determining classification information (l(t)) comprising a probability (clx) for the membership of the at least one object (0) to at least one predetermined class (CLx), - Providing the classification information l(t)) as input for a trained neural network (NN), which neural network (NN) is designed to determine an updated classification information (lac(t)) based on the input, wherein the updated classification information lac(t)) is a superposition of the classification information (l(t)) and at least one classification information lac(tn)) determined at a previous point in time, and - Output the updated classification information (lac(t)).

2. The method according to claim 1, wherein the neural network (NN) is a recurrent neural network.

3. The method according to claim 1 or 2, wherein the neural network (NN) is designed to take into account a transition probability (p) for a transition of a detected object (0) from a first predetermined class (CL1) to a second predetermined class (CL2) when determining the updated classification information (lac(t)).

4. Method according to one of the preceding claims, wherein an output of the neural network (NN), in particular the updated classification information (lac(t)), is limited to a predeterminable range of values, in particular to an interval with values between zero and one.

5. Method according to one of the preceding claims, wherein the neural network (NN) is designed to take into account an activation function, in particular a sigmoid function, when determining the updated classification information (lac(t)).

6. Method according to one of the preceding claims, wherein the input data (Eo(t)) are sensor data from a sensor, in particular a radar sensor or a camera.

7. A method, in particular a computer-implemented method, for training a neural network (NN), which neural network (NN) is designed to determine an updated classification information (lac(t)) based on at least one item of classification information (l(t)), wherein the updated classification information (lac(t)) is a superposition of the classification information (l(t)) and at least one item of classification information (lac(tn)) determined at a previous time, in particular a neural network (NN) for use in a method according to one of the preceding claims, wherein the method comprises the following method steps: - providing training data (To) which contain at least one piece of information about the at least one object (0), - Providing an indication (lref(t)) regarding a class (CL) for the at least one object (0) for all training data (To), - generating an output of the neural network (NN) for all training data (To) by processing the training data (To) by means of the neural network (NN) in accordance with parameters for the neural network (NN), - comparing the generated output with the indication (lref(t)) using at least one error function (LF), and - Optimize the parameters based on the comparison.

8. The method according to claim 7, wherein the parameters are optimized using a gradient method or a steepest descent method.

9. Method according to at least one of claims 7 or 8, wherein a regularization is carried out.

10. Method according to one of claims 7 to 9, wherein the error function (LF) is selected such that a classification error can be optimized by means of the error function (LF).

11. Method according to one of claims 7 to 10, wherein the error function (LF) is selected such that a probability error can be optimized by means of the error function (LF).

12. The method according to claim 10 and 11, wherein the error function (LF) is selected such that a classification error and a probability error are taken into account by means of the error function (LF).

13. Use of the method according to one of claims 1 to 6 for object classification, in particular within a driver assistance system.

14. A computer program comprising instructions which, when the computer program is executed by a computer, cause the computer to carry out the method according to any one of claims 1-6 or the method according to any one of claims 7-12.

15. A computer program product on which the computer program according to claim 14 is stored.

Images