Improving the performance of a neural network

Abstract

The present invention relates to a computer-implemented method for training a neural network (NN) configured to recognize at least one object class based on input data relating to a scene in which at least one object of the object class to be recognized is depicted, a trained neural network (NN), a computer program, and a computer-readable (storage) medium. The method comprises the following steps: - Obtaining a training data set (TD) with several training data examples (E) comprehensive information about at least one object; - Obtaining a reference (L) for the at least one object in each training data example (E); - Generating an output (O) of the neural network (NN) by processing the training data examples (E) through one or more layers of the neural network (NN) according to the parameters (P) assigned to the one or more layers; - Comparing the generated output (O) with the specification (L) using an objective function (F) and outputting a comparison value (CV); - Obtain at least one predetermined threshold value (T) for the comparison value (CV); and - Updating the parameters (P→P') of the neural network (NN) based on the comparison value (CV) and on the basis of the predetermined threshold (T).

Description

[0001] The present invention relates to a computer-implemented method for training a neural network, configured to recognize at least one object class based on input data relating to a scene in which at least one object of the object class to be recognized is depicted, a trained neural network, a computer program, and a computer-readable (storage) medium. The proposed method and the proposed trained neural network can preferably be used in an advanced driver assistance system.

[0002] Object recognition is widely used in various computer vision tasks and refers to the localization and classification of objects according to semantic categories, i.e., determining the location and type of an object. Object localization typically involves determining the object's location by defining its boundaries using a bounding box. A classification task refers to categorizing an object into a specific object class based on particular characteristics of the object.

[0003] Several techniques are available for object recognition, including classical techniques as well as techniques based on machine learning models, particularly using neural networks. A prominent application of object recognition is in advanced driver assistance systems (ADAS), which utilize object recognition techniques to analyze input data from various environmental sensors, such as radar, lidar, ultrasonic, or cameras. Typical ADAS functions serve two purposes: firstly, to assist the driver while allowing them to maintain control of the vehicle; and secondly, depending on the level of automation, to enable partially or fully autonomous driving.Exemplary functions of advanced driver assistance systems include methods for detecting various objects in a vehicle's environment, methods for determining lane markings or lane keeping, braking assistance functions, or various parking assistance functions. While the following description uses the example of object detection for advanced driver assistance systems, the present invention is by no means limited to this specific application. Rather, it is applicable to any method and application for object detection based on neural networks.

[0004] A common problem with object recognition based on neural networks relates to performance fluctuations with different input data or across different object classes, also known as class imbalances. This problem primarily concerns the training phase of the respective neural network. Accordingly, existing solutions mostly focus on optimizing the training of the neural networks.

[0005] One approach to reducing performance fluctuations involves applying a weighting process during neural network training. For example, it is known to increase the prevalence of training input data examples relating to minority object classes that would otherwise be numerically inferior to other object classes present in the training datasets. This can be achieved, for instance, by weighting different training data examples or by providing the neural network with specific subsets of the training data that are more likely to include the minority class. Another related approach is to modify the neural network's weights generated for specific minority classes, specifically increasing the weights for these minority classes relative to other weights.The network parameters are therefore updated with a higher weighting for the minority object classes. A summary of available mitigation strategies regarding imbalances between object classes was provided in 2024 by N. Crasto in "Class Imbalance in Object Detection: An Experimental Diagnosis and Study of Mitigation Strategies," available on arXiv, doi: arXiv:2403.07113.

[0006] Most approaches based on sampling strategies, weighting methods, and data augmentation techniques are widely known for addressing the problem of class imbalance in neural networks or related performance fluctuations. However, these methods are typically labor-intensive and require careful fine-tuning of various hyperparameters. Furthermore, increasing a weight related to a specific object class often affects all instances of that object class, even if performance is already adequate for certain object classes. Additionally, increasing the occurrence of a particular training data example can also increase the occurrence of other object classes within that training data example, even if they do not represent any of the minority object classes.Finally, increasing the occurrence of a particular training dataset or subset tends to increase not only the occurrence of minority object classes but also that of majority object classes. Therefore, these approaches tend to affect not only the targeted minority object class but also the majority object classes.

[0007] It has also been suggested to modify the loss function used during training, as described, for example, in "Focal Loss for Dense Object Detection" by TY. Lin et al. in 2017, available on arXiv, doi: arXiv:1708.02002. Such a modification of the loss function can serve to weight majority object classes—which are frequently represented in the training dataset—less heavily, thus focusing on minority object classes, which would otherwise have lower confidence in the neural network. This approach makes it possible to concentrate on minority object classes during training.

[0008] A compensation of the loss function was proposed in 2022 by B. Li et al. in “Equalized Focal Loss for Dense Long-Tailed Object Detection”, available on arXiv, doi: arXiv:2201.02593, which results in a separate adjustment for each object class, thus balancing the weights of background-foreground object classes and even foreground-foreground object classes.

[0009] Methods that rely on modifying the loss function, however, also require careful fine-tuning and cannot precisely separate the treatment of minority object classes from that of majority object classes. Modifying the loss function with respect to one object class affects the losses for all other object classes and / or other examples related to the same object class. Therefore, the overall performance of the trained neural network depends on the entanglement of the various instances in the training data.

[0010] A selection of training data examples relating to minority classes for region-based object detectors was proposed in 2016 by S. Abhinav et al. in "Training Region-based Object Detectors with Online Hard Example Mining," available on arXiv, doi: arXiv:1604.03540. This approach solves the entanglement problem of minority and majority object classes, but only for a network of region suggestions with a fixed number of predictions or selected object classes. "Hard examples"—i.e., input data relating to a minority object class—are defined only relative to the other selected classes, but not in a general form.

[0011] Finally, in 2017, O. Sener et al. presented an active learning method designed to search for frames from unlabeled data, which would improve the performance of the neural network compared to random selection for annotation. This method was described in "Active Learning for Convolutional Neural Networks: A Core-Set Approach. In International Conference on Learning Representations" [Aktives Lernen für faltende neunnetzwerken: ein Core-Set-Anansatz. In Internationale Konferenz über Lerndarstellungen], available on arXiv, doi: arXiv:1708.00489. However, searching for new training data examples, which then need to be labeled, incurs additional annotation costs and can therefore be inefficient.

[0012] Based on the disadvantages of the various approaches presented in the introduction, one objective of the present invention is to further improve the performance of neural networks for all object classes addressed by the neural network.

[0013] This underlying problem is solved by the method according to claim 1, by the trained neural network according to claim 11, the computer program according to claim 12 and the computer-readable storage medium according to claim 13.

[0014] According to a first aspect of the present invention, the underlying problem is solved by a computer-implemented method for training a neural network, which is designed to recognize at least one class of object on the basis of input data relating to a scene in which at least one object of the class of object to be recognized is represented.

[0015] The proposed procedure comprises the following steps: - Receiving a training dataset with multiple training data examples, comprehensive information about at least one object; - Obtaining a reference for at least one object in each training data example; - Generating an output of the neural network by processing the training data examples through one or more layers of the neural network

[0016] network according to the parameters assigned to one or more layers; - Comparing the generated output with the specified value using an objective function and outputting a comparison value; - Obtain at least one predetermined threshold for the comparison value; and - Updating the parameters of the neural network based on the comparison value and based on the predetermined threshold.

[0017] The parameter updates can be based directly on the comparison result or on any other value derived from the comparison result, such as a score. The minimum one predefined threshold can be any constant value. However, it can also be chosen so that it can be changed. For example, it can be changed depending on the comparison value or on the status of the training process (e.g., section, training dataset, or batch).

[0018] The proposed method is based on a modified parameter update procedure and therefore evaluates the object class imbalance problem differently than the existing solutions discussed in the introduction. In principle, an interval relative to a ground truth is defined by at least one threshold. For a given training data sample, the update is only performed if the comparison value lies within the defined interval. This means that a parameter update is only provided for training data samples belonging to object classes that the neural network has not yet sufficiently learned. Such classes include, for example, minority object classes. Object classes that are already classified with confidence close to a corresponding ground truth, e.g.,Majority object classes, on the other hand, do not need to be considered further during training – meaning no parameter updates are performed. A class imbalance can refer not only to an underrepresentation of a particular object class in a dataset used to train the neural network, but also to object classes that are evenly represented but difficult to detect and / or classify due to high diversity within the object class.

[0019] The problem of object class imbalance, or performance fluctuations across different object classes, is addressed by defining a criterion for updating network parameters during the training phase. This criterion sorts training data examples according to object classes with varying confidence levels in the neural network, such as minority and majority object classes (also referred to as easy and difficult examples). The parameters are then updated only for training data examples related to these classes, such as minority object classes or difficult examples. This helps the neural network focus on improving performance with respect to specific object classes, such as those that are in the minority or those with low confidence levels in the neural network.

[0020] The proposed training method can be combined with any other training method to train the neural network. Furthermore, additional methods can be applied to solve the object class imbalance problem.

[0021] One advantage of the proposed method is that these object classes can be separated without the need for careful tuning of the hyperparameters. Only one or a few hyperparameters are defined in the form of at least one predetermined threshold. Using the proposed training method, the overall performance of the neural network is significantly improved. Imbalances between the object classes are reduced by relaxing the objective function or loss function, thus achieving a high degree of flexibility in investigating the parameter optimization process.

[0022] In a preferred embodiment, the neural network is a neural network for which at least one output in the form of a heatmap is provided. In particular, preferably at least one output of the neural network can be provided in the form of a probability distribution; or a probability distribution can be derived based on at least one output of a specific layer of the neural network. If the training dataset comprises multiple training data examples in the form of training data images, the heatmap or the probability distribution can be provided as a function of the image pixels. In this way, each pixel of each training data image is evaluated.

[0023] It is advantageous if the neural network is an object detector, preferably based on a CenterNet network. An overview of several object detection methods based on neural networks is provided in the article "Object Detection Using Deep Learning, CNNs and Vision Transformers: A Review" by A. B. Amjoud et al., published in IEEE Access, Vol. 11, pp. 35479-35516, 2023, doi: 10.1109 / ACCESS.2023.3266093.

[0024] The CenterNet was proposed by K. Duang et al. in "CenterNet: Keypoint Triplets for Object Detection" [CenterNet: Keypoint Triplets for Object Detection], 2019, available on arXiv, doi: arXiv:1904.08189. If the neural network is based on a CenterNet, the proposed method is preferably used to update the network parameters with reference to a Center heatmap. However, other network parameters can also be updated based on the proposed method.

[0025] A preferred embodiment of the proposed method involves determining the predetermined threshold based on a histogram of the false-positive and / or false-negative results of the neural network's outputs. The histogram, in turn, can be derived from a training dataset, a validation dataset, a test dataset, or any other dataset containing data examples relating to the at least one object class recognized by the neural network.

[0026] To determine the at least one predetermined threshold, at least one recognition threshold of the neural network can also be taken into account. Typically, the recognition threshold of a neural network serves to exclude or reduce false positives during the inference phase. Such thresholds are not usually applied during the training of the neural network. However, according to the present invention, it can be considered for determining the predetermined threshold. Preferably, the absolute value of the predetermined threshold is smaller than the absolute value of the recognition threshold.

[0027] Preferably, hyperparameter optimization is performed with respect to the predetermined threshold. In this context, any suitable optimization method in accordance with the state of the art can be applied.

[0028] Furthermore, it is advantageous if the objective function is a loss function. Preferably, a focal loss function, an L0, L1, L2, or Linf loss function is used for the proposed method.

[0029] According to the method of the present invention, the output generated by processing the training data examples by one or more layers of the neural network is compared with the respective specification of the training data example. It is advantageous if a function value of the objective function or a quantity derived from the function value of the objective function is compared with the at least one predetermined threshold value. The function value thus serves as the comparison value in this case.

[0030] Another preferred embodiment of the method according to the present invention comprises obtaining at least two predetermined thresholds, wherein a first predetermined threshold is a lower predetermined threshold and a second predetermined threshold is an upper predetermined threshold. The two thresholds can define an interval around at least one ground truth or statement, wherein the interval can be distributed symmetrically or asymmetrically around the ground truth or statement.

[0031] The ground truth or statement with respect to which the interval can be defined can be a single ground truth or statement value, e.g., the respective ground truth or statement value for each training data example, or it can be any ground truth or statement metric derived from all the ground truths, statements, or markers of the training dataset.

[0032] Regarding the procedure for training the neural network, it is advantageous if at least one predetermined threshold relates to false-positive results of the neural network. In this case, it is preferable if updating the neural network parameters based on the comparison is only performed when the comparison value is higher than the predetermined threshold.

[0033] Alternatively or additionally, the at least one predetermined threshold can also refer to false-negative results of the neural network. In this case, updating the neural network parameters based on the comparison can only be performed if the comparison value is smaller than the predetermined threshold. Updating based on the comparison value can therefore involve applying a suitable mathematical relationship of the comparison value, e.g., a difference between a predicted output and a corresponding ground truth, relative to or in comparison with the predetermined threshold.

[0034] In a further preferred embodiment of the proposed method, a scaling factor is applied to perform the parameter update, wherein the scaling factor is based on at least one predetermined threshold. When a scaling factor is applied, the parameters of the neural network can be updated for object classes that have already been sufficiently learned by the neural network. In this case, the scaling factor is preferably a number less than one.

[0035] Hyperparameter optimization can also be performed with regard to the scaling factor. The scaling factor can be chosen in such a way that it can be changed, for example, depending on the stage of the training process (e.g., based on a section, a training dataset, or a batch).

[0036] Finally, a further preferred embodiment of the proposed method comprises determining the at least one predetermined threshold based on the specifications of the training data examples. For example, a deviation between a specification and the generated output of the neural network for each training data example that can be compared can be used to determine the predetermined threshold.

[0037] The method for training a neural network, configured to recognize at least one object class based on input data relating to a scene in which at least one object of the object class to be recognized is depicted according to one of the embodiments described in connection with the method, is preferably used to train an object detector, in particular an object detector implemented in an advanced driver assistance system (ADAS) or an automated driving system (AD). Corresponding ADAS or AD functions can be, for example, parking functions, safety functions, or cruising functions. However, any other ADAS or AD function based on a trained neural network can also be implemented using the proposed method.

[0038] In a second aspect, the invention relates to a trained neural network configured to recognize at least one object class based on input data relating to a scene in which at least one object of the object class to be recognized is depicted, wherein the neural network is trained by carrying out the method according to the present invention and with reference to one of the embodiments described in connection with the proposed method. The trained neural network can, for example, be an object detector.

[0039] In another aspect, the present invention relates to a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method according to the present invention and any of the described embodiments.

[0040] Finally, the present invention also relates to a computer-readable (storage) medium comprising instructions that can be executed by at least one processor to carry out the proposed method according to any of the described embodiments, or on which the computer program according to the present invention is stored.

[0041] It should be noted that all embodiments described in relation to the method are also relevant for its advantageous use, the trained neural network, the computer program and the computer-readable (storage) medium, and vice versa.

[0042] The invention and its preferred embodiments are described in more detail with reference to the following drawings. Fig. 1 refers to a block diagram illustrating a first embodiment of the method for training a neural network; Fig. 2 refers to a block diagram illustrating a second embodiment of the method for training a neural network; and Fig. Figure 3 illustrates the effect of using the predetermined threshold.

[0043] In the figures, the same elements are always marked with the same reference symbols.

[0044] In Fig. Figure 1 shows a block diagram relating to a first preferred embodiment of the method according to the present invention.

[0045] To train a neural network NN designed to recognize at least one object class based on input data relating to a scene in which at least one object of the object class to be recognized is depicted, a training dataset TD with several training data examples E (e.g. in the form of images) and a specification L for the at least one object in each training data example E are obtained in a first step.

[0046] The training data examples are provided to the neural network NN as input and processed by one or more layers of the neural network NN in accordance with the parameters P of the neural network.

[0047] The output O generated by the neural network is then evaluated using an objective function F, which is, for example, an error function F. errThe result of the comparison is provided in the form of a comparison value CV, e.g., CV=0-L.

[0048] Furthermore, a predetermined threshold value T is determined for the comparison value CV. Subsequently, the parameters P→P' are updated based on the comparison value CV and the predetermined threshold value T.

[0049] At the in Fig. In the embodiment shown in section 2, two predetermined threshold values ​​are defined, a lower predetermined threshold value T. L and an upper predetermined threshold Tu. In contrast to the one that focuses on Fig. The embodiment relating to 1 is carried out in the Fig. In the embodiment shown in section 2, the parameters P→P' are updated based on the comparison value CV and on the basis of the predetermined thresholds T. L and Tu.

[0050] The proposed method addresses the problem of class imbalance in object recognition by defining a criterion for updating parameters with respect to minority and majority object classes. This is achieved by using one or more predetermined thresholds to compare the outputs O generated by the neural network with the specifications L of the training data examples E. This means that parameter updates are performed only for training data examples E that relate to minority object classes.

[0051] Without losing generality, the following description refers to a neural network (NN) based on a CenterNet. This neural network comprises three different heads (outputs): a so-called center heatmap head, a keypoint head, and an offset head. The centers of the objects in the input data are primarily determined based on the center heatmap generated by the center heatmap head.

[0052] The center heatmap is used to predict the object's center. In the case of input data in the form of images, it is essentially a 2D map, and the neural network outputs or predicts a floating-point value between 0 and 1 for each pixel. If a pixel's value is higher than the values ​​of all neighboring pixels, it is considered a peak value, and if such a peak value exceeds a specific detection threshold, it is interpreted as a detection.

[0053] For training a CenterNet-based NN, the ground truth for a given training data example can be generated by splatting the center point of the object on a heatmap, e.g., by using a Gaussian kernel in the form: L(x,y)=e−(x−x0)2+(y−y0)22σ2

[0054] In this way, a Gaussian peak value L is generated around the center point of a detectable object in a specific training data example, as shown in Fig. Figure 3a illustrates this. It should be noted that the use of a Gaussian kernel is not mandatory; other functions can also be used to generate the markings or information.

[0055] During the training phase, the neural network NN is trained to predict similar outputs O on its heatmap, i.e., a Gaussian peak value in the given example. The outputs O are then compared to the data, and a comparison value is calculated using an objective function F. For example, a loss function in the form of a focal loss f foc The comparison value CV decreases as the difference between the input L and the generated output O of the neural network NN decreases. Here, the comparison value CV is a function value of the applied error function f. foc .

[0056] According to the present invention, a limitation on the generated output is relaxed by defining a predetermined threshold. For example, if the comparison value is below the predetermined threshold, the error generated by the neural network (NN) is considered low enough that it is no longer necessary to update the parameters for this specific training data example. For instance, a scaling factor s can be applied to update the parameters P→P' of the neural network (NN). The values ​​for updating the parameters P→P' of the neural network (NN) can be significantly reduced, for example, with the help of the applied scaling factor s, as soon as the comparison result CV falls below the predetermined threshold T.

[0057] This, in turn, allows the training process to focus on the minority object classes, also known as hard examples, by defocusing on the majority object classes or easy examples. For most object classes, network performance is usually sufficient.

[0058] If θ refers to all parameters P of the neural network, a function f(θ) can be minimized in another way to update the parameters P→P' of the neural network NN as a function of the comparison value CV with respect to at least one predetermined threshold, as in Fig. 3b in the case of an upper predetermined threshold Tu and a lower predetermined threshold T L shown. If the comparison value CV lies within an interval defined by the upper and lower predetermined threshold values ​​Tu and T, respectively. L When a range is stretched, the scaling factor s can be applied, for example: Min(f(Θ))={CV∗s,if TL <CV<TUCV,sonst)

[0059] This procedure is used in Fig. Figure 3b illustrates a 2D curve of the peak value P of the Ground Truth as well as the upper and lower predetermined threshold values ​​T. U or T L The additional dashed line represents the recognition threshold of the neural network NN. This is determined using the upper and lower predetermined thresholds T. U or T L A range or interval can be defined within which the predictions of the neural network NN are considered good enough, or for which the confidence of the neural network NN is considered sufficient.

[0060] The proposed method can also be applied to other versions of the CenterNet-based neural network NN; and of course, it is also applicable to other neural networks.

[0061] In summary, the present invention offers a high degree of flexibility for carrying out optimizations. Furthermore, it makes it possible to remove potentially harmful gradients that can occur, for example, in background pixels that do not relate to a detected object.

[0062] Training the neural network can be performed on one or more CPUs, GPUs, or TPUs. It can be applied either as a post-processing step or directly during the calculation of the comparison values ​​(CV). QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited non-patent literature

[0000] B. Li et al. in “Equalized Focal Loss for Dense Long-Tailed Object Detection”, available on arXiv, doi: arXiv:2201.02593

[0008] S. Abhinav et al. in “Training Region-based Object Detectors with Online Hard Example Mining” [Training region-based object detectors with Online Hard Example Mining], available on arXiv, doi: arXiv:1604.03540

[0010] O. Sener et al. 2017

[0011] A. B: Amjoud et al., published in IEEE Access, Vol. 11, pp. 35479-35516, 2023, doi: 10.1109 / ACCESS.2023.3266093

[0023] K. Duang et al. in “CenterNet: Keypoint Triplets for Object Detection” [CenterNet: Keypoint Triplets for Object Detection], 2019, available on arXiv, doi: arXiv:1904.08189

[0024]

Claims

[1] A computer-implemented method for training a neural network (NN) designed to recognize at least one class of object based on input data relating to a scene in which at least one object of the class of object to be recognized is represented, wherein the method comprises the following steps: - Obtaining a training data set (TD) with several training data examples (E) comprehensive information about at least one object; - Obtaining a reference (L) for the at least one object in each training data example (E); - Generating an output (O) of the neural network (NN) by processing the training data examples (E) through one or more layers of the neural network (NN) according to the parameters (P) assigned to the one or more layers; - Comparing the generated output (O) with the specification (L) using an objective function (F) and outputting a comparison value (CV); - Obtain at least one predetermined threshold value (T) for the comparison value (CV); and - Updating the parameters (P→P') of the neural network (NN) based on the comparison value (CV) and on the basis of the predetermined threshold (T). [2] Method according to any of the preceding claims, wherein the neural network (NN) is a neural network for which at least one output is provided in the form of a heatmap. [3] Method according to any of the preceding claims, wherein the neural network (NN) is an object detector, preferably wherein the neural network is based on a CenterNet network. [4] Method according to any of the preceding claims, wherein the predetermined threshold (T) is determined on the basis of a histogram of false-positive results and / or false-negative results of outputs of the neural network (NN). [5] Method according to one of the preceding claims, wherein hyperparameter optimization is performed with respect to the predetermined threshold (T). [6] Method according to any of the preceding claims, wherein a function value of the objective function (F) is compared with the at least one predetermined threshold value (T). [7] Method according to any of the preceding claims, wherein at least two predetermined threshold values ​​(T) are obtained, wherein a first predetermined threshold value is a lower predetermined threshold value (T L ) is and where a second predetermined threshold is an upper predetermined threshold (Tu). [8] Method according to any of the preceding claims, wherein the at least one predetermined threshold (T) relates to false-positive results of the neural network (NN), and wherein the updating of the parameters (P→P') of the neural network (NN) is performed only when the comparison value (CV) is higher than the predetermined threshold (T), and / or wherein the at least one predetermined threshold (T) relates to false-negative results of the neural network (NN), and wherein the updating of the parameters (P→P') of the neural network (NN) is performed only when the comparison value (CV) is lower than the predetermined threshold (T). [9] Method according to any of the preceding claims, wherein a scaling factor (s) is applied on the basis of the at least one predetermined threshold (T) to perform the updating of the parameters (P→P'). [10] Method according to any of the preceding claims, wherein the at least one predetermined threshold (T) is determined on the basis of the information (L). [11] Use of the method according to any of the preceding claims for training an object detector, in particular an object detector implemented in an advanced driver assistance system. [12] Trained neural network (NN) designed to recognize at least one class of object based on input data relating to a scene in which at least one object of the class of object to be recognized is represented, wherein the neural network is trained by performing the method according to any one of claims 1-10. [13] Computer program comprising instructions which, when the program is executed by a computer, cause the computer to perform the method according to any one of claims 1-10. [14] Computer-readable (storage) medium comprising instructions that can be executed by at least one processor to carry out the method according to any one of claims 1-10, or on which the computer program according to claim 13 is stored.

Images