Object classification based on measurement data from multiple perspectives with pseudo-tags

Abstract

A method (100) for training one or more neural networks (1), the neural networks processing measurement data (2), the method comprising the steps of: providing (110) training examples (2a) of the measurement data (2), the training examples comprising both training examples (2a1) marked with a target classification score (2b) and training examples (2a2) not marked; the training examples (2a) are processed (120) by the one or more neural networks (1) as classification scores (4), and with respect to the marked training examples (2a1), a predetermined cost function (5) is used to evaluate (130) how much the classification scores (4) correspond to respective target classification scores (2b), and-how much the intermediate results (3) formed from the training examples (2a1) that are similar to each other are similar to each other and how much the intermediate results (3) formed from the training examples (2a1) that are dissimilar to each other are dissimilar to each other; optimizing (140) a parameter (1a) characterizing the behavior of the one or more neural networks (1), with the goal that the evaluation (5a) by the cost function (5) will be expected to be improved when continuing to process the training examples (2a1); checking (150) whether intermediate results (3) formed for a subset of the training examples (2a) containing at least one unmarked training example (2a2) are similar to each other according to a predetermined criterion (6); if so, converting (160) the untagged training examples (2a2) of the subset into tagged training examples (2a1) using the priority category (4 *) as a tag (2b); and training (170) the one or more neural networks (1) with the training examples (2a *) upgraded in this way.

Description

Technical Field

[0001] The present invention relates to training a neural network to recognize and classify objects based on measurement data recorded from different viewing angles and / or using different measurement modalities. Here, iteratively generated pseudo labels are used to improve the training quality. Background Art

[0002] In order to at least partially automatically drive a vehicle in road traffic, a representation of the vehicle's surroundings is required, which also describes the objects located in the surroundings. Therefore, the vehicle's surroundings are usually monitored using a plurality of cameras and / or other sensors, such as radar sensors or lidar sensors. The respectively acquired measurement data are then evaluated using a neural classification network as to which objects are present in the vehicle's surroundings.

[0003] US2021 / 012166 A1, WO 2020 / 061489 A1, US10,762,359 B2 and JP 6614611 B2 disclose the use of "contrastive loss" to train such neural networks. In this way, the neural networks can be coordinated with each other, for example, so that they map images showing the same object to the same representation. However, this still does not exempt the obligation to provide enough labeled training examples for each camera view. Summary of the invention

[0004] The invention provides a method for training one or more neural networks. Specifically, neural networks are used here to process measurement data, in particular images, recorded from different viewing angles and / or using different measurement modalities into classification scores for one or more classes of a predefined classification. These classes may, for example, relate in particular to different types of objects present in the area sensed when recording the measurement data.

[0005] The method starts by providing training examples of measurement data, which include both training examples labeled with target classification scores and unlabeled training examples.

[0006] The training examples are processed by one or more neural networks into a classification score. In this case, an intermediate result is also detected for forming the classification score. This intermediate result can be, for example, a representation of the measurement data, which has a dimension that is significantly lower than the measurement data itself, but still higher than the classification score finally determined.

[0007] The classification scores can take continuous values. However, according to preset rules, priority categories are also derived from these continuous values. For example, the category with the largest classification score can be evaluated as the priority category.

[0008] Now, we evaluate the labeled training examples using the preset cost function:

[0009] ● How well the classification scores correspond to the corresponding target classification scores, and

[0010] ●The extent to which intermediate results formed from training examples that are similar to each other are similar to each other, and the extent to which intermediate results formed from training examples that are dissimilar to each other are dissimilar to each other.

[0011] In this case, any metric can be used to measure the similarity of the training examples. For example, the similarity or equality of the target classification scores can also be incorporated into the metric.

[0012] For this purpose, the cost function may for example comprise, among others, a classification loss measuring the consistency with a target classification score and a contrastive loss measuring the similarity of intermediate results.

[0013] Now, the parameters characterizing the behavior of one or more neural networks are optimized with the goal that the evaluation by the cost function is expected to improve when further processing the training examples. For example, the value of the loss function can be back-propagated to the gradient along which the individual parameters are changed in the next learning step. For example, there may be a distribution of work in one or more neural networks, where a certain part of the architecture forms an intermediate result, while another part of the architecture determines the classification score from the intermediate result. Thus, the contrastive loss mainly acts on the part that forms the intermediate result, while the classification loss mainly acts on the part that determines the classification score.

[0014] Now check to what extent there are subsets of training examples that satisfy the following conditions:

[0015] ●The subset contains at least one unlabeled training example;

[0016] ● According to a preset criterion, the intermediate results formed from the training examples of the subset are similar to each other.

[0017] Additionally, you can optionally inspect these intermediate results:

[0018] ● whether they are mapped to classification scores indicating at least the same priority category when further processed in one or more neural networks, and / or

[0019] ●Whether it is mapped to classification scores that are considered semantically similar according to the preset fusion strategy.

[0020] For example, if the representations of three consecutive images in a video data stream are similar to one another, but these representations are mapped to different priority classes (e.g. twice "passenger car" and once "minivan"), then a majority decision can be made, for example. A limousine and a convertible, which are classified into different classes, can also be considered similar to one another, for example, because they both belong to the superordinate class "passenger car". This depends on the respective application.

[0021] Training examples that fail this check can still be used for training with the contrastive loss, so they do not have to be discarded completely.

[0022] In this case, spatial and / or temporal filtering and other preprocessing may also be optionally performed. For example, objects that may be observed from multiple viewpoints may be suggested based on triangulation, odometer, simultaneous localization and mapping (SLAM) or other known algorithms. Such objects may also be used based on manual annotation of such objects to compare classification scores and intermediate results determined from different training examples with each other. Thus, the comparison does not have to involve the entire image content, for example, but can focus on the relevant objects.

[0023] If there are unlabeled training examples with the same priority category and similar intermediate results, the unlabeled training examples of the subset are converted into labeled training examples using the priority category as a label ("pseudo-label"). Then, the one or more neural networks are trained using the training examples upgraded in this way. The method can continue iteratively until a preset termination condition is met. The termination condition can include, for example, that there is no significant increase in training examples with new "pseudo-labels" from one iteration to another.

[0024] Thus, if, for example, several neural networks processing training examples recorded from different viewpoints agree that these training examples indicate the presence of an object of the "vehicle" class, and if at the same time the intermediate results generated from these training examples are sufficiently similar, then there is a high probability that these training samples actually indicate the presence of a vehicle. The originally unlabeled training examples can therefore from now on be used as training examples of the "vehicle" class.

[0025] For example, if an overtaking foreign vehicle is observed while monitoring the vehicle's surroundings, it is not possible for the foreign vehicle to be both in front of and behind the own vehicle at the same time. Rather, the foreign vehicle will be visible first behind, then beside, and finally in front of the own vehicle, and in this case shift between the detection areas of different cameras, which each see the vehicle from a different perspective. By adding the above filtering and preprocessing, pseudo labels can be obtained that are applicable to a similar degree as manually assigned labels.

[0026] If a vehicle is turning and is observed by only one camera during this process, the camera sees the vehicle from multiple perspectives. From these multiple perspectives, multiple images of the vehicle can be obtained, which are coupled to each other in a certain way, that is, they should not contradict each other.

[0027] With this training method, it is possible to iteratively increase the proportion of labeled training examples in the training examples, starting with only a small number of training samples. After the training is completed, the one or more neural networks can then be used immediately to classify further unseen measurement data. Independently of this, however, the training examples, which now have a larger proportion of labeled training examples than before, can also be used to train other neural networks.

[0028] This represents a significant cost savings for overall training, as manually labeling training examples is the largest driver of training cost.

[0029] In an advantageous embodiment, the cost function is used to evaluate the extent to which intermediate results obtained from the unlabeled training examples are similar to each other, the intermediate results being mapped by the one or more neural networks to at least the same priority category. The unlabeled training examples can then also be used to train the one or more neural networks to form the same intermediate results for the same object.

[0030] In a particularly advantageous embodiment, at least one neural network is selected that includes a feature extractor and a classifier. In this case, the training examples are fed to the feature extractor. The output of the feature extractor is fed to the classifier as an intermediate result. The contrastive loss can then act primarily on the parameters of the feature extractor, while the classification loss can act primarily on the parameters of the classifier.

[0031] The feature extractor may in particular for example comprise a sequence of multiple convolutional layers, each convolutional layer forming a feature map of its input by applying one or more filter kernels in a preset grid to its input. The last feature map in the sequence of feature maps thus generated has a dimension that is significantly smaller than, for example, an image used as a training example, but at the same time its dimension is still significantly larger than the final output classification score.

[0032] In particular, the classifier may include, for example, at least one fully networked layer. Such a layer may, for example, compress the feature map into a vector of classification scores for the available classes.

[0033] In one embodiment, the parameters of the one or more neural networks can be newly initialized for the training with the updated training examples. This embodiment has the advantage that the new training is based on a large set of labeled training examples from the outset and no erroneous developments in the parameters that could have been introduced by previous training with only a small number of labeled training examples are present. The trade-off is that the computing time invested in the previous training is also discarded.

[0034] In an alternative design, the training with the updated training examples is therefore based on the existing state of the parameters of the one or more neural networks. This design is particularly advantageous in the case where the existing training examples are very numerous and / or very complex. On the one hand, the computational overhead of discarding them with a complete restart of the training is relatively high. On the other hand, a large set of training examples can correct possible erroneous developments from previous training.

[0035] In a further particularly advantageous embodiment, records of measurement data recorded from different viewing angles and / or with different imaging modalities are fed to the one or more neural networks after training. These records are typically measurement data that have not been seen by the one or more neural networks in previous training. However, this is not mandatory. In this case, the term "record" is to be understood in conjunction with a database, similar to its English meaning. A record corresponds to a single entry in a database, which can have specific properties, similar to a single card in a card box. For example, a record can include an image, a radar scan or a lidar scan. The term "dataset" also applies, but in the field of machine learning refers to the totality of all records, similar to an entire card box.

[0036] By using the previously described training with "pseudo-labels", it is possible to achieve in practice a better ratio of classification accuracy to training overhead using records of measurement data not seen during training than in the case of training with only manually labeled training examples. Manual labeling is the "gold standard" in terms of accuracy, but the overhead is much higher than in the case of fully automated training with "pseudo-labels".

[0037] In a further advantageous embodiment, the similarity of the intermediate results determined from different recordings of the measurement data and the consistency of the priority classes determined in each case from these recordings are evaluated as an indication that these recordings indicate the presence of the same object in one or more detection areas of one or more sensors. The intermediate results still contain significantly more information than the maximally compressed classification score. In this way, in particular, for example, when a large number of objects are being identified simultaneously from the measurement data, "ghost detections" of object instances that are not actually present can be suppressed.

[0038] In another advantageous embodiment, an assessment can be made that the records indicate the presence of the same object in one or more detection areas, depending on whether the spatial and / or temporal relationship between the records meets a predefined condition. In this way, it can be taken into account that the same object cannot be in two places far apart at the same time in reality.

[0039] In another advantageous embodiment, measurement data or training examples recorded by multiple sensors with different spatial detection areas are selected. For example, the surroundings of the vehicle can be monitored using multiple sensors whose detection areas partially overlap so that the surroundings are fully covered.

[0040] The measurement data or training examples can include, in particular, camera images, video images, thermal images, ultrasound images, radar data and / or lidar data. Especially when monitoring the surroundings of a vehicle, more than one measurement modality is usually used. It is difficult to guarantee that a single measurement modality works properly in all cases and under all traffic conditions. For example, a camera can be overdriven by direct sunlight so that it only displays a white surface as an image. However, this interference does not act on the radar sensor that is operating simultaneously, so that at least limited observations can still be performed using the radar sensor. The training method proposed here can well guide one or more neural networks to merge the measurement data recorded using multiple measurement modalities into the detection of one or more objects.

[0041] In another advantageous embodiment, a control signal is determined from the output of one or more trained neural networks. The control signal is then used to control the vehicle, the driver assistance system, the system for quality control, the system for monitoring the area and / or the system for medical imaging. The probability that the respective controlled system reacts adequately to the situation represented by the recording of the input measurement data is thus advantageously increased. The use of pseudo labels during training also contributes in particular to improving the performance in the actual operation of the neural network. In particular, the probability that the controlled system reacts to "ghost detections" of objects in the measurement data is reduced. Such "ghost detections" could, for example, lead to the automatic full braking of the controlled vehicle without a reasonable (and visible to other traffic participants) reason for this.

[0042] In particular, the method can be fully or partially computer-implemented. Therefore, the invention also relates to a computer program having machine-readable instructions, which, when executed on one or more computers and / or computing instances, cause the one or more computers or computing instances to perform the described method. In this sense, control devices and embedded systems of technical devices of vehicles that are also capable of executing machine-readable instructions are also considered computers. The computing instance can be, for example, a virtual machine, a container or a serverless execution environment in which the machine-readable instructions can be executed.

[0043] Likewise, the invention relates to a machine-readable data carrier with a computer program and / or a download product. A download product is a digital product that can be transmitted via a data network, i.e. can be downloaded by a user of the data network, which can be offered for immediate downloading in an online store, for example.

[0044] Furthermore, one or more computers can be equipped with the computer program, the machine-readable data carrier or the download product. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Further measures for improving the invention will be described in detail below together with the description of preferred exemplary embodiments of the invention based on the drawings.

[0046] Figure 1 An embodiment of a method 100 for training one or more neural networks 1 is shown;

[0047] Figure 2 A schematic diagram of training according to method 100 is shown;

[0048] Figure 3 A schematic diagram of obtaining pseudo labels within the scope of method 100 is shown. DETAILED DESCRIPTION

[0049] Figure 1 is a schematic flow chart of an embodiment of a method 100 for training one or more neural networks 1. The one or more neural networks 1 process measurement data 2, in particular images, recorded from different viewing angles and / or with different measurement modalities into classification scores 4 for one or more classes of a preset classification.

[0050] In step 110, training examples 2a of measurement data 2 are provided. These training examples 2a include both training examples 2a1 labeled with target classification scores 2b and unlabeled training examples 2a2.

[0051] In step 120, the training examples 2a are processed by one or more neural networks 1 into a classification score 4. During this processing, intermediate results 3 are also detected, from which the classification score 4 is formed.

[0052] In step 130, the labeled training example 2a1 is evaluated using a preset loss function (loss function) 5,

[0053] How well the classification score 4 corresponds to the corresponding target classification score 2b (classification loss), and

[0054] ●To what extent the intermediate results 3 formed from training examples 2a1 with the same target classification score 2b are similar to each other.

[0055] Optionally, according to block 131, it is also possible to evaluate, using a loss function 5, how similar are the intermediate results 3 obtained from these training examples 2a2, which are mapped at least to the same priority class 4* by one or more neural networks 1, with respect to unlabeled training examples 2a2. Thus, training with respect to the contrastive loss can also utilize unlabeled training examples 2a2.

[0056] In step 140, the parameters 1a characterizing the behavior of the one or more neural networks 1 are optimized with the goal that the evaluation 5a by the loss function 5 is expected to improve when the training examples 2a1 are further processed. The final optimized state of the parameters 1a is denoted by the reference numeral 1a*. Thus, the state in which the training of the one or more neural networks 1 is completed is denoted by the reference numeral 1*.

[0057] In step 150, it is checked whether the intermediate results 3 formed for the subset of training examples 2a containing at least one unlabeled training example 2a2 are similar to each other according to a preset criterion 6. As mentioned above, the intermediate results 3 may also be optionally checked.

[0058] - is mapped to a classification score of 4 indicating at least the same priority category 4*, and / or

[0059] -Whether it is mapped to a classification score that is considered semantically similar according to a preset fusion strategy 4.

[0060] If the check result is positive (true value 1), the unlabeled training examples 2a2 of the subset are converted into labeled training examples 2a1 using the priority category 4* as label 2b in step 160. Thus, a set of upgraded training examples 2a* is obtained overall.

[0061] In step 170, one or more neural networks 1 are trained using these upgraded training examples 2a*.

[0062] In this case, the parameters 1a of one or more neural networks 1 can be newly initialized according to block 171.

[0063] Alternatively, according to block 172 , training with upgraded training examples 2 a * may be based on an existing state of parameters 1 a of one or more neural networks 1 .

[0064] exist Figure 1 In the example shown in , the termination condition of the training iteration is that no other unlabeled training examples 2a2 (true value 0) are found in step 150 to which new pseudo labels can be set.

[0065] After the training, records of measurement data 2 recorded from different viewing angles and / or using different imaging modalities are fed to the trained neural network or neural networks 1 *.

[0066] The similarity of the intermediate results 3 determined from different records of the measurement data 2 can then be evaluated in step 190 as an indication that these records indicate the presence of the same object in one or more detection areas of one or more sensors.

[0067] In this case, according to block 191 an assessment can be made that the records indicate the presence of the same object in one or more detection areas, additionally depending on whether the spatial and / or temporal relationship between the records meets a predefined condition.

[0068] In step 200 , actuation signal 200 a may be determined from output 4 of one or more trained neural networks 1 *.

[0069] In step 210 , vehicle 50 , driver assistance system 60 , system 70 for quality control, system 80 for monitoring an area and / or system 90 for medical imaging can then be controlled using control signal 200 a .

[0070] Figure 2 The diagram illustrates the state pursued using the previously described training. Figure 2 In the example shown in , there are some training examples 2a1 labeled with target classification scores 2b, and there are other training examples 2a1 labeled with other target classification scores 2b'. Figure 2 In the example shown in , for reasons of overview, the similarity of labeled training examples 2a1 is measured by whether these labeled training examples 2a1 belong to the same target category 2b.

[0071] The contribution of the classification loss to the cost function during the training process results in that training examples 4a1 labeled with a target classification score 2b (e.g. a "one-hot encoding" score for a particular class) are also accurately mapped by the one or more neural networks 1 to the class 2b as the preferred class 4*. The contribution of the contrastive loss to the cost function 5 results in that the intermediate results 3 generated for this purpose in the process are close to each other.

[0072] In contrast, the training example 2a1 marked with the target classification score 2b' is also mapped to the category 2b' which is the priority category 4*. Therefore, the intermediate result 3 generated in this process is also far away from the other intermediate results 3.

[0073] Figure 3 The diagram illustrates how pseudo labels are obtained. Figure 3In the example shown in , three unlabeled training examples 2a2 are mapped to the same priority category 4*. At the same time, the intermediate results 3 obtained in this case are close to each other and are therefore similar. In response to this, the priority category 4* is set as a new pseudo label 2b and assigned to the so far unlabeled training examples 2a2. Therefore, these training examples 2a2 become labeled training examples 2a1.

Claims

1. A method (100) for training one or more neural networks (1) for processing measurement data (2), in particular images, recorded from different viewing angles and / or using different measurement modalities into classification scores (4) for one or more classes of a predefined classification, the method comprising the following steps: ● providing (110) training examples (2a) of measurement data (2), said training examples comprising both training examples (2a1) labeled with target classification scores (2b) and unlabeled training examples (2a2); Processing (120) the training examples (2a) into classification scores (4) by the one or more neural networks (1), wherein intermediate results (3) used to form the classification scores (4) are also detected; ● With respect to the labeled training examples (2a1), evaluating (130) using a preset cost function (5), o How well the classification score (4) corresponds to the corresponding target classification score (2b), and ○ To what extent the intermediate results (3) formed from the training examples (2a1) that are similar to each other are similar to each other, and to what extent the intermediate results (3) formed from the training examples (2a1) that are dissimilar to each other are dissimilar to each other; Optimizing (140) parameters (1a) characterizing the behavior of the one or more neural networks (1) with the goal of improving the evaluation (5a) by the cost function (5) as the training examples (2a1) continue to be processed; Checking (150) whether intermediate results (3) formed for a subset of the training examples (2a) comprising at least one unlabeled training example (2a2) are similar to one another according to a predefined criterion (6); If this is the case, converting (160) the unlabeled training examples (2a2) of the subset into labeled training examples (2a1) using the priority class (4*) as label (2b); and • Training (170) the one or more neural networks (1) using the training examples (2a*) updated in this way.

2. The method according to claim 1, wherein additionally: an intermediate result formed from a subset of the training examples (2a) is checked ● is mapped to a classification score (4) that indicates at least the same priority category (4*), and / or ●Whether it is mapped to a classification score that is considered semantically similar according to the preset fusion strategy (4).

3. A method (100) according to any one of claims 1 to 2, wherein the cost function (5) is used to evaluate (131) with respect to the unlabeled training examples (2a2) to what extent intermediate results (3) obtained from the training examples (2a2) are similar to each other, the intermediate results being mapped by the one or more neural networks (1) to at least the same priority category (4*).

4. The method (100) according to any one of claims 1 to 3, wherein at least one neural network (1) is selected that includes a feature extractor and a classifier, wherein the training examples (2a) are fed to the feature extractor and the output of the feature extractor is fed to the classifier as an intermediate result (3).

5. The method (100) according to claim 4, wherein the feature extractor comprises a sequence of multiple convolutional layers, each convolutional layer forming a feature map of the input by applying one or more filter kernels to the input of the convolutional layer in a preset grid.

6. The method (100) according to any one of claims 4 to 5, wherein the classifier comprises at least one fully networked layer.

7. The method (100) according to any one of claims 1 to 6, wherein for training with the upgraded training examples (2a*), parameters (1a) of the one or more neural networks (1) are newly initialized (171).

8. The method (100) according to any one of claims 1 to 6, wherein the training with the upgraded training examples (2a*) is based (172) on an existing state of parameters (1a) of the one or more neural networks (1).

9. A method (100) according to any one of claims 1 to 8, wherein after training, a record (180) of measurement data (2) recorded from different viewing angles and / or using different imaging modalities is fed to one or more trained neural networks (1*).

10. The method (100) according to claim 9, wherein the similarity evaluation (190) of the intermediate results (3) determined from different records of the measurement data (2) is an indication for the records that the same object is present in one or more detection areas of one or more sensors.

11. The method (100) according to claim 10, wherein an assessment (191) is made that the records indicate the presence of the same object in one or more detection areas, additionally depending on whether the spatial and / or temporal relationship between the records meets a preset condition.

12. The method (100) according to any one of claims 1 to 11, wherein measurement data (2) or training examples (2a) recorded by a plurality of sensors having different spatial detection areas are selected.

13. The method (100) according to any one of claims 1 to 12, wherein measurement data (2) or training examples (2a) are selected comprising camera images, video images, thermal images, ultrasound images, radar data and / or lidar data.

14. The method (100) according to any one of claims 1 to 13, wherein Determining (200) a control signal (200a) from the output (4) of the trained one or more neural networks (1*), and ● Using the control signal (200a) to control (210) a vehicle (50), a driver assistance system (60), a system for quality control (70), a system for monitoring an area (80) and / or a system for medical imaging (90).

15. A computer program comprising machine-readable instructions which, when executed on one or more computers and / or computing instances, cause the one or more computers or computing instances to perform a method (100) according to any one of claims 1 to 14.

16. A machine-readable data carrier and / or download product having a computer program according to claim 15.

17. One or more computers having a computer program according to claim 15 and / or a machine-readable medium and / or a downloaded product according to claim 16.

Images