A method for training neural networks using semi-pseudo labels.

Abstract

The present invention provides a method for training a complex machine learning model based on a dataset including data points and ground truth labels for at least a portion of the data points of the dataset, the data points including images, the method comprising: using the reduced complexity machine learning model to predict pseudo labels for the dataset; training a complex machine learning model using the augmented dataset that includes the ground truth labels and pseudo labels as output; Includes.

Description

[Technical field]

[0001] The present invention relates to a method for training a complex machine learning model based on a dataset that includes data points and ground truth labels for at least a portion of the data points of the dataset.

[0002] The invention also relates to a computer readable storage medium storing program code including instructions for carrying out such a method. [Background technology]

[0003] Object detection is a key part of autonomous driving systems, as it builds increasingly complex layers on top of perception systems that themselves rely on fast and accurate obstacle detection. Object detection is typically performed by convolutional neural networks trained by supervised learning. Supervised learning is a method in which a model is fed with input data, and its main goal is to learn a function that maps the input data to a corresponding output. Convolutional neural networks, the best models of the visual domain, are highly over-parameterized, so large amounts of annotated data are required to learn the mapping function. Thus, a significant amount of manual effort is required to annotate the data with sufficient quality and quantity, which is an expensive and error-prone method. Furthermore, obtaining accurate ground truth data may be impossible due to human or sensor constraints. For example, the detection range of LiDAR limits the annotation of distant objects whose presence the autonomous driving system must know because their position and location on the highway change quickly. Radar can overcome this kind of limitation, but has a narrower field of view and only a subset of the categories of objects of interest are detectable by it. A human limitation is the inability to accurately estimate the spatial location of objects in 3D from 2D images without 3D cues, such as point clouds collected by LiDAR or radar detection. As a result, novel solutions are needed to overcome this limitation and extend current training methods to enable accurate 3D object detection for real-time applications.

[0004] Several techniques have been developed to facilitate neural network training. One of the most common solutions is transfer learning, where a neural network is trained on a specific dataset, such as ImageNet, and then fine-tuned on another dataset (e.g., a model trained to recognize cars can be trained to classify trucks using transfer learning). Self-supervised learning, which utilizes unlabeled data to train a model to perform a proxy task and then fine-tunes it in a supervised manner on downstream tasks, has led to breakthroughs in language models. Pseudo-labeling is a simple solution that uses the model's predictions that are the same as the true labels during training. However, none of these solutions help the model make predictions that are not part of the training distribution. Summary of the Invention

[0005] It is an object of the present invention to provide a method and apparatus for training complex machine learning models that overcomes one or more of the above-mentioned problems of the prior art.

[0006] A first aspect of the present invention provides a method for training a complex machine learning model based on a dataset comprising data points and ground truth labels for at least a portion of the data points of the dataset, the data points comprising images, the method comprising: using the reduced complexity machine learning model to predict pseudo labels for the dataset; training a complex machine learning model using the augmented dataset that includes the ground truth labels and pseudo labels as output; Includes.

[0007] The method of the first aspect has the advantage that an extended dataset can be automatically created based on an initial dataset, which may be "incomplete" in the sense that objects are visible in the image but are not labeled or are not fully labeled, and the extended dataset is used to train a (complex) machine learning model, allowing optimal machine learning even in the absence of training data.

[0008] Pseudo labels can be "pseudo" in the sense that they are not part of the ground truth dataset, but are estimated by a reduced-complexity machine learning model. Experiments show that these pseudo labels can help improve the accuracy of machine learning models, even if some of the pseudo labels turn out to be incorrect compared to human labeling.

[0009] The data points may be input data points in the sense that they are provided as input for training, and the labels comprise the output.

[0010] The data points include images. For example, a machine learning model may be trained to detect objects in images. As such, each data point may include zero, one or more objects. For example, the images may be images of the surroundings of the vehicle, and the objects may be objects related to navigating the vehicle, such as vehicles, motorcycles, pedestrians, traffic signs, and traffic lights.

[0011] In a first implementation of the method according to the first aspect, the ground truth labels comprise coordinates, dimensions and / or orientation of a 3D bounding box, the pseudo labels comprise coordinates, dimensions and / or orientation of a 2D bounding box, and / or the ground truth labels and / or the pseudo labels comprise objectness scores and / or probabilities for a set of object categories.

[0012] Here, the orientation of a 2D bounding box may refer to an angular direction, for example the angle of the 2D bounding box relative to the horizon.

[0013] This implementation has the advantage that it is possible to predict bounding boxes for objects.

[0014] It is understood that the 3D bounding box may correspond to a cuboid, while the 2D bounding box may be a rectangle. The objectness score may be a score indicating how likely it is that what is seen within said bounding box is an object.

[0015] In a further embodiment, the ground truth labels are labels of objects within annotated regions of the dataset, and the dataset includes unlabeled data points of objects outside the annotated regions of the dataset, preferably the annotated regions are the near field and the unannotated regions are the far field of the camera.

[0016] This embodiment can be used in particular in the context of obtaining a training data set for autonomous driving based on sensors with limited range. For example, the sensor can include one or more LiDAR sensors that can only identify objects within a certain near field of the LiDAR sensor or sensors. In such a case, objects within the required operating range can be seen in images obtained using the camera, but labels can be obtained using the LiDAR data only for objects that are within the range of the LiDAR sensor or sensors.

[0017] In a further embodiment, the method comprises: acquiring 3D data; augmenting the dataset with a virtual image based on a virtual camera that virtually captures the 3D data; Further comprising: Capturing the image includes moving a position of the virtual camera and acquiring a shifted and / or zoomed image from the moved camera position.

[0018] This implementation has the advantage that the original data set can be augmented with virtual images that are virtually captured using a virtual camera.

[0019] It will be appreciated that some of the images of the dataset may have been acquired with a real camera, while other images may have been acquired virtually using a virtual camera, in other words the dataset may only include virtual images acquired from the 3D data.

[0020] Preferably, the 3D data has been acquired using a LiDAR and / or using other 3D imaging systems.

[0021] Preferably, the 3D data is part of the dataset. The 3D data may include a label. As outlined above, annotations may be incomplete and some objects in the 3D data may be unlabeled. When the virtual camera moves, unlabeled objects may move into the virtual field of view of the virtual camera. Thus, pseudo-labeling of such unlabeled objects may be of particular relevance in this context.

[0022] In a further embodiment, acquiring images from the displaced camera positions comprises adjusting a camera matrix describing a mapping of positions in the 3D data to the 2D image. Preferably, the step of adjusting the camera matrix includes adjusting the principal point and focal length based on the original camera matrix and 2D scaling. It is understood that zooming in with the camera affects the camera matrix in a complex manner. Therefore, the virtual image acquisitions need to reflect these changes so that the virtual images acquired based on the virtual shifts in the camera position are valid training points, i.e., they accurately reflect how the actual images would look if acquired using a camera with such principal point and focal length.

[0023] In a further embodiment, the method includes performing de-duplication to remove pseudo labels for which the dataset already contains ground truth labels.

[0024] This has the advantage that, if ground truth labels are available, it can ensure that training is performed only on the ground truth labels and not on pseudo labels, which may be of lower quality.

[0025] In a further embodiment, the method further comprises an initial step of detecting, based on the image, an object for which a pseudo label needs to be predicted.

[0026] In this implementation, no manual annotation is required and off-the-shelf 2D object detectors can be used for this purpose, e.g., the FCOS detector, a fully convolutional one-stage object detector. In the preferred embodiment, a simple reduced complexity machine learning model is also used to detect objects.

[0027] Optionally, a simple algorithm can be used to detect the presence of an object, and a reduced complexity machine learning model is used to determine the object's label.

[0028] In a preferred embodiment, the step of performing deduplication comprises: determining a ratio of intersection to union of a 2D bounding box of the pseudo label and a 2D projection of a 3D bounding box of the ground truth label; removing the pseudo label from the augmented dataset if the ratio is greater than a predefined threshold; Includes.

[0029] This embodiment is applicable, for example, to a scenario in which a reduced complexity machine learning model predicts a label including a 2D bounding box based on one or more 2D images, e.g., stereo images. This object detection in 2D images can detect objects that are already labeled in the ground truth dataset. However, in this case, the 2D bounding box of the pseudo label is expected to have a high overlap with the 2D projection of the 3D bounding box of the ground truth label corresponding to the same object. Therefore, this embodiment can eliminate label overlap efficiently and with high accuracy.

[0030] In a further embodiment, training the complex machine learning model includes using a loss function that disables loss terms depending on available 3D characteristics for the given ground truth data. Preferably, a Boolean flag is used to indicate in the augmented dataset whether a label is a ground truth label or a predicted pseudo label.

[0031] This has the advantage that the training is not distorted by data points for which no ground truth information about their 3D positions is available in practice.

[0032] In a further embodiment, the method further comprises using complex machine learning models for object detection in autonomous driving.

[0033] Experiments have shown that the method of the first aspect is particularly useful in autonomous driving situations.

[0034] Preferably, the label comprises an object class, preferably comprising one or more of a vehicle, a motorbike, a pedestrian, a traffic sign, and a traffic light.

[0035] In a further embodiment, a complex machine learning model is used to predict the center point of the 3D cuboid based on a prediction of the 2D projection of the 3D cuboid and the predicted depth.

[0036] In a further embodiment, the depth is predicted based on the disparity, which is calculated based on the camera baseline, focal length, and depth. Experiments show that this results in a smoother loss landscape that accelerates convergence.

[0037] Optionally, the predicted depth can be adjusted based on the focal length of the actual camera, for example the camera used to capture the image in which the object is to be detected.

[0038] Camera settings different from the one used for the training dataset do not pose a problem: due to the use of 3D zoom, the process is invariant to camera settings with different focal lengths. The reason for this is that during training, we change the focal length of the camera to perform the zoom dilation. As a positive side effect, this makes the method less sensitive to cameras with different focal lengths, since the network learns to handle them.

[0039] Without the 3D zoom extension, the predicted depth for different camera settings is still incorrect due to the formulation of the depth target using a formula with parallax and focal length. However, 3D zoom helps to solve the problem.

[0040] In other embodiments, the adjustments can be based on the difference between the focal length of the camera used when applying the trained model and the focal length of the camera used to build the training dataset.

[0041] In a further embodiment, the method further comprises a de-duplication step of de-duping duplicate annotations.

[0042] In a further embodiment, the complex machine learning model is a multi-task machine learning model configured to solve a first task and a second task in parallel, and the reduced complexity machine learning model is configured to solve the first task.

[0043] This has the advantage that complex machine learning models can be trained with labels available only for the first task, but this training for the first task also leads to improvements for the (e.g., more difficult) second task.

[0044] Preferably, the first task comprises the prediction of a 2D bounding box and the second task comprises the prediction of a 3D bounding box.

[0045] In this way, complex machine learning models can learn to make accurate predictions in 3D based on training data with 2D labels as well as training data with 3D labels.

[0046] In a further embodiment of the method of the second aspect, the training of a complex machine learning model, in particular based on multi-view images such as stereo images, comprises: a projection loss term based on the 3D projection of a 3D cuboid; and A disparity loss term based on the disparity between the center points of the 2D projected 3D bounding boxes on the left and right stereo images; and A width loss term based on the difference between the widths of the 2D projections of the 3D bounding boxes on the left and right stereo images, The method includes using a loss term including one or more of:

[0047] Experiments show that these loss terms are particularly useful for obtaining accurate predictions.

[0048] A further aspect of the present invention refers to a computer-readable storage medium storing program code comprising instructions which, when executed by a processor, perform the method of the second aspect or one of the embodiments of the second aspect.

[0049] In order to more clearly describe the technical features of the embodiments of the present invention, the accompanying drawings provided for describing the embodiments are briefly introduced below. The accompanying drawings in the following description are only some embodiments of the present invention, and modifications to these embodiments are possible without departing from the scope of the present invention defined in the claims. [Brief description of the drawings]

[0050] [Figure 1] FIG. 13 is a schematic diagram of the effect of zoom expansion on image and model space data. [Diagram 2] 2 is a flow chart of a method according to the present invention; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0051] The improvement of machine learning methods by constructing extended datasets has been discussed in the prior art. For example, the concept of pseudo-labeling is introduced in [1] as a simple and efficient self-supervised method for deep neural networks. The main idea of ​​pseudo-labeling is to consider the predictions of a trained model as ground truth. Unlabeled data is usually easy to obtain and can be annotated using the predictions of the trained model. The same model is then retrained simultaneously on labeled and pseudo-labeled data. Our proposed solution is based on this concept, but there is a fundamental difference between the solutions. Pseudo-labeling uses the same model to generate labels for the same task, while our semi-pseudo-labeling method utilizes pseudo-labels generated by different models for more complex tasks. In [2], the authors used a convolutional neural network trained on an active pixel sensor to generate pseudo-labels for object detection on a dynamic vision sensor. The main difference compared to our solution is that pseudo-labels are used for the same task, i.e. 2D car detection with a different sensor modality. Also, the NVIDIA patent [3] uses pseudo-labels to train an object detection neural network. However, the invention described in the patent uses regular pseudo-labeling to train a neural network performing the same task, i.e., 2D object detection, in contrast to our solution, where the task is not the same. In addition, this solution requires the use of a neighborhood proposal network, which exhibits a two-stage network architecture that may not meet real-time criteria, while our 3D object detection network uses a single-stage architecture that utilizes semi-pseudo-labeling during training. IBM used pseudo-labeling in [4] to generate and augment their data labeling method. While their proposed solution created pseudo-labels for unlabeled data, our method allows for the use of annotated data created for simpler tasks as pseudo-labels and does not rely exclusively on unlabeled data.

[0052] Transfer learning [5], [6] is a process when a model is trained to perform a specific task and its knowledge is utilized to solve another (related) problem. Transfer learning involves pre-training a model (usually on a large dataset) and then customizing this model to a given task by fine-tuning the model by using the trained weights or by adding additional classifiers on top of the frozen weights. Transfer learning can be effectively used to transfer knowledge between related problems that have similarities to our proposed method, which can be considered as an extended version of transfer learning (e.g., knowledge of a classifier trained to recognize cats can be easily transferred to a model that classifies dogs). However, we utilize semi-pseudo labels to perform more complex tasks (3D object detection with 2D pseudo labels) that may not be solvable using regular transfer learning. Moreover, our solution, in contrast to transfer learning, allows for the simultaneous learning of different tasks.

[0053] Data augmentation [7] is a standard technique to extend training data and prevent overfitting using geometric transformations, color space expansion, random erasure, mixed images, etc. Most data augmentation techniques work in 2D

[18] , but recent work has started to extend their domain to 3D [8],

[18] . To the best of our knowledge, none of these solutions introduced zoom augmentation in 3D using a virtual camera, as we propose. The closest solution that attempts to solve the limited perception range is described in

[19] . The method in this paper proposes to decompose the entire image into multiple image patches, each of which contains at least one entire car and has limited depth variation. During inference, a pyramidal tiling of images is generated for inference, which increases the execution time. Furthermore, the investigation range did not exceed 50 meters.

[0054] The vanilla zoom expansion method has several problems, as shown in Figure 1. Specifically, Figure 1 shows an object in model space 100.

[0055] On the left side, a situation is shown where a first object 114a is within a first area 110a that is within the total coverage range 130 of the dataset. The total coverage range 130 may correspond to the near field 130 of the camera.

[0056] Since object 114a is within full coverage, the label of object 114a is known as the ground truth. A second object 116a is within a second area 132 that is outside the range of the labeled dataset, but is still within the required operating range 134. This situation leads to an original image 120a in which 2D representations 124a, 126a are visible.

[0057] In the center of Fig. 1, an original image 120a is shown to be reduced to obtain a reduced image 120b with reduced 2D representations 124b, 126b of the object, where the annotated area 110b has moved outside the limits 130 of the original dataset.

[0058] 1, the original image 120a has been enlarged to obtain an enlarged image 120c with enlarged views 124c, 126c, which corresponds to a model space in which both objects 114a, 116a lie within the range 130 of the original data set.

[0059] To train a network to predict 3D attributes of dynamic objects, accurate 3D position, size, and orientation data in the model space is required. To overcome the limitations of the dataset, the main problem to be solved is to handle unannotated objects 116a, 116b, 116c whose images are visible within the required operating distance range 134.

[0060] Areas 112a, 112b, 112c represent the unannotated areas with unannotated objects 116a, 116b, 116c. Range 134 represents the required operating region where the developed algorithm must detect all objects, and range 130 is the distance limit of the annotated data. The three columns represent three options during zoom dilation. The first case (left) is the original version without dilation. The second (center) case is where the input image is scaled down to mimic more distant objects in the image space, so the corresponding ground truth in the model space needs to be consistently adjusted. The third case (right) is where the input image is scaled up, bringing the objects closer to the camera. The figure highlights the discrepancy between applying various zoom levels, since the annotated areas on the transformed cases (second, third) overlap with the original unannotated areas.

[0061] FIG. 1 illustrates the discrepancies when applying vanilla zoom extension. The first area 110a, 110b, 110c represents the area where all images have visible objects annotated, while the second area 112a, 112b, 112c is where our annotations are incomplete and contain false negatives. When applying vanilla zoom extension techniques to extend the operating region of the developed algorithm, discrepancies can occur, i.e., when the zoom extension dataset contains the original reduced images of FIG. 1 (cases #1 and #2), it can be seen that the GT (ground truth) frames contradict each other. In case #2, we need to detect objects beyond the original GT limit (upper limit of the area 130), while in case #1, we do not even have information about the presence of unlabeled objects 116, so it cannot be utilized in the loss function. In our case, to overcome the limitation and make the zoom extension feasible, additional information is needed to fill in the missing data, i.e., at least the unannotated objects in the image space. Data shortages may be filled by human supervision, but this is not feasible as it is not scalable. Pseudo-labeling is a promising solution, but in our case it does not recover the entire 3D information, and the 2d information is sufficient to eliminate blockers and make zoom extension feasible, widening the limits of the dataset and extending the operating domain of the developed detection algorithm. A pre-trained state-of-the-art 2D bounding box network can be used to detect objects that are visible in all images. This additional information can be used to mask the corresponding loss term and overcome the dataset discrepancy.

[0062] The main goal of supervised learning is to define a mapping from the input space to the output by learning the values ​​of the parameters that formally minimize the error of an approximation function [9].

number

[0063] To train a model using supervised learning, a training set is required.

number

number

number

[0064] The pseudo-labeling method introduces a separate dataset where the labels of the unlabeled dataset are generated by a trained M(X;θ) model.

number

number

number

[0065] The final model is trained on the union of the annotated and pseudo-labeled datasets.

[0066] The main purpose of semi-pseudo-labeling is to generate a separate model M that performs more complex tasks. C (X CL ;θ C ) to train a model M S (X SL ;θ S) to generate pseudo-labels.

[0067] Both simple and complex tasks have annotated training sets for those specific tasks.

number

number

number

number

number

number

[0068] With regard to the above equations, it should be pointed out that the method is related to object detection where the number of output data points is different from the number of input data points (i.e., one image can contain several objects). Equations (2)-(6) represent a general form for the labels as (x, y) pairs, where x is the input image and y is the ground truth label of image x. In practice, this label can be a fixed-size array of objects (where objects are represented by their features). If the number of detectable objects is smaller than the size of the label vector, in embodiments it can be filled with zeros (indicating non-existent objects). For example, image 1 in FIG. 1 with two objects can be represented as a fixed-size array with five elements, where the first two elements are object features and the remaining three elements are empty objects.

[0069] label_for_img1=[(img1_obj1_x,img1_obj1_y,img1_obj1_z,…),(img1_obj2_x,img1_obj2_y,img1_obj2_z,…),(0,0,…),(0,0,…),(0,0,…)] Similarly, image 2 with three objects can be represented as follows:

[0070] label_for_img2=[(img2_obj1_x,img2_obj1_y,img2_obj1_z,…),(img2_obj2_x,img2_ob j2_y,img2_obj2_z,…),(img2_obj3_x,img2_obj3_y,img2_obj3_z,…),(0,0,…),(0,0,…)] Then the collective labels of the batches of image 1 and image 2 can be expressed as:

[0071] GT=[label_for_img1,label_for_img2] This is a schematic diagram for illustrative purposes only.

[0072] Although the above formula refers to m elements for the ground truth and augmented datasets, in practice, the "augmented dataset" typically contains more labels than the original ground truth. For example, image 1 has a car 20 meters away from the ego vehicle with a ground truth label, and a car 120 meters away from the ego vehicle found by a 2D detector and marked as a semi-pseudo label. The augmented label for this image is then represented as the union of the ground truth (available before pseudo labeling) and the pseudo-labeled distant objects.

[0073] Datasets for complex tasks (see Equation 5) typically do not contain information for all data points, e.g., there are distant cars in the images without ground truth labels.

[0074] With the originally available complex ground truth labels, the complex labels can contain all the information that the simple labels contain. For example, we can compute a 2D bounding box for an existing 3D bounding box annotation. This can be useful during neural network training, because the NN can detect pseudo-labeled (e.g., a car in the distance) objects with only known 2D features and, by zoom extension, learn the corresponding 3D features from the data, even if these 3D features are not explicitly marked in the label. This phenomenon is possible due to the generalization ability of neural networks.

[0075] Complex Model M C This explains how the simple model is run on the complex model dataset and pseudo labels are generated for the input images of the complex model dataset, insofar as pseudo labels are generated using the input data of the simple model. In this way, we can obtain labels for the objects visualized in Figure 1 that are not found in the original 3D ground truth.

[0076] Considering our use case, the ground truth parameters are the 2D and 3D attributes of the objects to be detected. The semi-pseudo-labeled ground truth consists of 2D information only and can come from manual annotations or as the output of a pre-trained 2D object detection neural network.

[0077] As outlined above, a simple model can be a 2D bounding box detector utilized to pseudo-label distant objects that are missing from a complex 3D ground truth for various reasons (e.g., there were no LiDAR returns from distant objects at all). A simple model can be a 2D object detector, and a complex model can be a 3D object detector.

[0078] FIG. 2 illustrates a method for training a complex machine learning model based on a dataset that includes data points and ground truth labels for at least a portion of the data points of the dataset, where the data points include images.

[0079] The method includes a first step 210 of using a reduced complexity machine learning model to predict pseudo labels for a dataset.

[0080] The method includes a second step 220 of training a complex machine learning model using the augmented dataset that includes the ground truth labels and pseudo labels as output.

[0081] The method may include an optional third step 230 of detecting objects using a trained complex machine learning model.

[0082] The above three steps 210, 220, 230 may be executed in different processing units. For example, the first step 210 and the second step 220 may be executed in different units in a data center. The trained complex machine learning model may be transferred to the vehicle and the third step 230 may be executed at least partially in the vehicle.

[0083] The semi-pseudo-labeling method in Figure 2 was developed to train a 3D object detection neural network to perform predictions outside the training data distribution. The requirement was to extend the detection range of the autonomous driving system to 200 meters while the distance range of the annotated data does not exceed 100 meters. In addition, some detectable classes were missing from the training data. However, the method is also applicable to other data and other application scenarios.

[0084] The preferred embodiment uses a YOLO

[10] -like single-stage object detector convolutional neural network (see details in Section 3.5) that leverages our semi-pseudo-labeling method and 3D data augmentation (see below). A regular 2D bounding box detector computes the number of objects in the input space X SL contains HD resolution stereo image pairs and the label space C S is (x, y, w, h, o, c1, ..., c n ) tuples, a simple model M S (X SL ;θ S ) in which x is the x coordinate of the center of the bounding box in image space, y is the y coordinate of the bounding box center in image space, w is the width of the bounding box in image space, h is the height of the bounding box in image space, o is the objectness score, · c i is the probability that the object belongs to the i-th category.

[0085] The model, in our case, is X SLWe perform 2D object detection on the 3D annotated dataset, i.e., the same as the HD images. The resulting 2D detections find distant objects that are not annotated by 3D bounding boxes and are added as semi-pseudo labels. Finally, a 3D object detector is trained on the combination of the 3D annotated data and the semi-pseudo labeled 2D bounding boxes. The label space of the 3D detector is (x, y, w, h, o, c1, ..., c n , P, D, O) tuple, where: x is the horizontal coordinate of the 2D projection of the 3D bounding box center, y is the vertical coordinate of the 2D projection of the 3D bounding box center, w is the width of the 2D projection of the 3D bounding box, h is the height of the 2D projection of the 3D bounding box, o is the objectness score, · c i is the probability that the object belongs to the i-th category. P is the 3D vector of the center point of the 3D bounding box in model space (which is reconstructed using the auxiliary regression instance-based disparity and the projection center point value cf, see Sec. 3.4), D is a 3D vector containing the dimensions (width, height, length) of the 3D bounding box, O is a 4D vector of the orientation of the 3D bounding box expressed as a quaternion.

[0086] A deduplication step is needed to avoid double annotations in the semi-pseudo-labeled dataset, which can be performed by examining the ratio of IoU (intersection over union) between the semi-pseudo-labeled annotation and the 2D projection of the 3D bounding box. If the ratio exceeds a threshold, the pseudo-labeled annotation should be filtered out.

[0087] Since semi-pseudo labels (i.e., 2D detection) do not contain 3D characteristics, the label space of semi-pseudo labels is more limited than the 3D label space. To avoid an ambiguous training process, we created a binary mask for semi-pseudo labels that was utilized to mask the 3D loss term in the semi-pseudo label case. With this solution and single-stage architecture and label space representation, we were able to simultaneously train a neural network to detect objects in 2D and 3D spaces.

[0088] Most 2D data augmentations are easy to generalize to 3D. However, zooming is not trivial because changes in image scale change the position and egocentric orientation of annotations in 3D space. We developed a 3D zoom augmentation using a virtual camera. The method consists of two main steps. The first is to zoom in or out of the image. In this way, we can emulate an object moving closer or farther from the camera. The second step is to modify the camera matrix (which describes the camera's mapping from 3D points in the world to 2D points in the image) to follow the 2D transformation and keep the 3D annotations intact. This can be done by a linear transformation and a virtual camera that adjusts its principal point and focal length to account for the original camera matrix and the 2D scaling transformation. The change in the camera intrinsic parameters mimics a change in the egocentric orientation of a given object, while its apparent orientation, which is a regression parameter during training, remains the same.

[0089] Applying a random shift of the images performs a decoupling of image location and object distance. This extension prevents the detection system from fixating on specific camera specifics. Calibrating the predicted depth on the actual camera focal length or regressing on the focal_length / disparity values ​​makes the trained network more robust to camera specific changes.

[0090] Preferably, the 3D zoom extension can be implemented by first randomly drawing the scaling factor between the lower and upper bounds. If the lower and upper bounds are less than 1, a zoom-out operation is performed. If the lower and upper bounds are greater than 1, a zoom-in operation is performed. If the lower bound is less than 1 and the upper bound is greater than 1, either a zoom-in or a zoom-out is performed. The 2D part of the zoom works similarly to the traditional case of scaling the image using the above-mentioned scaling factor (in the case of zoom-out, the image is padded with zeros to have the original image size). Then, the camera matrix corresponding to the image can be adjusted by scaling the focal length component with the scaling factor. If the 2D image is shifted out of zoom, the camera matrix can be adjusted by shifting the principal point component. Thus, the extension of the corresponding image and 3D label is performed in a consistent manner.

[0091] 3D augmentation has the advantageous effect that it is possible to accurately detect objects that are not part of the training distribution (i.e., to detect distant objects without ground truth labels), as mentioned above. Although it is true that semi-pseudo-labeling alone can be sufficient for detection, 3D properties, especially depth estimation, will be suboptimal due to the fact that neural networks cannot properly extrapolate outside the training distribution.

[0092] To prove the applicability of the proposed semi-pseudo labeling method and 3D data augmentation, we designed and implemented a convolutional neural network for stereo 3D object detection. The application domain of the neural network is autonomous driving. Thus, the detectable objects are vehicles, motorcycles, pedestrians, traffic signs, and traffic lights. Above, we have described the input and output of the detection network.

[0093] aiMotive has developed a data collection and auto-annotation method to create training data for 3D object detection networks. The data collection system utilizes LiDAR to scan the environment in 3D and an auto-annotation algorithm to generate ground truth training data. One of the limitations of this system is the perception range of the LiDAR, which reaches 200 meters as a detection range, while detected objects are closer than 100 meters. The semi-pseudo labeling method and 3D data augmentation are a solution to this problem.

[0094] Since real-time performance is essential for autonomous driving, we designed a single-stage detection architecture like YOLOv3

[10] . As a first step, the left and right images of a stereo image pair can be fed into a proprietary neural network. The left and right branches of the network use shared backbone weights. Once the left and right image pairs are processed by the backbone, the resulting embeddings are fused (we used concatenation as the fusion method) before being passed to a feature pyramid network

[11] .

[0095] The 3D label space representation described above is used in the network to predict the 2D and 3D properties of detected objects. The ground truth was augmented with a Boolean flag indicating whether the annotated object was semi-pseudo labeled or not. This value was used in the loss function to mask the 3D loss term in the semi-pseudo label case so as not to penalize the weights corresponding to the 3D properties during backpropagation when the ground truth is not known.

[0096] We train the network using multi-task learning

[19] , where 2D and 3D detection are learned in parallel. To this end, we exploit additional constraints to facilitate training. In addition to the 2D (x, y, w, h) and 3D (position, orientation, size) properties, we use the following loss term:

[0097] 2D projection of a 3D cuboid, Instance-based parallax, Instance-based width differences.

[0098] Instead of directly learning the 3D center point of the cuboid, we designed a network that predicts the 2D projection of the 3D cuboid. The center point of the 3D bounding box can be reconstructed later from the depth and its 2D projection. The network can be trained using the depth directly; however, we replaced it with a disparity that can be calculated from a formula that takes into account the camera baseline, focal length, and depth (disparity = focal_length * base_distance / depth). This representation resulted in a smoother loss landscape that accelerated convergence. The instance-based disparity and width difference are two auxiliary loss terms that calculate the disparity between the center points of the 2D projected 3D bounding boxes on the left and right images and the width difference of the 2D projection of the 3D bounding box on the left and right images, respectively. Note that the two instance-based loss terms are not used during inference and their only purpose is to facilitate training. To the best of our knowledge, this formulation has not been found in the literature. Finally, the dimension prediction part of the network uses a prior distribution (i.e., precomputed category means), and only deltas are predicted instead of directly regressing the dimensions.

[0099] As mentioned before, we framed the training of the network as a multi-task learning problem. Our loss function consists of two parts: 2D and 3D loss terms. The loss function for 2D features is adapted from the YOLO paper. For the 3D loss term, we innovatively combine several machine learning techniques besides instance-based disparity and width difference losses. The first technique used is to lift the loss to 3D instead of computing the loss for a specific individual loss term (e.g., 2D projection of the cuboid center point, orientation, etc.). The 3D loss is computed by reconstructing the bounding cuboid in 3D and then computing the L2 loss for the predicted corner points and the ground truth corner points of the cuboid. Another trick is to disentangle the loss terms as proposed by

[12] . As mentioned before, we utilized a masking solution to avoid penalizing the network when predicting 3D features for semi-pseudo labels that do not have 3D annotations. The final loss is the sum of the 2D loss and the 3D loss.

[0100] We trained the network using automatically annotated 3D and semi-pseudo-labeled datasets. As an additional dataset extension, we added a manually annotated 2D bounding box dataset to the training set as another semi-pseudo-labeled dataset.

[0101] There are several other existing solutions to the 3D detection problem. However, these methods most often require slower two-stage detector architectures

[13] ,

[14] ,

[15] ,

[16] , alternative sensor modalities

[17] , or have significantly shorter detection ranges

[13] .

[0102] The 3D object detection neural network described above meets autonomous driving standards, works in real time, and significantly increases the detection range. Inference time on a Titan XP GPU is about 18 ms (about 55 FPS). By using quantization and storing the weights as integers instead of 32-bit floating point, this execution time can be further reduced by about 2x. With semi-pseudo labeling and 3D data augmentation, the neural network is able to detect objects up to 200 meters away, even though the training dataset did not contain any objects beyond 130 meters.

[0103] Since we formulated the depth prediction taking into account the focal length and baseline, our solution is a general method that works with any camera and stereo rig setup. Another advantage is that the network only requires images in inference mode, which prevents the use of expensive lidar and reduces the complexity of sensing.

[0104] One typical use case of the proposed solution and the designed neural network is an SAE Level 2+ highway autopilot software. The detector can be an integral part of the perception system, which must be able to recognize distant static and dynamic objects in order to provide useful information to higher levels.

[0105] The semi-pseudo-labeling training method can be integrated into other object detectors or used for other application areas. In our case, a stereo 3D object detector is developed, but it is easy to design a mono 3D detector that follows the ideas of the proposed method.

[0106] 3D object detectors trained with the presented method can be utilized in highway or urban environments. Due to its design, the network can be used with different camera and stereo rig setups.

[0107] One area of ​​exploitation is the cost reduction during the annotation process: cheaper LiDAR can be used as part of the recording setup, and the perception range can be extended by semi-pseudo labeling. The cost of human annotation can also be reduced by using 3D boxes generated by the presented network as pseudo labels and refined by manual annotators.

[0108] References [1]Pseudo-Label:The Simple and Efficient Semi-Supervised Learning Method for Deep Neural Networks- https: / / scholar.google.com / scholar?oi=bibs&cluster=16547318329102522555&btnI=1&hl=ja [2] Chen, Nicholas FY. "Pseudo-labels for supervised learning on dynamic vision sensor data,applied to object detection under ego-motion." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 2018. [3] https: / / patents.google.com / patent / US20200394458A1 / en?oq=US20200394458A1 [4] https: / / patents.google.com / patent / US20200082210A1 / en?oq=US20200082210A1 [5] Stevo. Bozinovski and Ante Fulgosi (1976), "The influence of pattern similarity and transfer learning upon training of a base perceptron B2." (original in Croatian) Proceedings of the Symposium Informatica 3-121-5. [6] Stevo Bozinovski (2020) “Reminder of the first paper on transfer learning in neural networks,1976” Informatica 44:291-302. [7] Shorten, C., & Khoshgoftaar, T. M. (2019). A survey of image data augmentation for deep learning. Journal of Big Data, 6(1), 1–48. [8] Xu, J., Li, M., & Zhu, Z. (October 2020). Automatic data augmentation for 3d medical image segmentation. In International Conference on Medical Image Compu-ting and Computer-Assisted Intervention, pp. 378-387. Springer, Cham. [9] Ian Goodfellow, Yoshua Bengio, and Aaron Courville (2016). Deep Learning. MIT Press.

[10] Redmon, J., & Farhadi, A. (2018), Yolov3: An incremental improvement.arXiv preprint arXiv:1804.02767.

[11] Lin, T.Y., Dollar, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017). Feature pyramid networks for object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 2117-2125.

[12] Simonelli, A., Bulo, S. R., Porzi, L., Lopez-Antequera, M., & Kontschieder, P. (2019). Disentangling monocular 3d object detection. In Proceedings of the IEEE / CVF International Conference on Computer Vision, pp. 1991-1999.

[13] Qian, R., Garg, D., Wang, Y., You, Y., Belongie, S., Hariharan, B., ...& Chao, W. L. (2020). End-to-end pseudo-lidar for image-based 3d object detection. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (pp. 5881-5890).

[14] Qin, Z., Wang, J., & Lu, Y. (2019). Triangulation learning network: from monocular to stereo 3d object detection. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp. 7615-7623.

[15] Li, P., Chen, X., & Shen, S. (2019). Stereo r-cnn based 3d object detection for autonomous driving. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp. 7644-7652.

[16] TuSimple patent:https: / / patents.google.com / patent / US10839234B2 / en?oq=US10839234B2

[17] https: / / patents.google.com / patent / US20190188541A1 / en?oq=US20190188541A1

[18] Liu, Z., Wu, Z., & Toth, R. (2020). Smoke: single-stage monocular 3d object detection via keypoint estimation. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 996-997.

[19] Simonelli, A., Bulo, SR, Porzi, L., Ricci, E., & Kontschieder, P (2019), Towards Generalization Across Depth for Monocular 3D Object Detection.arXiv pre-print arXiv:1912.08035.

[19] Caruana, R. (1997). Multitask learning. Machine Learning, 28(1), 41-75.

Claims

1. 1. A method for training a complex machine learning model based on a dataset including data points and ground truth labels for at least a portion of the data points of the dataset, the data points including images; using (210) a reduced complexity machine learning model to predict pseudo labels for the dataset; training (220) the complex machine learning model using an augmented dataset that includes the ground truth labels and the pseudo labels as output; A method comprising:

2. the ground truth labels include coordinates, dimensions, and / or orientation of a 3D bounding box; The pseudo-label includes the coordinates, dimensions, and / or orientation of a 2D bounding box; and / or the ground truth labels and / or pseudo labels include objectness scores and / or probabilities for a set of object categories; The method of claim 1.

3. 3. The method of claim 1, wherein the ground truth labels are labels of objects within annotated regions (110a, 110b, 110c) of the dataset, the dataset including unlabeled data points for objects outside the annotated regions of the dataset, preferably the annotated regions corresponding to a near field (130) of a camera and the unannotated regions (112a, 112b, 112c) corresponding to a far field (132) of the camera.

4. acquiring 3D data; augmenting the dataset with a virtual image based on a virtual camera that virtually captures the 3D data; Further comprising: the step of acquiring the image includes the step of moving a position of the virtual camera to acquire a shifted and / or zoomed image from the moved camera position; 4. The method according to any one of claims 1 to 3.

5. 5. The method of claim 4, wherein the step of acquiring an image from the moved camera position comprises adjusting a camera matrix describing a mapping of positions in the 3D data to a 2D image, preferably wherein the step of adjusting the camera matrix comprises adjusting the principal point and the focal length based on an original camera matrix and a 2D scaling.

6. an initial step of detecting objects based on said images, for which pseudo labels need to be predicted; and / or performing de-duplication to remove pseudo-labels for which the dataset already contains ground truth labels; The method of claim 1 , further comprising:

7. said step of performing deduplication further comprising: determining a ratio of intersections to unions of 2D bounding boxes of the pseudo labels and 2D projections of 3D bounding boxes of the ground truth labels; if the ratio is greater than a predefined threshold, removing the pseudo label from the augmented dataset; Including, The method according to claim 6.

8. 8. The method of claim 1, wherein the step of training the complex machine learning model comprises using a loss function that does not penalize error contributions corresponding to 3D features for which ground truth labels are unavailable, and preferably a Boolean flag is used to indicate in the augmented dataset whether a label is a ground truth label or a predicted pseudo label.

9. 9. The method of any one of claims 1 to 8, further comprising using the complex machine learning model for detecting objects during automated driving, wherein the labels comprise object classes preferably comprising one or more of: vehicles, motorcycles, pedestrians, traffic signs, and traffic lights.

10. The method of claim 1 , wherein the complex machine learning model is used to predict a center point of a 3D cuboid based on a prediction of the 2D projection of the 3D cuboid and a predicted depth.

11. The method of claim 10 , wherein the depth is predicted based on a disparity calculated based on a camera baseline, focal length and depth, and / or the predicted depth is adjusted based on a current camera focal length.

12. 12. The method of claim 1, wherein the complex machine learning model is a multi-task machine learning model configured to solve a first task and a second task in parallel, and the reduced complexity machine learning model is configured to solve the first task, preferably the first task comprising predicting a 2D bounding box and the second task comprising predicting a 3D bounding box.

13. training said complex machine learning model, a projection loss term based on a 3D projection of a 3D cuboid; and a disparity loss term based on the disparity between the center points of the 2D projected 3D bounding boxes on the left and right stereo images; a width loss term based on the difference in width of the 2D projection of the 3D bounding box on the left and right stereo images; The method of claim 1 , comprising using loss terms including one or more of:

14. Apparatus configured to carry out the method according to any one of claims 1 to 13.

15. A computer readable storage medium storing program code comprising instructions which, when executed by a processor, performs the method of any one of claims 1 to 13.

Images