Self-learning label refinement system and method for improving monocular object detection

Abstract

To provide systems and methods for self-learned label refinement for generating a higher quality training set.SOLUTION: A method 600 for generating a training set causes a processor to train a model using an image as an input to the model and 2D bounding based on 3D bounding boxes as ground truths 604, select a first subset from predicted 2D bounding boxes previously outputted by the model 606, retrain the model using the image as the input and the first subset as ground truths 608, select a second set of predicted 2D bounding boxes previously outputted by the model 610, and generate the training set by selecting the 3D bounding boxes from a master set of 3D bounding boxes that have corresponding 2D bounding boxes that form the second subset 612.SELECTED DRAWING: Figure 11

Description

[Technical Field]

[0001] The subject matter described herein generally relates to systems and methods for self-learning label refinement to improve monocular object detection. [Background technology]

[0002] The background art description provided is intended to generally present the context of the disclosure. The inventor's inventions, to the extent that they can be described in this background art section, and aspects of the description that may not qualify as prior art at the time of filing, are not admitted expressly or impliedly as prior art against the present technology.

[0003] Some neural network models are capable of three-dimensional (3D) monocular object detection. Furthermore, these neural network models receive as input images captured by an imaging device such as a camera. Neural network models have been trained to identify objects located within the images in 3D space and to create appropriate 3D bounding boxes around these images. This is particularly challenging because the input images are two-dimensional (2D) in nature.

[0004] These neural network models may be trained in a variety of different ways. For supervised training, which requires an annotated training set that serves as ground truth, the accuracy of the annotations in the training set directly impacts the training and, therefore, the performance of these neural networks. Compounding this problem is that creating training sets for training models that perform monocular 3D object detection is very expensive, because the annotations must identify the 3D location of objects on the 2D plane that is the image.

[0005] To create the 3D location of objects, some annotations are based on point cloud information captured from light detection and ranging (LIDAR) sensors. While these training sets can provide useful data for creating annotations, they have drawbacks. For example, the LIDAR sensor used to capture the point cloud information and the camera used to capture the corresponding images may not be precisely aligned, resulting in parallax issues. Furthermore, the timing between when the point cloud is created by the LIDAR sensor and when the camera captures the images may not be precisely synchronized, resulting in synchronization issues. Parallax in synchronization issues may result in the creation of faulty annotations. Using a training set with faulty annotations to train a model may affect the model's ultimate performance. Summary of the Invention

[0006] This section generally summarizes the disclosure and is not an all-inclusive description of its entire scope or all of its features.

[0007] In one embodiment, a system for filtering and refining labels for a training set includes a processor and a memory coupled to the processor. The memory includes a training set creation module having instructions for causing the processor to create a training set of 3D bounding boxes by filtering out 3D bounding boxes from a master set. To accomplish this, the training set creation module causes the processor to create 2D bounding boxes for objects based on a master set of 3D bounding boxes for objects in images, and causes the processor to train a model using the images as input and the 2D bounding boxes as ground truth. During training, the model outputs a first set of predicted 2D bounding boxes and a confidence score for the first set of predicted 2D bounding boxes.

[0008] The training set creation module then causes the processor to select a first subset from the first set of predicted 2D bounding boxes based on the confidence scores for the first set of predicted 2D bounding boxes, and causes the processor to retrain the model using the images as input and the first subset as ground truth, During retraining, the model outputs a second set of predicted 2D bounding boxes and confidence scores for the second set of predicted 2D bounding boxes.

[0009] The training set creation module then causes the processor to create a training set by selecting a second subset of predicted 2D bounding boxes from the second set based on the confidence scores for the second set of predicted 2D bounding boxes and selecting 3D bounding boxes from the master set of 3D bounding boxes having corresponding 2D bounding boxes that form the second subset.

[0010] In another embodiment, a method for filtering and refining labels of a training set includes creating 2D bounding boxes for objects based on a master set of 3D bounding boxes for objects in images, and training a model using the images as input and the 2D bounding boxes as ground truth, wherein the model outputs a first set of predicted 2D bounding boxes and a confidence score for the first set of predicted 2D bounding boxes.

[0011] The method then includes selecting a first subset from the first set of predicted 2D bounding boxes based on the confidence scores for the first set of predicted 2D bounding boxes, and retraining the model using the images as input and the first subset as ground truth, where again the model outputs a second set of predicted 2D bounding boxes and confidence scores for the second set of predicted 2D bounding boxes.

[0012] The method then includes the steps of selecting a second subset of predicted 2D bounding boxes from the second set of predicted 2D bounding boxes based on the confidence scores for the second set of predicted 2D bounding boxes, and creating a training set by selecting 3D bounding boxes from the master set of 3D bounding boxes with corresponding 2D bounding boxes that form the second subset.

[0013] In yet another embodiment, a method for filtering and refining labels for a training set may include training a model using images as input and 2D bounding boxes as ground truth based on a master set of 3D bounding boxes, where the model outputs a set of predicted 2D bounding boxes and a confidence score for the set of predicted 2D bounding boxes. The method may then perform a step of selecting a subset from the set of predicted 2D bounding boxes based on the confidence score for the first set of predicted 2D bounding boxes. The method may then perform a step of creating a training set by selecting 3D bounding boxes from the master set of 3D bounding boxes with corresponding 2D bounding boxes that form the subset.

[0014] Further areas of applicability and various ways of enhancing the disclosed technology will become apparent from the description provided. The description and specific examples within this Summary are intended for purposes of illustration only and are not intended to limit the scope of the disclosure. [Brief explanation of the drawings]

[0015] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate various systems, methods, and other embodiments of the disclosure. It should be understood that element boundaries (e.g., boxes, groups of boxes, or other shapes) depicted in the figures represent one embodiment of the boundaries. In some embodiments, one element may be designed as multiple elements, or multiple elements may be designed as one element. In some embodiments, an element shown as an internal component of another element may be implemented as an external component, and vice versa. Additionally, elements may not be drawn to scale.

[0016] [Figure 1] FIG. 1 illustrates a scene where data is collected using a LIDAR sensor and a camera to create training data for training a model for 3D monocular object detection. [Figure 2A] FIG. 1 illustrates a point cloud and a 3D bounding box identifying an object created from the point cloud. [Figure 2B] FIG. 2B illustrates an image including the 3D bounding box of FIG. 2A identifying an object located within the image. [Figure 3] FIG. 1 illustrates an example of a 3D bounding box in an image that is misaligned with an object located in the image. [Figure 4] FIG. 10 illustrates another example of a 3D bounding box in an image that identifies an object obscured by other elements located in the image. [Figure 5] FIG. 1 illustrates a training set creation system for filtering out 3D bounding boxes in a master training set to create a higher quality training set. [Figure 6] FIG. 10 shows a flowchart detailing the initial training of a 2D monocular object detection model, which is used to filter out 3D bounding boxes in a master set to create a higher quality training set. [Figure 7] FIG. 1 illustrates an example of a projection of a 2D bounding box based on a 3D bounding box. [Figure 8]FIG. 10 shows a flowchart detailing the second training of a 2D monocular object detection model, which is used to filter out 3D bounding boxes in the master set to create a higher quality training set. [Figure 9] FIG. 10 shows a flowchart detailing the ranking and selection of a portion of the 3D bounding boxes in the master set to create a higher quality training set. [Figure 10] FIG. 1 shows a flowchart detailing the training of a 3D object detection model using a higher quality training set derived from a master set. [Figure 11] FIG. 1 illustrates how to filter out 3D bounding boxes in a master training set to create a higher quality training set. DETAILED DESCRIPTION OF THE INVENTION

[0017] Systems and methods are described for creating a training set by filtering and / or refining the labels of the training set. The training set may then be used to train a monocular 3D object detection model. Furthermore, in one example, the monocular 3D object detection model may identify one or more objects in 3D space within images captured by an image sensor. To accomplish this, the monocular 3D object detection model may first need to be trained using ground truth 3D bounding boxes. However, as discussed in the Background section, the ground truth 3D bounding boxes may be based on point cloud data captured from a LIDAR sensor and may suffer from disparity and / or synchronization issues.

[0018] The systems and methods described herein utilize a 2D object detection model to filter out 3D bounding boxes from a master set that may have errors due to disparity and / or synchronization issues. The systems and methods first convert the 3D bounding boxes from the master set into 2D bounding boxes that identify objects in the image. When creating the 2D bounding boxes, information about the relationship between a particular 2D bounding box and the 3D bounding box on which it is based may be captured and later used to select an appropriate 3D bounding box for training a monocular 3D object detection model.

[0019] After training the 2D object detection model using the 2D bounding boxes as ground truth and the associated images as input, the 2D object detection model outputs a first set of predicted 2D bounding boxes and associated confidence scores. The confidence scores are used to filter the first set of predicted 2D bounding boxes into a subset. The subset is then used to retrain the 2D object detection model. Similarly, after retraining, the 2D object detection model outputs a second set of predicted 2D bounding boxes and associated confidence scores. The second subset is selected from the second set of predicted 2D bounding boxes based on the confidence scores.

[0020] Therefore, the 2D bounding boxes forming the second subset are likely to correctly identify objects in the image in 2D space. Using the 2D bounding boxes from the second subset, corresponding 3D bounding boxes from the master set are identified and selected to form a training set. By doing so, the 3D bounding boxes with corresponding 2D bounding boxes from the second subset should be of high quality and less prone to parallax and / or synchronization issues. The 3D bounding boxes forming the training set can then be used to train a monocular 3D object detection model.

[0021] To better understand how the system and method work, it will be explained how a ground truth 3D bounding box is first created. Referring to FIG. 1 , a scene 10 including a road 11 is shown. A vehicle 12 including a LIDAR sensor 14 and a camera sensor 16 is positioned on the road 11. By targeting an object with a laser and measuring the time it takes for the reflected light to return to a receiver in the LIDAR sensor 14, the LIDAR sensor 14 can determine distance. In one example, the LIDAR sensor 14 may output the distance to the object in the form of a point cloud. The camera sensor 16 may be any type of image sensor and can capture images of the scene 10.

[0022] In this example, scene 10 includes vehicles 20 and 22 located on road 11. Here, LIDAR sensor 14 may output information that can be used to create a point cloud that includes points representing vehicles 20 and 22. Similarly, camera sensor 16 may output images that include vehicles 20 and 22. It should be noted that in this example, there are alignment issues with LIDAR sensor 14 and camera sensor 16. Furthermore, LIDAR sensor 14 and camera sensor 16 are mounted on vehicle 12 at slightly different locations. As such, this alignment difference can create parallax issues. In addition to parallax issues, it should be noted that camera sensor 16 and LIDAR sensor 14 may capture images and point clouds, respectively, at slightly different moments in time, which can create synchronization issues.

[0023] As previously described, a monocular 3D object detection model can receive input images from an image sensor, such as a camera sensor, and output 3D bounding boxes that identify objects in the images in 3D space. The 3D object detection model may be one or more neural networks that may need to be trained. In one example, the 3D object detection model may be trained in a supervised manner, where images are provided to the 3D object detection model, which outputs predicted 3D bounding boxes for objects in the images. The predicted 3D bounding boxes are then compared to ground truth 3D bounding boxes to generate a loss. Based on this loss, one or more model weights of the 3D object detection model are adjusted. During the training process, the performance of the 3D object detection model should improve over time.

[0024] 3D bounding boxes used as ground truth for training 3D object detection models are typically based on point cloud information created by a LIDAR sensor, such as LIDAR sensor 14. Further, with reference to FIGS. 2A and 2B, an example of a point cloud 30 and image 40 used to create a ground truth 3D bounding box for training purposes is shown. Here, points from point cloud 30 were used to identify objects within point cloud 30. In this example, objects within point cloud 30 were identified by 3D bounding boxes 32A-32E and 34A-34B. 3D bounding boxes 32A-32E identified vehicles, while 3D bounding boxes 34A-34B identified pedestrians. As best shown in FIG. 2B, 3D bounding boxes 32A-32E and 34A-34B created from points from point cloud 30 were overlaid on image 40. In this example, a monocular 3D object detector model will be trained using image 40 as input to the model, and the 3D bounding boxes 32A-32E and 34A-34B will serve as ground truth for training the model in a supervised fashion.

[0025] However, as previously mentioned, because the 3D bounding boxes that serve as ground truth are based on point clouds created by LIDAR sensors, there may be parallax and / or synchronization issues. For example, Figure 3 shows an image 50 that includes a vehicle 54. A 3D bounding box 52 is shown misaligned with the vehicle 54. Because the 3D bounding box 52 is not properly aligned with the vehicle 54, using the 3D bounding box 52 to train a model may be counterproductive.

[0026] 4 illustrates another common problem. Here, image 60 contains vehicle 64 and vehicle 70, identified by bounding boxes 66 and 72, respectively. Typically, vehicle 64 is correctly identified by bounding box 66. However, vehicle 70 is obscured by foliage 68. Therefore, vehicle 70 is not readily visible in image 60 and will not be identified by a monocular 3D object detection model, regardless of the amount of training. Therefore, using bounding box 72 to train the model may be counterproductive.

[0027] FIG. 5 illustrates a training set creation system 100 that can filter out 3D bounding boxes to create a better training set, such as those shown in FIGS. 3 and 4. As shown, the training set creation system 100 includes one or more processors 110. Thus, the processor 110 may be part of the training set creation system 100, or the training set creation system 100 may access the processor 110 through a data bus or other communication path. In one or more embodiments, the processor 110 is an application-specific integrated circuit configured to implement the functionality associated with the training set creation module 122. Typically, the processor 110 is an electronic processor, such as a microprocessor, capable of performing the various functions described herein. In one embodiment, the training set creation system 100 includes memory 120 that stores the training set creation module 122. The memory 120 may be random access memory (RAM), read-only memory (ROM), a hard disk drive, flash memory, or other suitable memory for storing the training set creation module 122. The training set creation module 122 is, for example, computer-readable instructions that, when executed by the processor 110, cause the processor 110 to perform various functions disclosed herein.

[0028] Additionally, in one embodiment, training set creation system 100 includes one or more data stores 130. In one embodiment, data store 130 is an electronic data structure, such as a database, stored in memory 120 or other memory and configured with routines that can be executed by processor 110 to analyze, provide, organize, create, and the like. Thus, in one embodiment, data store 130 stores data used by training set creation module 122 in performing various functions. In one embodiment, data store 130 stores master training data 140, which may include information for training monocular 3D object detection model 170. Furthermore, master training data 140 may include images 142 having a master set 144 of ground truth 3D bounding boxes 144A-144H annotated to identify objects in images 142.

[0029] Monocular 3D object detection model 170 may take any of a number of different forms. In one example, monocular 3D object detection model 170 may be one or more neural networks capable of receiving images and outputting 3D bounding boxes that identify objects in 3D space within the received images. Monocular 3D object detection model 170 may be utilized in a number of different applications. In one such application, monocular 3D object detection model 170 may be utilized in an autonomous vehicle application.

[0030] As previously mentioned, some 3D bounding boxes that serve as ground truth, such as some of the 3D bounding boxes 144A-144H of the master set 144, suffer from problems such as disparity, synchronization, and other issues. As described herein, the training set creation module 122 causes the processor 110 to utilize the monocular 2D object detection model 160 to filter out the 3D bounding boxes 144A-144H of the master set 144 to create the training set 145. In this example, the training set creation system 100 filters out some of the 3D bounding boxes from the master set 144, leaving the training set 145 with 3D bounding boxes 144A, 144D, 144F, and 144H. The 3D bounding boxes 144A, 144D, 144F, and 144H will be deemed by the training set creation system 100 to be of higher quality in that they suffer less from disparity and / or synchronization issues, and / or other issues. Thus, the filtered training data 150 for training would include the training set 145 and the images 142 .

[0031] It should be noted that the master training data 140 and the filtered training data 150 include a single image, and the single image is shown as having multiple 3D bounding boxes that identify objects within the single image. However, it should be understood that the master training data 140 and / or the filtered training data 150 may include multiple images, and each of the multiple images may include any number of bounding boxes that serve as ground truth for identifying objects within the image.

[0032] As such, 3D bounding boxes 144A, 144D, 144F, and 144H forming training set 145 would be utilized to train monocular 3D object detection model 170. In this example, training of monocular 3D object detection model 170 may be supervised, where a loss from a loss function is calculated and one or more model weights 172 of monocular 3D object detection model 170 would be adjusted based on the loss. By filtering master set 144 to create training set 145, monocular 3D object detection model 170 will receive a higher quality training set, which may positively impact the performance of monocular 3D object detection model 170.

[0033] To better understand this process, refer to Figure 6, which illustrates a process flow. Here, training set creation module 122 causes processor 110 to create a set 244 of 2D bounding boxes 244A-244H for objects in image 142 based on the 3D bounding boxes 144A-144H of master set 144. Further, training set creation module 122 may cause processor 110 to associate 2D bounding boxes 244A-244H with corresponding 3D bounding boxes 144A-144H. In this example, 2D bounding box 244A is associated with 3D bounding box 144A, 2D bounding box 244B is associated with 3D bounding box 144B, etc.

[0034] The 2D bounding boxes 244A-244H may be created by projecting the object's 3D bounding box 144A-144H onto the image plane of the image 142. Using this projection, the training set creation module 122 causes the processor 110 to draw axis-aligned bounding boxes that cover the corners of the 3D bounding boxes 144A-144H to create the 2D bounding boxes 244A-244H. For example, referring to FIG. 7, a 3D bounding box 144A is shown that outlines the object 80 in 3D space with eight corners 180A-180H. In this example, the training set creation module 122 causes the processor 110 to draw a 2D bounding box 244A defined by four corners 190A-190D that cover the eight corners 180A-180H of the 3D bounding box 144A.

[0035] Once the 2D bounding boxes 244A-244H are created, the monocular 2D object detection model 160 is trained using the image 142 as input and the 2D bounding boxes 244A-244H as ground truth. Furthermore, the monocular 2D object detection model 160 will output a set of predicted 2D bounding boxes 344 using the image 142 as input and a confidence score associated with each of the predicted 2D bounding boxes. In this example, the monocular 2D object detection model 160 output predicted 2D bounding boxes 344A-344D and 344F-344H, each with a confidence score. Furthermore, the training set creation module 122 causes the processor 110 to associate the predicted 2D bounding boxes 344A-344D and 344F-344H with the corresponding 3D bounding boxes 144A-144H from the master set 144. This may be accomplished by utilizing previously determined association information regarding the association between the 2D bounding boxes 244A-244H and the corresponding 3D bounding boxes 144A-144H.

[0036] In particular, monocular 2D object detection model 160 did not output a bounding box for at least one object in image 142. In some cases, this may be because the undetected object is not clearly visible, similar to that described in FIG. 4, or has other issues that prevent the object from being detected by monocular 2D object detection model 160.

[0037] Additionally, the monocular 2D object detection model 160 may also receive a threshold 202. The threshold 202 may provide a parameter value that indicates a minimum confidence score for 2D bounding boxes that the monocular 2D object detection model 160 will produce. The lower the threshold 202, the more bounding boxes will be returned by the monocular 2D object detection model 160. Thus, the threshold 202 may be set to return only seven predicted 2D bounding boxes 344A-344D and 344F-344H instead of eight.

[0038] Using the predicted 2D bounding boxes 344A-344D and 344F-344H of set 344 and the 2D bounding boxes 244A-244H as ground truth, training set creation module 122 may cause a processor to utilize a loss function 206 to determine a loss. The loss may be utilized to adjust one or more model weights 162 of the monocular 2D object detection model 160 to improve the performance of the monocular 2D object detection model 160.

[0039] A brief discussion of monocular 2D object detection model 160 follows. Monocular 2D object detection model 160 may be any type of monocular 2D object detection model that receives an image as input and outputs one or more 2D bounding boxes that identify objects in the image. In one example, monocular 2D object detection model 160 may be a fully convolutional single-stage object detector. However, any type of monocular 2D object detection model may be utilized.

[0040] Once the predicted 2D bounding boxes 344A-344D and 344F-344H are created, the training set creation module 122 may cause the processor 110 to select a subset from the predicted 2D bounding boxes 344A-344D and 344F-344H based on the confidence scores. Further, with reference to FIG. 8 , the processor 110 selects the predicted 2D bounding boxes 344A-344D, 344F, and 344H to form the subset 345. In this example, the confidence scores associated with each of the predicted 2D bounding boxes 344A-344D, 344F, and 344H may indicate a high likelihood that an object is located within the predicted 2D bounding boxes 344A-344D, 344F, and 344H.

[0041] Selection of subset 345 may occur by having processor 110 rank-order set 344 of predicted 2D bounding boxes 344A-344D and 344F-344H based on confidence scores and selecting subset 345 from set 344. Subset 345 may include predicted 2D bounding boxes 344A-344D and 344F-344H of set 344 that have confidence scores that meet a predetermined threshold.

[0042] Using the subset 345, the training set creation module 122 may cause the processor 110 to retrain the monocular 2D object detection model 160 using the image 142 as input and the subset 345, including the predicted 2D bounding boxes 344A-344D, 344F, and 344H, as ground truth. It should be understood that the terms “retraining” or “retraining” can be interpreted the same as “training” or “training,” respectively. Here, the 2D monocular 2D object detection model 160 outputs a set 444 of predicted 2D bounding boxes 444A, 444B, 444D, 444F, and 444H and associated confidence scores. The training set creation module 122 may cause the processor 110 to calculate a loss using the loss function 206. Based on this loss, the processor 110 may adjust one or more model weights 162 of the 2D monocular 2D object detection model 160. Optionally, the monocular 2D object detection model 160 may receive the threshold 202 described above.

[0043] Again, training set creation module 122 causes processor 110 to associate predicted 2D bounding boxes 444A, 444B, 444D, 444F, and 444H with corresponding 3D bounding boxes 144A-144H from master set 144. Thus, information is saved that predicted 2D bounding boxes 444A, 444B, 444D, 444F, and 444H correspond to 3D bounding boxes 144A, 144B, 144D, 144F, and 144H. As before, this can be accomplished by utilizing previously determined association information.

[0044] The training set creation module 122 may cause the processor 110 to perform the retraining illustrated in Figure 8 as many times as desired. For example, the training set creation module 122 may cause the processor 110 to select another subset from the set 444 and perform additional retraining of the monocular 2D object detection model 160 in an iterative manner. Although only one retraining is shown in this example, it should be understood that the retraining illustrated in Figure 8 may occur any number of times.

[0045] 9 , training set creation module 122 may cause processor 110 to select another subset 445 of predicted 2D bounding boxes 444A, 444B, 444D, 444F, and 444H based on the confidence scores for predicted 2D bounding boxes 444A, 444B, 444D, 444F, and 444H. In this example, the confidence scores associated with each of predicted 2D bounding boxes 444A, 444D, 444F, and 444H may indicate a higher likelihood that the object is located within predicted 2D bounding boxes 444A, 444D, 444F, and 444H. Similar to the selection of subset 345, the selection of subset 445 may occur by having processor 110 rank-order set 444 of predicted 2D bounding boxes 444A, 444B, 444D, 444F, and 444H based on the confidence scores and selecting subset 445 from set 444.

[0046] Thus, training set creation module 122 may cause processor 110 to identify predicted 2D bounding boxes 444A, 444D, 444F, and 444H as having a high likelihood that an object in image 142 is located within this subset of 2D bounding boxes 445. Training set 145 may then be created by finding which of 3D bounding boxes 144A-144H are associated with predicted 2D bounding boxes 444A, 444D, 444F, and 444H. Here, because 3D bounding boxes 144A, 144D, 144F, and 144H are associated with predicted 2D bounding boxes 444A, 444D, 444F, and 444H, training set creation module 122 may cause processor 110 to select 3D bounding boxes 144A, 144D, 144F, and 144H to form training set 145. This identification of the corresponding 3D bounding box may occur by utilizing the related information described above.

[0047] The 3D bounding boxes 144A, 144D, 144F, and 144H may then be stored in filtered training data 150 and used to train a monocular 3D object detection model 170. By using and training a monocular 2D object detection model 160 to correctly determine the location of actual objects in the images 142 using ground truth based on the 3D bounding boxes 144A-144H, the training set creation system 100 can filter out 3D bounding boxes that are not correctly aligned with the objects in the images 142 and / or that are prone to synchronization errors such as those shown and described in Figures 3 and 4.

[0048] 10 illustrates training of a monocular 3D object detection model 170 using the training set 145. It should be understood that the training of the monocular 3D object detection model 170 may be performed by the training set creation system 100, or by another system that simply utilizes the filtered training set 145 to train the monocular 3D object detection model 170.

[0049] Here, monocular 3D object detection model 170 receives image 142 and outputs predicted 3D bounding boxes 544A, 544F, and 544H, which form set 544. Processor 110 uses loss function 212 to determine the loss between predicted 3D bounding boxes 544A, 544F, and 544H and 3D bounding boxes 144A, 144D, 144F, and 144H, which serve as ground truth. Processor 110 may then use the loss to adjust model weights 172 of the monocular 3D object detection model to improve the performance of monocular 3D object detection model 170.

[0050] Therefore, by training the monocular 3D object detection model 170 using the training set creation system 100 and the filtered training data 150 from the master training data 140, the training data 150 will be populated with higher quality ground truth 3D bounding boxes that will mitigate the disparity-related issues in our synchronization error. Ultimately, because the monocular 3D object detection model 170 will be trained with better training data, the monocular 3D object detection model will achieve improved performance.

[0051] Referring to Figure 11, a method 600 for creating a training set is shown. Method 600 will be described in terms of training set creation system 100 of Figure 5 with reference to the process flows shown in Figures 6, 8, and 9. However, it should be understood that this is just one example of implementing method 600. While method 600 is described in conjunction with training set creation system 100, it should be understood that method 600 is not limited to being performed within training set creation system 100, but instead is an example of a system that may perform method 600.

[0052] In step 602, training set creation module 122 causes processor 110 to create a set 244 of 2D bounding boxes 244A-244H of objects in image 142 based on the 3D bounding boxes 144A-144H of master set 144. As previously described, the 2D bounding boxes 244A-244H may be created by projecting the 3D bounding boxes 144A-144H of the objects onto the image plane of image 142. Using this projection, training set creation module 122 causes processor 110 to draw axis-aligned bounding boxes that cover the corners of the 3D bounding boxes 144A-144H to create the 2D bounding boxes 244A-244H.

[0053] In step 604, training set creation module 122 causes processor 110 to train monocular 2D object detection model 160 using image 142 as input and 2D bounding boxes 244A-244H as ground truth. Furthermore, monocular 2D object detection model 160 will output a set of predicted 2D bounding boxes 344 using image 142 as input and a confidence score associated with each of the predicted 2D bounding boxes. In this example, monocular 2D object detection model 160 output predicted 2D bounding boxes 344A-344D and 344F-344H, each with a confidence score.

[0054] In step 606, training set creation module 122 may cause processor 110 to select a subset from predicted 2D bounding boxes 344A-344D and 344F-344H based on the confidence scores. Further referring to FIG. 8, processor 110 selects predicted 2D bounding boxes 344A-344D, 344F, and 344H to form subset 345. In this example, the confidence scores associated with each of predicted 2D bounding boxes 344A-344D, 344F, and 344H may indicate a high likelihood that an object is located within predicted 2D bounding boxes 344A-344D, 344F, and 344H.

[0055] In step 608, training set creation module 122 may cause processor 110 to retrain monocular 2D object detection model 160 using image 142 as input and subset 345 including predicted 2D bounding boxes 344A-344D, 344F, and 344H as ground truth. 2D monocular 2D object detection model 160 now outputs set 444 of predicted 2D bounding boxes 444A, 444B, 444D, 444F, and 444H and associated confidence scores. Training set creation module 122 may cause processor 110 to perform the retraining illustrated in FIG. 8 as many times as desired.

[0056] In step 610, the training set creation module 122 may cause the processor 110 to select another subset 445 of the predicted 2D bounding boxes 444A, 444B, 444D, 444F, and 444H based on the confidence scores for the predicted 2D bounding boxes 444A, 444B, 444D, 444F, and 444H.

[0057] In step 612, training set creation module 122 may cause processor 110 to select 3D bounding boxes 144A, 144D, 144F, and 144H as they relate to predicted 2D bounding boxes 444A, 444D, 444F, and 444H to form training set 145. This identification of corresponding 3D bounding boxes may occur by utilizing the association information described above. 3D bounding boxes 144A, 144D, 144F, and 144H may then be saved in filtered training data 150 and used to train monocular 3D object detection model 170.

[0058] Therefore, using method 600, training data 150 will be populated with higher quality ground truth 3D bounding boxes that will mitigate disparity-related issues in our synchronization errors, leading to improved training of monocular 3D object detection models.

[0059] Detailed embodiments are disclosed herein. However, it should be understood that the disclosed embodiments are intended only as examples. Therefore, the specific structural and functional details disclosed herein should not be construed as limiting, but rather as a basis for the claims and as a representative basis for teaching those skilled in the art to variously employ aspects of the present specification in virtually any appropriate detailed configuration. Furthermore, the terms and phrases used herein are not intended to be limiting, but rather to provide an understandable description of possible implementations. While various embodiments are shown in FIGS. 1-11, the embodiments are not limited to the structures or applications shown.

[0060] According to various embodiments, the flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks may occur in a different order than noted in the figures. For example, depending on the functionality involved, two blocks shown in succession may be executed substantially concurrently, or may often be executed in the reverse order.

[0061] The above-described systems, components, and / or processes can be implemented in hardware or a combination of hardware and software, either centralized within one processing system or distributed across several interconnected processing systems. Any type of processing system or other apparatus adapted to perform the methods described herein is suitable. A typical combination of hardware and software can be a processing system having computer-usable program code that, when loaded and executed, controls the processing system to perform the methods described herein. The systems, components, and / or processes can also be embedded in a computer-readable storage, such as a machine-readable computer program product or other data program storage device, tangibly embodying a program of instructions executable by the machine to perform the methods and processes described herein. These elements can also be embedded in an application product that includes all features enabling the execution of the methods described herein and that can perform these methods when loaded into a processing system.

[0062] Furthermore, the arrangements described herein may have the form of a computer program product embodied in one or more computer-readable medium(s) having computer-readable program code embodied therein, e.g., stored thereon. Any combination of one or more computer-readable medium(s) may be utilized. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The phrase "computer-readable storage medium" refers to a non-transitory recording medium. A computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the above. More specific examples (a non-exhaustive list) of computer-readable storage media would include the following: portable computer diskettes, hard disk drives (HDDs), solid-state drives (SSDs), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD), optical storage device, magnetic storage device, or any suitable combination of the above. In the context of this specification, a computer-readable storage medium may be any tangible medium that may contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0063] Generally, as used herein, a module includes a routine, program, object, component, data structure, etc. that performs a particular task or implements a particular data type. In a further aspect, memory generally stores the referenced module. The memory associated with a module may be a buffer or cache embedded in a processor, RAM, ROM, flash memory, or other suitable electronic storage medium. In still further aspects, modules contemplated by the present disclosure are implemented as an application-specific integrated circuit (ASIC), a hardware component of a system-on-chip (SoC), a programmable logic array (PLA), or other suitable hardware component embedded with a defined configuration set (e.g., instructions) to perform the disclosed functions.

[0064] Program code embodied in a computer-readable medium may be transmitted using any suitable medium, including, but not limited to, wireless, wired, fiber optic, cable, RF, etc., or any suitable combination of the above. Computer program code for carrying out operations for aspects of the present configurations may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, Smalltalk, C++, or the like, and conventional procedural programming languages such as the "C" programming language or similar programming languages. The program code may run as a standalone software package, entirely on the user's computer, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., via the Internet using an Internet Service Provider).

[0065] The term "a," as used herein, is defined as one or more, not one. The term "multiple," as used herein, is defined as two or more, not two. The term "another," as used herein, is defined as at least a second or more. The terms "including" and / or "having," as used herein, are defined as including (i.e., open language). The phrase "and at least one of," as used herein, means and encompasses any and all possible combinations of one or more of the associated listed items. By way of example, the phrase "at least one of A, B, and C" includes A only, B only, C only, or any combination thereof (e.g., AB, AC, BC, or ABC).

[0066] The aspects herein may be embodied in other forms without departing from the spirit or essential characteristics thereof, and reference should accordingly be made to the following claims, rather than the foregoing specification, as indicating the scope of the present invention. The invention disclosed in this specification includes the following aspects. [Aspect 1] a processor; a memory coupled to the processor, The memory includes a training set creation module that, when executed by the processor, causing the processor to create 2D bounding boxes for objects in images based on a master set of 3D bounding boxes for the objects; causing the processor to train a model using the images as input and the 2D bounding boxes as ground truth, the model outputting a first set of predicted 2D bounding boxes and a confidence score for the first set of predicted 2D bounding boxes; causing the processor to select a first subset from the first set of predicted 2D bounding boxes based on the confidence scores for the first set of predicted 2D bounding boxes; causing the processor to retrain the model using the images as the input and the first subset as ground truth, the model outputting a second set of predicted 2D bounding boxes and a confidence score for the second set of predicted 2D bounding boxes; causing the processor to select a second subset of predicted 2D bounding boxes from the second set based on the confidence scores for the second set of predicted 2D bounding boxes; instructions to cause the processor to create a training set by selecting the 3D bounding boxes from the master set of 3D bounding boxes with corresponding 2D bounding boxes forming the second subset; system. [Aspect 2] The system of aspect 1, wherein the training set creation module further includes instructions that, when executed by the processor, cause the processor to project the 3D bounding box of the object onto an image plane of the image to create the 2D bounding box of the object. Aspect 3 The system of aspect 1, wherein the training set creation module further includes instructions that, when executed by the processor, cause the processor to rank the first set of predicted 2D bounding boxes based on the confidence scores and select the first subset from the first set of predicted 2D bounding boxes, the first subset including the predicted 2D bounding boxes of the first set having a confidence score that meets a predetermined threshold. Aspect 4 The system of aspect 1, wherein the training set creation module further includes instructions that, when executed by the processor, cause the processor to rank the second set of predicted 2D bounding boxes based on the confidence scores and to select the second subset from the second set of predicted 2D bounding boxes, the second subset including the predicted 2D bounding boxes of the second set having a confidence score that meets a predetermined threshold. Aspect 5 2. The system of claim 1, wherein the training set creation module further includes instructions that, when executed by the processor, cause the processor to provide a threshold to the model, and the model outputs a predicted 2D bounding box that satisfies the threshold. Aspect 6 2. The system of claim 1, wherein the 3D bounding box is created based on information captured from a LIDAR sensor of a scene containing the object. Aspect 7 2. The system of claim 1, wherein the confidence score indicates the likelihood that an object is defined by the predicted 2D bounding box. Aspect 8 The system of aspect 1, wherein the training set creation module further includes instructions that, when executed by the processor, cause the processor to draw an axis-aligned bounding box that covers the corners of the 3D bounding box to create the 2D bounding box. Aspect 9 creating 2D bounding boxes for objects in images based on a master set of 3D bounding boxes for the objects; training a model using the images as input and the 2D bounding boxes as ground truth, the model outputting a first set of predicted 2D bounding boxes and a confidence score for the first set of predicted 2D bounding boxes; selecting a first subset from the first set of predicted 2D bounding boxes based on the confidence scores for the first set of predicted 2D bounding boxes; retraining the model using the images as the input and the first subset as ground truth, wherein the model outputs a second set of predicted 2D bounding boxes and a confidence score for the second set of predicted 2D bounding boxes; selecting a second subset of predicted 2D bounding boxes from the second set of predicted 2D bounding boxes based on the confidence scores for the second set of predicted 2D bounding boxes; and creating a training set by selecting the 3D bounding boxes from the master set of 3D bounding boxes with corresponding 2D bounding boxes forming the second subset; A method comprising: Aspect 10 10. The method of aspect 9, further comprising projecting the 3D bounding box of the object onto an image plane of the image to create the 2D bounding box of the object. Aspect 11 10. The method of claim 9, further comprising: ranking the first set of predicted 2D bounding boxes based on the confidence scores; and selecting the first subset from the first set of predicted 2D bounding boxes, wherein the first subset includes the predicted 2D bounding boxes of the first set that have confidence scores that meet a predetermined threshold. Aspect 12 10. The method of claim 9, further comprising: ranking the second set of predicted 2D bounding boxes based on the confidence scores; and selecting the second subset from the second set of predicted 2D bounding boxes, wherein the second subset includes the predicted 2D bounding boxes of the second set that have confidence scores that meet a predetermined threshold. Aspect 13 10. The method of embodiment 9, further comprising providing a threshold to the model, wherein the model outputs a predicted 2D bounding box that satisfies the threshold. Aspect 14 10. The method of claim 9, wherein the 3D bounding box is created based on information captured from a LIDAR sensor of a scene containing the object. Aspect 15 10. The method of claim 9, wherein the confidence score indicates the likelihood that an object is defined by the predicted 2D bounding box. Aspect 16 10. The method of embodiment 9, further comprising drawing an axis-aligned bounding box that covers corners of the 3D bounding box to create the 2D bounding box. Aspect 17 training a model using images as input and 2D bounding boxes as ground truth based on a master set of 3D bounding boxes, the model outputting a set of predicted 2D bounding boxes and a confidence score for the set of predicted 2D bounding boxes; selecting a subset from the set of predicted 2D bounding boxes based on the confidence scores for the set of predicted 2D bounding boxes; and creating a training set by selecting the 3D bounding boxes from the master set of 3D bounding boxes with corresponding 2D bounding boxes forming the subset; A method comprising: Aspect 18 20. The method of claim 17, further comprising providing a threshold to the model, wherein the model outputs a predicted 2D bounding box that satisfies the threshold. Aspect 19 20. The method of claim 17, wherein the confidence score indicates the likelihood that an object is defined by the predicted 2D bounding box. Aspect 20 18. The method of embodiment 17, further comprising drawing an axis-aligned bounding box that covers corners of the 3D bounding box of the master set to create the 2D bounding box.

Claims

1. a processor; a memory coupled to the processor, The memory includes a training set creation module that, when executed by the processor, causing the processor to create 2D bounding boxes for objects in images based on a master set of 3D bounding boxes for the objects; causing the processor to train a model using the images as input and the 2D bounding boxes as ground truth, the model outputting a first set of predicted 2D bounding boxes and a confidence score for the first set of predicted 2D bounding boxes; causing the processor to select a first subset from the first set of predicted 2D bounding boxes based on the confidence scores for the first set of predicted 2D bounding boxes; causing the processor to retrain the model using the images as the input and the first subset as ground truth, the model outputting a second set of predicted 2D bounding boxes and a confidence score for the second set of predicted 2D bounding boxes; causing the processor to select a second subset of predicted 2D bounding boxes from the second set based on the confidence scores for the second set of predicted 2D bounding boxes; instructions to cause the processor to create a training set by selecting, from the master set, the 3D bounding boxes that correspond to the 2D bounding boxes forming the second subset; system.

2. 2. The system of claim 1 , wherein the training set creation module further comprises instructions that, when executed by the processor, cause the processor to project the 3D bounding box of the object onto an image plane of the image to create the 2D bounding box of the object.

3. 2. The system of claim 1 , wherein the training set creation module further comprises instructions that, when executed by the processor, cause the processor to rank the first set of predicted 2D bounding boxes based on the confidence scores and to select the first subset from the first set of predicted 2D bounding boxes, the first subset comprising the predicted 2D bounding boxes of the first set having confidence scores that meet a predetermined threshold.

4. 2. The system of claim 1 , wherein the training set creation module further comprises instructions that, when executed by the processor, cause the processor to rank the second set of predicted 2D bounding boxes based on the confidence scores and to select the second subset from the second set of predicted 2D bounding boxes, the second subset comprising the predicted 2D bounding boxes of the second set having confidence scores that meet a predetermined threshold.

5. 2. The system of claim 1 , wherein the training set creation module further comprises instructions that, when executed by the processor, cause the processor to provide a threshold to the model, and wherein the model outputs predicted 2D bounding boxes that satisfy the threshold.

6. The system of claim 1 , wherein the 3D bounding box is created based on information captured from a LIDAR sensor of a scene containing the object.

7. The system of claim 1 , wherein the confidence score indicates a likelihood that an object is defined by the predicted 2D bounding box.

8. 2. The system of claim 1, wherein the training set creation module further comprises instructions that, when executed by the processor, cause the processor to draw an axis-aligned bounding box that covers corners of the 3D bounding box to create the 2D bounding box.

9. creating 2D bounding boxes for objects in images based on a master set of 3D bounding boxes for said objects; training a model using the images as input and the 2D bounding boxes as ground truth, the model outputting a first set of predicted 2D bounding boxes and a confidence score for the first set of predicted 2D bounding boxes; selecting a first subset from the first set of predicted 2D bounding boxes based on the confidence scores for the first set of predicted 2D bounding boxes; retraining the model using the images as the input and the first subset as ground truth, wherein the model outputs a second set of predicted 2D bounding boxes and a confidence score for the second set of predicted 2D bounding boxes; selecting a second subset of predicted 2D bounding boxes from the second set of predicted 2D bounding boxes based on the confidence scores for the second set of predicted 2D bounding boxes; and creating a training set by selecting from the master set the 3D bounding boxes that correspond to the 2D bounding boxes forming the second subset; A method comprising:

10. The method of claim 9 , further comprising projecting the 3D bounding box of the object onto an image plane of the image to create the 2D bounding box of the object.

11. 10. The method of claim 9, further comprising: ranking the first set of predicted 2D bounding boxes based on the confidence scores; and selecting the first subset from the first set of predicted 2D bounding boxes, the first subset comprising the predicted 2D bounding boxes of the first set having confidence scores that meet a predetermined threshold.

12. 10. The method of claim 9, further comprising: ranking the second set of predicted 2D bounding boxes based on the confidence scores; and selecting the second subset from the second set of predicted 2D bounding boxes, the second subset comprising the predicted 2D bounding boxes of the second set having confidence scores that meet a predetermined threshold.

13. The method of claim 9 , further comprising providing a threshold to the model, wherein the model outputs a predicted 2D bounding box that satisfies the threshold.

14. The method of claim 9 , wherein the 3D bounding box is created based on information captured from a LIDAR sensor of a scene containing the object.

15. The method of claim 9 , wherein the confidence score indicates the likelihood that an object is defined by the predicted 2D bounding box.

16. The method of claim 9 , further comprising drawing an axis-aligned bounding box that covers the corners of the 3D bounding box to create the 2D bounding box.

17. training a model using images as input and 2D bounding boxes based on a master set of 3D bounding boxes of objects in the images as ground truth, the 2D bounding boxes being based on a master set of 3D bounding boxes of objects in the images, the model outputting a set of predicted 2D bounding boxes and a confidence score for the set of predicted 2D bounding boxes; selecting a subset from the set of predicted 2D bounding boxes based on the confidence scores for the set of predicted 2D bounding boxes; and creating a training set by selecting, from the master set, the 3D bounding boxes that correspond to the 2D bounding boxes forming the subset; A method comprising:

18. 20. The method of claim 17, further comprising providing a threshold to the model, wherein the model outputs a predicted 2D bounding box that satisfies the threshold.

19. The method of claim 17 , wherein the confidence score indicates the likelihood that an object is defined by the predicted 2D bounding box.

20. The method of claim 17 , further comprising drawing an axis-aligned bounding box that covers the corners of the 3D bounding box of the master set to create the 2D bounding box.

Images