A method for generating training data for training a monocular 3D object detection model, a 3D auto-labeling device, and a training method and training device using the same.

Abstract

This generates training data for a model that generates 3D object data from diverse 2D image data. [Solution] The method using a 3D auto-labeling device includes the steps of: performing 2D object detection on the original image to obtain 2D bounding box information and class information; obtaining object orientation information through an orientation estimation model; combining the 2D bounding box information with pre-set shape information for each class to obtain 3D object size information corresponding to the class information; generating a 3D bounding box in a pre-set virtual 3D space by referring to the depth information, orientation information, and 3D object size information of the 2D bounding box center point obtained by the depth estimation model; and generating a 3D pseudo-label including the 3D bounding box, annotating the original image with the 3D pseudo-label to generate training data.

Description

Technical Field

[0001] The present invention relates to a method for generating training data for learning a monocular 3D object detection model, a 3D auto-labeling device, a method for learning a monocular 3D object detection model using the training data thus generated, and a learning device.

Background Art

[0002] Among object detection techniques, monocular 3D object detection is a technique that attempts to estimate the information of the 3D bounding box of an object using only a single RGB video / image without using an additional sensor.

[0003] However, most of the currently published 3D object detection datasets are biased towards road driving scenes targeting autonomous driving scenarios. In indoor datasets, those aimed at furniture arrangement and indoor path finding are mainly used, and data on moving objects and people is extremely lacking at present.

[0004] For these reasons, there is a problem that it is not easy to apply to domains with more complex and commercial needs, such as industrial sites and robot operations.

[0005] Furthermore, conventional methods for obtaining 3D object detection data include utilizing simulations in engines that simulate virtual environments such as CARLA, Unreal / Unity, and Isaac Sim, which extract camera parameters (intrinsic / extrinsic), depth maps, instance masks, and 3D bounding box information of objects from accurate physical and geometric information within the engine to construct 3D object detection data; and generating a 3D bounding box from point clouds and camera data obtained by synchronizing and calibrating high-precision LiDAR sensors and cameras, and using 3D sensor data to construct 3D object detection data. However, both methods have the problem of not being able to adequately reflect the scale (size), orientation (pose), joint structure (articulation), and deformability of diverse dynamic objects that appear in the real world. For example, objects such as shopping carts, multi-jointed industrial robots, bicycles loaded with goods, and pets rarely appear in datasets, or if they do appear, their number and diversity of shapes are extremely limited.

[0006] Furthermore, when using simulations, the creation of diverse object shapes requires the configuration of relatively expensive assets, and realistically reproducing spaces such as actual shopping centers and warehouses poses a problem in terms of significant cost and time.

[0007] Furthermore, methods that combine data from 3D sensors such as LiDAR are problematic because they involve high costs, high power consumption, and complex equipment, making them difficult to practically implement in small consumer cameras or industrial edge devices. While depth estimation using stereo cameras can be configured at a relatively low cost, it still requires precise calibration and involves complex hardware configurations, making its application difficult.

[0008] Therefore, there is a need for improvement measures to solve the above problems. [Overview of the project] [Problems that the invention aims to solve]

[0009] The purpose of this invention is to solve all of the problems mentioned above.

[0010] Furthermore, the present invention aims to generate training data on environments and objects that could not be adequately handled with existing training data, by generating training data that is automatically pseudo-3D labeled from 2D images using a 2D object detection model, a direction estimation model, and a depth estimation model, thereby overcoming the limitations of existing datasets, namely dataset bias and dependence on other sensors.

[0011] Furthermore, another objective of the present invention is to reduce the variation in quality among the generated training data by setting up a normalized virtual space called a pre-configured virtual 3D space and using it to generate pseudo-3D labels.

[0012] Furthermore, another objective of the present invention is to enable stable learning of a monocular 3D object detection model by effectively correcting errors that may occur in the process of generating pseudo-3D labels through a 3D auto-labeling device, by using a 2D-3D consistency loss that ensures 3D information maintains consistency with 2D information during training of the monocular 3D object detection model. [Means for solving the problem]

[0013] According to one embodiment of the present invention, in a method for generating training data for training a monocular 3D object detection model, (a) when a source image is acquired, a 3D auto-labeling device (i) performs 2D object detection on the source image through a 2D object detection model to acquire 2D bounding box information (the 2D bounding box information includes the center point coordinates of the 2D bounding box and the width and height information of the 2D bounding box) and class information corresponding to at least one object on the source image, (ii) acquires the orientation information of the object with respect to the center point of the 2D bounding box through a direction estimation model, and (iii) obtains a preset shape (shape) for each class. A method is provided that includes: (e) obtaining object shape information corresponding to the class information by referring to the information, and obtaining 3D object size information corresponding to the class information by combining the 2D bounding box information and the object shape information; (b) the 3D auto-labeling device generating a 3D bounding box in a preset virtual 3D space by referring to the depth information of the center point, the direction information and the 3D object size information obtained by referring to a depth map for the original image generated through a depth estimation model; and (c) the 3D auto-labeling device generating a 3D pseudo-label including the 3D bounding box, and annotating the original image with the 3D pseudo-label to generate training data.

[0014] In one example, in step (b) above, the 3D auto-labeling device refers to the depth information, inversely projects the center point onto the preset virtual 3D space according to the scale factor, obtains the center 3D coordinates corresponding to the center point, and generates the 3D bounding box by referring to 3D object size adjustment information obtained by adjusting the 3D object size information according to the scale factor and the direction information, with the center 3D coordinates as the center.

[0015] In one example, the scale factor is the coordinate system conversion factor between the original image and the preset virtual 3D space, and the scale factor is s, which is the scale factor of the x coordinate orig , y ,

[0017] , orig , , vy , , x , , vy , , vx , v , v , y , , , x ,

[0016] , vx , , , and s, which is the scale factor of the y coordinate y including the s x and the s y are determined as follows JPEG0007862902000002.jpg1123 and JPEG0007862902000003.jpg1323 and the W v is the width of the preset virtual 3D space, the W orig is the width of the original image, the H v is the height of the preset virtual 3D space, the H orig is the height of the original image

[0016] In one example, the preset virtual 3D space is a perspective space with a vanishing point, the preset virtual 3D space is set as the virtual coordinate system of a virtual camera, and the virtual internal matrix Kv of the virtual camera is defined as follows JPEG0007862902000004.jpg2045 where fv is the virtual focal length, the c vx is the x coordinate of the virtual principal point, the c vy is the y coordinate of the virtual principal point, the c vx is the s x multiplied by c, which is the x coordinate of the principal point of the original image x and the c vy is the s y multiplied by c, which is the y coordinate of the principal point of the original image y and the product of

[0017] In one example, when the 3D auto-labeling device back-projects the center point into the preset virtual 3D space according to the scale factor, the central 3D coordinates (u corresponding to the center point are based on the following formulav , v v , Z v ) obtain, and at that time, JPEG0007862902000005.jpg838 JPEG0007862902000006.jpg1040 JPEG0007862902000007.jpg1845 The variables u and v are the coordinates of the center point, Zcam is the depth information of the center point, and fx is the focal length of the original image.

[0018] In one example, in step (c), the 3D auto-labeling device, in generating the 3D pseudo-label, includes in the 3D pseudo-label the coordinate information obtained by converting the 3D bounding box to coordinates on the original image, and the class information, by referring to the scale factor.

[0019] In one example, the 3D auto-labeling device converts the 3D bounding box to coordinates on the original image based on the following formula, and in doing so, JPEG0007862902000008.jpg1434 JPEG0007862902000009.jpg1575 JPEG0007862902000010.jpg1676 Said Z cam This is depth information on the original image, and X cam is the x-coordinate in the original image, and is the Y cam is the y coordinate in the original image, and u v is the x-coordinate of any one point in the 3D bounding box, and v v The y coordinate is that of any one of the points in the 3D bounding box, and the Z v is the z-coordinate of any one of the points in the 3D bounding box, and f x and f y is the focal length of the original image.

[0020] In one example, in step (b) above, when the 3D auto-labeling device generates the 3D bounding boxes in the preset virtual 3D space, if at least two of the generated 3D bounding boxes, namely the first 3D bounding box and the second 3D bounding box, overlap each other, (i) a sub-process to adjust the position of the first 3D bounding box by referring to a first point which is any one point within a predetermined area centered on the first center point of the first 2D bounding box corresponding to the first 3D bounding box, and (ii) a second 2D bounding box corresponding to the second 3D bounding box (iii) Execute one of the following subprocesses to adjust the position of the second 3D bounding box by referring to a second point which is any one point within a predetermined region centered on the second center point of the box, and (iii) an additional subprocess to adjust the positions of the first 3D bounding box and the second 3D bounding box by referring to a third point which is any one point within a predetermined region centered on the first center point of the first 2D bounding box corresponding to the first 3D bounding box, and a fourth point which is any one point within a predetermined region centered on the second center point of the second 2D bounding box corresponding to the second 3D bounding box.

[0021] Furthermore, according to another embodiment of the present invention, in a method for training a monocular 3D object detection model using training data generated through 3D autolabeling, (a) when a source image is acquired by a 3D autolabeling device, (i-1) 2D object detection is performed on the source image through a 2D object detection model to acquire 2D bounding box information (the 2D bounding box information includes the center point coordinates of the 2D bounding box and the width and height information of the 2D bounding box) and class information corresponding to at least one object on the source image, (i-2) direction information of the object is acquired with respect to the center point of the 2D bounding box through a direction estimation model, (i-3) object shape information corresponding to the class information is acquired by referring to pre-set shape information for each class, and the 2D bounding box information and the object shape information are combined to obtain 3D corresponding to the class information Steps to be performed: (ii) a process of obtaining object size information, (ii) a process of generating a 3D bounding box in a pre-configured virtual 3D space by referring to the depth information of the center point obtained by referring to a depth map for the original image generated through a depth estimation model, the direction information and the 3D object size information, and (iii) a process of generating a 3D pseudo-label including the 3D bounding box and annotating the original image with the 3D pseudo-label, wherein the learning device obtains at least one second learning data (the second learning data includes the first learning data and fine-labeled learning data), inputs the second learning data into a monocular 3D object detection model, and generates 2D information and 3D information respectively as a result of detecting at least one learning object on the learning image contained in the second learning data using the monocular 3D object detection model;A method is also provided which includes (b) causing the learning device to train the monocular 3D object detection model to minimize an integrated loss that includes at least some of the 2D-3D consistency loss, depth Kullback-Leibler divergence loss, region loss, and query diversity loss, by referring to the 2D information and the 3D information.

[0022] In one example, the 2D information includes the size information of the predicted 2D bounding box, the predicted class information of the training object, the projected center information, and the diversity information, while the 3D information includes the size information of the predicted 3D bounding box, the predicted depth information, the depth Kullback-Leibler divergence information, and the predicted orientation information of the training object.

[0023] In one example, in step (b), the 2D-3D consistency loss is generated by referring to the difference between the information obtained by converting the predicted 3D bounding box to coordinates on the training image and the coordinate information of the predicted 2D bounding box.

[0024] In one example, in step (b), the depth Kullback-Leibler divergence loss is generated using the depth Kullback-Leibler divergence information, in which case the depth Kullback-Leibler divergence information is generated by referring to the ground truth depth information contained in the ground truth label for the training image and the predicted depth information.

[0025] In one example, in step (b), the region loss is generated by generating a Dice loss and a BCE (Binary Cross-entropy) loss, respectively, using the predicted 3D bounding box and the ground truth labels for the training images, and assigning predetermined weights to the Dice loss and the BCE loss, respectively.

[0026] In one example, in step (b), the query diversity loss is generated using batch information from the monocular 3D object detection model and 3D query information output through the 3D decoder of the monocular 3D object detection model.

[0027] Furthermore, according to yet another embodiment of the present invention, a 3D autolabeling device for generating training data for training a monocular 3D object detection model includes at least one memory for storing instructions; and at least one processor configured to execute the instructions, wherein the processor (i) when a source image is acquired, (i) performs 2D object detection on the source image through a 2D object detection model to acquire 2D bounding box information (the 2D bounding box information includes the center point coordinates of the 2D bounding box and the width and height information of the 2D bounding box) and class information corresponding to at least one object on the source image; and (ii) through a direction estimation model, the 2D bounding box A 3D auto-labeling device is provided that performs the following processes: (iii) acquiring orientation information of the object with respect to a center point; (iii) acquiring object shape information corresponding to the class information by referring to pre-set shape information for each class; and acquiring 3D object size information corresponding to the class information by combining the 2D bounding box information and the object shape information; (II) generating a 3D bounding box in a pre-set virtual 3D space by referring to the depth information of the center point, the orientation information and the 3D object size information acquired by referring to a depth map for the original image generated through a depth estimation model; and (III) generating a 3D pseudo-label including the 3D bounding box, and annotating the original image with the 3D pseudo-label to generate training data.

[0028] In one example, in process (II) above, the processor refers to the depth information, backprojects the center point onto the pre-configured virtual 3D space according to the scale factor, obtains the center 3D coordinates corresponding to the center point, and generates the 3D bounding box by referring to 3D object size adjustment information obtained by adjusting the 3D object size information according to the scale factor and the direction information, with the center 3D coordinates as the center.

[0029] In one example, the scaling factor is the coordinate system transformation factor between the original image and the pre-defined virtual 3D space, and the scaling factor is the scale factor of the x-coordinate s x and the y-coordinate scale factor s y and the above s x and the s y The following was decided: JPEG0007862902000011.jpg1022 And, JPEG0007862902000012.jpg1221 And the W v The width of the pre-set virtual 3D space is W. orig H is the width of the original image, and v H is the height of the pre-set virtual 3D space, and orig This is the height of the original image.

[0030] In one example, the pre-configured virtual 3D space is a perspective space which is a 3D space having a vanishing point, and the pre-configured virtual 3D space is set as the virtual coordinate system of the virtual camera, and the virtual internal matrix K of the virtual camera v It is defined as follows: JPEG0007862902000013.jpg1842 The aforementioned fv is the virtual focal length, and the aforementioned c vx is the x-coordinate of the virtual principal point, and the above c vy is the y-coordinate of the virtual principal point, and the above c vx is the aforementioned s x and the x-coordinate c of the principal point of the original image.x It is the product of the above c vy is the aforementioned s y and the y-coordinate c of the principal point of the original image. y It is the product of the two.

[0031] In one example, the processor backprojects the center point onto the pre-set virtual 3D space according to the scale factor, and the center 3D coordinates corresponding to the center point are determined by the following formula (u v , v v , Z v ) obtain, and at that time, JPEG0007862902000014.jpg838 JPEG0007862902000015.jpg1040 JPEG0007862902000016.jpg1845 The aforementioned u and v are the coordinates of the center point, and the aforementioned Z cam is the depth information of the center point, and f x is the focal length of the original image.

[0032] In one example, in process (III), when the processor generates the 3D pseudo-label, it includes in the 3D pseudo-label the coordinate information obtained by converting the 3D bounding box to coordinates on the original image, and the class information, by referring to the scale factor.

[0033] In one example, the processor converts the 3D bounding box to coordinates on the original image based on the following formula, and in doing so, JPEG0007862902000017.jpg1434 JPEG0007862902000018.jpg1575 JPEG0007862902000019.jpg1676 The Zcam is depth information on the original image, and the X cam is the x-coordinate in the original image, and is the Y cam is the y coordinate in the original image, and u vis the x-coordinate of any one point in the 3D bounding box, and v v The y coordinate is that of any one of the points in the 3D bounding box, and the Z v is the z-coordinate of any one of the points in the 3D bounding box, and f x and f y is the focal length of the original image.

[0034] In one example, in process (II), when the processor generates the 3D bounding boxes in the pre-configured virtual 3D space, if at least two of the generated 3D bounding boxes, namely the first 3D bounding box and the second 3D bounding box, overlap each other, (i) a subprocess adjusts the position of the first 3D bounding box by referring to a first point which is any one point within a predetermined area centered on the first center point of the first 2D bounding box corresponding to the first 3D bounding box, and (ii) a second 2D bounding box corresponding to the second 3D bounding box (iii) One of the following subprocesses is performed to adjust the position of the second 3D bounding box by referring to a second point which is any one point within a predetermined region centered on the second center point of the first 2D bounding box corresponding to the first 3D bounding box, and a fourth point which is any one point within a predetermined region centered on the second center point of the second 2D bounding box corresponding to the second 3D bounding box.

[0035] Furthermore, according to yet another embodiment of the present invention, a learning device for learning a monocular 3D object detection model using learning data generated through 3D autolabeling includes at least one memory for storing instructions; and at least one processor configured to execute the instructions, wherein the processor (i) when a source image is acquired by the 3D autolabeling device, (i-1) performs 2D object detection on the source image through a 2D object detection model to acquire 2D bounding box information (the 2D bounding box information includes the center point coordinates of the 2D bounding box and the width and height information of the 2D bounding box) and class information corresponding to at least one object on the source image; (i-2) acquires direction information of the object with respect to the center point of the 2D bounding box through a direction estimation model; and (i-3) a pre-set shape for each class. (ii) a process of obtaining object shape information corresponding to the class information by referring to the information, and obtaining 3D object size information corresponding to the class information by combining the 2D bounding box information and the object shape information, (ii) a process of generating a 3D bounding box in a pre-set virtual 3D space by referring to the depth information of the center point obtained by referring to the depth map for the original image generated through the depth estimation model, the direction information and the 3D object size information, and (iii) a process of generating a 3D pseudo-label including the 3D bounding box and annotating the original image with the 3D pseudo-label, and at least one second training data (the second training data includes the first training data and fine-labeled training data) is obtained, the second training data is input into a monocular 3D object detection model, and the monocular 3D object detection model is used toA learning device is provided that performs the following processes: (II) generating 2D information and 3D information respectively as a result of detecting at least one learning object on a learning image contained in the second learning data; and (II) training the monocular 3D object detection model to minimize an integrated loss, including a 2D-3D consistency loss, a depth Kullback-Leibler divergence loss, a region loss, and a query diversity loss, by referring to the 2D information and the 3D information.

[0036] In one example, the 2D information includes the size information of the predicted 2D bounding box, the predicted class information of the training object, the projected center information, and the diversity information, while the 3D information includes the size information of the predicted 3D bounding box, the predicted depth information, the depth Kullback-Leibler divergence information, and the predicted orientation information of the training object.

[0037] In one example, in process (II) above, the 2D-3D consistency loss is generated by referring to the difference between the information obtained by converting the predicted 3D bounding box to coordinates on the training image and the coordinate information of the predicted 2D bounding box.

[0038] In one example, in process (II) above, the depth Kullback-Leibler divergence loss is generated using the depth Kullback-Leibler divergence information, in which case the depth Kullback-Leibler divergence information is generated by referring to the ground truth depth information contained in the ground truth label for the training image and the predicted depth information.

[0039] In one example, in process (II) above, the region loss is generated by using the predicted 3D bounding box and the ground truth labels for the training images to generate a Dice loss and a BCE (Binary Cross-entropy) loss, respectively, and assigning predetermined weights to each of the Dice loss and the BCE loss.

[0040] In one example, in process (II) above, the query diversity loss is generated using batch information from the monocular 3D object detection model and 3D query information output through the 3D decoder of the monocular 3D object detection model. [Effects of the Invention]

[0041] This invention generates training data that automatically assigns pseudo-3D labels to 2D images using a 2D object detection model, a direction estimation model, and a depth estimation model. This allows for the generation of training data about environments and objects that could not be adequately handled with existing training data, and overcomes the limitations of existing datasets, such as dataset bias and dependence on other sensors.

[0042] Furthermore, the present invention has the effect of reducing the variation in quality among the generated training data by setting a normalized virtual space called a pre-configured virtual 3D space and generating pseudo-3D labels using this space.

[0043] Furthermore, the present invention has the effect of enabling stable learning of a monocular 3D object detection model by effectively correcting errors that may occur in the process of generating pseudo-3D labels through a 3D auto-labeling device, by using a 2D-3D consistency loss that ensures 3D information maintains consistency with 2D information during training of the monocular 3D object detection model. [Brief explanation of the drawing]

[0044] The following drawings, attached for use in describing embodiments of the present invention, represent only a portion of the embodiments, and a person with ordinary skill in the art to which the present invention pertains (hereinafter referred to as "ordinary art") can obtain other drawings based on these drawings without performing any inventive work.

[0045] [Figure 1] This figure shows a schematic configuration of a 3D auto-labeling device that generates training data for training a monocular 3D object detection model according to one embodiment of the present invention. [Figure 2] This figure schematically shows a flowchart of a method for generating training data for training a monocular 3D object detection model according to one embodiment of the present invention. [Figures 3a-3d] This figure shows an example of a method for generating training data for training a monocular 3D object detection model according to one embodiment of the present invention. [Figure 4a-4b] This figure shows an example of a method for generating training data for training a monocular 3D object detection model according to one embodiment of the present invention. [Figure 5] This figure shows an example of a method for generating training data for training a monocular 3D object detection model according to one embodiment of the present invention. [Figure 6]This figure shows a schematic configuration of a learning device that learns a monocular 3D object detection model using training data generated through 3D autolabeling according to an embodiment of the present invention. [Modes for carrying out the invention]

[0046] The detailed description of the present invention, as described below, refers to the accompanying drawings illustrating specific embodiments in which the present invention may be carried out. These embodiments are described in sufficient detail to enable a person of the ordinary skill to carry out the present invention. It should be understood that the various embodiments of the present invention are different from one another but do not need to be mutually exclusive. For example, certain shapes, structures and characteristics described herein can be realized by modifying one embodiment to another without departing from the spirit and scope of the present invention. It should also be understood that the position or arrangement of individual components within each embodiment can be modified without departing from the spirit and scope of the present invention. Therefore, the detailed description described below should not be taken as restrictive, and the scope of the present invention should be understood to encompass the scope claimed in the claims and all equivalent scopes thereto. In the drawings, similar reference numerals indicate identical or similar components in various aspects.

[0047] In the following, several preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings, so that a person with ordinary skill in the art to which the present invention pertains can easily implement the present invention.

[0048] Figure 1 shows a schematic configuration of a 3D auto-labeling device 100 that generates training data for training a monocular 3D object detection model according to one embodiment of the present invention.

[0049] As shown in Figure 1, the 3D auto-labeling apparatus 100 of the present invention may include a memory 110 and a processor 120.

[0050] Here, memory 110 can store instructions to be executed by processor 120, specifically, instructions which are code generated for the purpose of enabling the 3D auto-labeling device 100 to function in a particular manner, and which can be stored in computer-accessible or computer-readable memory intended for computers and other programmable data processing equipment. The instructions can carry out processes for performing the functions described in the specification of the present invention.

[0051] The processor 120 may also include hardware components such as an MPU (Micro Processing Unit) or CPU (Central Processing Unit), cache memory, and a data bus. Furthermore, the 3D auto-labeling device 100 may also include software components such as an operating system and applications that perform a specific purpose.

[0052] However, this does not exclude the case in which the processor 120 of the 3D auto-labeling device 100 includes an integrated processor, which is a medium, processor, and memory integrated form for carrying out the present invention.

[0053] Furthermore, the 3D auto-labeling device 100 can be linked with the 2D object detection model 700, the direction estimation model 800, and the depth estimation model 900. In Figure 1, the 2D object detection model 700, the direction estimation model 800, and the depth estimation model 900 are shown as being configured separately from the 3D auto-labeling device 100, but this is not limited to this configuration. At least a portion of the 2D object detection model 700, the direction estimation model 800, and the depth estimation model 900 may be included in the computing device 100 and linked together.

[0054] Here, the 2D object detection model 700, the direction estimation model 800, and the depth estimation model 900 may already be trained.

[0055] Furthermore, the 2D object detection model 700, the direction estimation model 800, and the depth estimation model 900 may be any conventionally known model, but are not limited to them.

[0056] A method using the 3D auto-labeling device 100 according to one embodiment of the present invention, configured as described above, will be explained below with reference to Figure 2.

[0057] Figure 2 schematically shows a flowchart of a method for generating training data for training a monocular 3D object detection model according to one embodiment of the present invention.

[0058] Referring to Figure 2, the 3D auto-labeling device 100, upon acquiring a source image, can (i) perform 2D object detection on the source image through the 2D object detection model 700 to obtain 2D bounding box information and class information corresponding to at least one object on the source image, (ii) obtain object orientation information based on the center point of the 2D bounding box through the orientation estimation model 800, and (iii) obtain object shape information corresponding to the class information by referring to pre-set shape information for each class, and combine the 2D bounding box information and object shape information to obtain 3D object size information corresponding to the class information (S210).

[0059] Here, the 2D bounding box information can include the coordinates of the center point of the 2D bounding box and the width and height information of the 2D bounding box.

[0060] Furthermore, the direction estimation model 800 can generate object direction information based on a center point using a 2D bounding box as the ROI (Region of Interest), and this direction information can include at least some of the object's yaw, pitch, and roll.

[0061] Furthermore, the pre-configured shape information for each class may be information that pre-configures the average and / or typical shape information for each class, since objects of the same class generally have similar shape characteristics. For example, a "chair" may have shape characteristics consisting of a seat, backrest, and legs; an "automobile" may have shape characteristics consisting of a rectangular body and four wheels; and a "person" may have shape characteristics consisting of a head, torso, two arms, and two legs.

[0062] Thus, the 3D auto-labeling device 100 can use object shape information, which is the shape information corresponding to the class information of the object detected in the original image, from among the pre-set shape information for each class. In other words, the 3D auto-labeling device 100 does not use the shape of the object itself represented on the original image, but rather adjusts the object shape information obtained by referring to the pre-set shape information for each class so that it can fit to the 2D bounding box information, and obtains 3D object size information related to the object shape information. However, it is not limited to this, and it is also possible to obtain 3D object size information related to the object itself represented on the original image by referring to information that combines the object shape information and the 2D bounding box information. Here, the 3D object size information may include height (h) information, width (w) information, and length (l) information, and the length information may be thickness information.

[0063] Furthermore, the 3D auto-labeling device 100 can input the original image into the depth estimation model 900 and use the depth estimation model 900 to generate a depth map for the original image. Note that the process of generating a depth map through such a depth estimation model 900 may be performed separately and independently from the processes of acquiring 2D bounding box information, acquiring direction information, and acquiring 3D object size information described above.

[0064] The 3D auto-labeling device 100 can then generate a 3D bounding box in a pre-configured virtual 3D space by referring to the depth information, orientation information, and 3D object size information of the center point obtained by referring to the depth map for the original image generated through the depth estimation model (S220).

[0065] Here, the depth information may be the depth information of the center pixel corresponding to the center point on the depth map.

[0066] At this time, the 3D auto-labeling device 100 can refer to depth information, project the center point back onto a pre-set virtual 3D space according to the scale magnification, obtain the center 3D coordinates corresponding to the center point, and generate a 3D bounding box by referring to 3D object size adjustment information, which is obtained by adjusting the 3D object size information according to the scale magnification and direction information, with the center 3D coordinates as the center.

[0067] In other words, the 3D auto-labeling device 100 can obtain a central 3D coordinate by converting the x and y coordinates of the center point on the original image into x and y coordinates in a pre-set virtual 3D space according to the scale magnification, and using the depth information as the z coordinate in the pre-set virtual 3D space. Then, it can rotate the 3D object size information in three dimensions around the central 3D coordinate according to the direction information, enlarge or reduce the 3D object size information according to the scale magnification to obtain 3D object size adjustment information, and generate a 3D bounding box corresponding to the 3D object size adjustment information.

[0068] Here, the scale factor is the coordinate system transformation factor between the original image and the predefined virtual 3D space, and the scale factor is the scale factor of the x-coordinate, s x and the y-coordinate scale factor s y and include, s x and s y This can be determined as follows:

[0069] JPEG0007862902000020.jpg1658

[0070] Here, W v W is the width of the pre-defined virtual 3D space. orig This is the width of the original image, H v H is the height of the pre-set virtual 3D space. orig This can be the height of the original image.

[0071] Furthermore, the pre-configured virtual 3D space is a perspective space, which is a 3D space with vanishing points, and the pre-configured virtual 3D space is set as the virtual coordinate system of the virtual camera, and the virtual internal matrix K of the virtual camera v It can be defined as follows:

[0072] JPEG0007862902000021.jpg2045

[0073] Here, fv c is the virtual focal length, vx is the x-coordinate of the virtual principal point, and c vy is the y-coordinate of the virtual principal point, and c vx is s x and the x-coordinate c of the principal point of the original image x It is the product of and c vy is s y and the y-coordinate c of the principal point of the original image. y It can also be the product of the two.

[0074] As a result, when the 3D auto-labeling device 100 back-projects the center point onto a pre-set virtual 3D space according to the scale magnification, the center 3D coordinates corresponding to the center point are determined based on the following formula (u v , v v , Z v ) can be obtained.

[0075] JPEG0007862902000022.jpg838

[0076] JPEG0007862902000023.jpg1040

[0077] JPEG0007862902000024.jpg1845

[0078] Here, u is the x-coordinate of the center point, v is the y-coordinate of the center point, and Z cam This is the depth information of the center point, and f x This could also be the focal length of the original image.

[0079] Furthermore, when the 3D auto-labeling device 100 generates 3D bounding boxes in a pre-set virtual 3D space, if at least two of the generated 3D bounding boxes, namely the first 3D bounding box and the second 3D bounding box, overlap each other, (i) a sub-process adjusts the position of the first 3D bounding box by referring to a first point, which is any point within a predetermined area centered on the first center point of the first 2D bounding box corresponding to the first 3D bounding box, and (ii) the second 2D bounding box corresponding to the second 3D bounding box (iii) One of the following subprocesses can be executed to adjust the position of the second 3D bounding box by referring to a second point which is any one point within a predetermined region centered on the second center point, and (iii) one of the subprocesses to adjust the positions of the first 3D bounding box and the second 3D bounding box by referring to a third point which is any one point within a predetermined region centered on the first center point of the first 2D bounding box corresponding to the first 3D bounding box, and a fourth point which is any one point within a predetermined region centered on the second center point of the second 2D bounding box corresponding to the second 3D bounding box.

[0080] Next, the 3D auto-labeling device 100 can generate a 3D pseudo-label including the 3D bounding box, and annotate the original image with the 3D pseudo-label to generate training data (S230).

[0081] In this case, when the 3D auto-labeling device 100 generates a 3D pseudo-label, it can refer to the scale factor and include coordinate information obtained by converting the 3D bounding box to coordinates on the original image, as well as class information, in the 3D pseudo-label.

[0082] Here, the 3D auto-labeling device 100 can convert the 3D bounding box into coordinates on the original image based on the following formula.

[0083] JPEG0007862902000025.jpg1332

[0084] JPEG0007862902000026.jpg1366

[0085] JPEG0007862902000027.jpg1568

[0086] Here, Z cam This is depth information on the original image, X cam is the x-coordinate in the original image, and Y cam is the y-coordinate in the original image, and u v is the x-coordinate of any point in the 3D bounding box, and v v is the y-coordinate of any point in the 3D bounding box, and Z v is the z-coordinate of any one of the aforementioned points in the 3D bounding box, and f x and f y This could also be the focal length of the original image.

[0087] Furthermore, when the 3D auto-labeling device 100 converts the 3D bounding box to coordinates on the original image, it is possible to convert only the coordinates of the eight vertices of the 3D bounding box to coordinates on the original image, or to convert only some of the eight vertices of the 3D bounding box to coordinates on the original image, but it is not limited to these methods.

[0088] Furthermore, it goes without saying that after step S230, the process of 3D auto-labeling other original images that have not yet been assigned pseudo-3D labels can be repeated to generate a training dataset.

[0089] A method for generating training data for training the monocular 3D object detection model of the present invention, which consists of steps S210 to S230 as described above, will be described below with reference to Figures 3 to 5, as follows. Note that the following description assumes the generation of a 3D pseudo-label for any one object on the original image, and is not limited to the embodiment described below.

[0090] First, assuming that a source image 300 like the one in Figure 3a is acquired, the 3D auto-labeling device 100 detects an object 310, which is a "person," on the source image 300 through the 2D object detection model 700. As a result, it acquires the coordinates of the center point 320 of the 2D bounding box, as shown in Figure 3b, and 2D bounding box information 330, which includes the width and height information of the 2D bounding box, and can acquire "person" as the class information of the object 310. Then, through the direction estimation model 800, as shown in Figure 3c, the 3D auto-labeling device 100 can acquire direction information 340, which includes yaw, pitch, and roll information regarding the degree of bending between the upper and lower body of the object 310, the "person," and the degree to which the upper and lower body are tilted to the right or left. Furthermore, the 3D auto-labeling device 100 can obtain predetermined object shape information corresponding to the class information "person" by referring to pre-set shape information for each class, and combine this with 2D bounding box information 320 to obtain 3D object size information (not shown).

[0091] Furthermore, the 3D auto-labeling device 100 can obtain a depth map 350 for the original image 300, as shown in Figure 3d, through the depth estimation model 900, and can determine the depth information of the center point 320 by referring to the depth map 350.

[0092] Next, the 3D auto-labeling device 100 can obtain the central 3D coordinates 410 by inversely projecting the center point 320 onto a pre-configured virtual 3D space 400 as shown in Figure 4a, according to the scale factor. Then, as shown in Figure 4b, the 3D auto-labeling device 100 can generate a corresponding 3D bounding box 420 by rotating the 3D object size information (not shown) in three dimensions around the central 3D coordinates 410 according to the direction information, and referencing the 3D object size adjustment information (not shown) which has been enlarged or reduced according to the scale factor.

[0093] Next, as shown in Figure 5, the 3D auto-labeling device 100 can obtain coordinate information 360 by converting the 3D bounding box 420 generated in the virtual 3D space 400 to coordinates on the original image 300 by referring to the scale factor. Then, the 3D auto-labeling device 100 can generate training data by including the coordinate information 360 and the class information "person" in a 3D pseudo-label and annotating the object 310 on the original image 300.

[0094] Note that the boxes, points, arrows, etc. shown in Figures 3 to 5 are displayed to illustrate the above embodiment and are not actually displayed in the original image or a pre-configured virtual 3D space. The information related to them may be coordinate information in the original image or a pre-configured virtual 3D space.

[0095] On the other hand, the following section will describe in detail how to train a monocular 3D object detection model using training data generated through 3D autolabeling, which is a method consisting of steps S210 to S230 as described above.

[0096] Figure 6 shows a schematic configuration of a learning device 600 that learns a monocular 3D object detection model using training data generated through 3D autolabeling according to one embodiment of the present invention.

[0097] As shown in Figure 6, the learning device 600 of the present invention may include a memory 610 and a processor 620.

[0098] Here, memory 610 can store instructions to be executed by processor 620, specifically, instructions which are code generated for the purpose of enabling learning device 600 to function in a particular manner, and which can be stored in computer-accessible or computer-readable memory intended for computers and other programmable data processing equipment. The instructions can carry out processes for performing the functions described in the specification of the present invention.

[0099] Furthermore, the processor 620 may include hardware configurations such as an MPU (Micro Processing Unit) or CPU (Central Processing Unit), cache memory, and data bus. The learning device 600 may also further include software configurations for an operating system and applications that perform specific purposes.

[0100] However, this does not exclude the case in which the processor 620 of the learning device 600 includes an integrated processor, which is a medium, processor, and memory integrated form for carrying out the present invention.

[0101] Furthermore, the learning device 600 may include a monocular 3D object detection model 630. Although Figure 6 shows the monocular 3D object detection model 630 as being included in the learning device 100, it is not limited to this, and the monocular 3D object detection model 630 may be configured separately from the learning device 600 and still function in conjunction with it.

[0102] Here, the monocular 3D object detection model 630 may be any conventionally known model, such as MonoDGP (Monocular 3D Object Detection with Decoupled-Query and Geometry-Error Priors), or it may be a model with some structural modifications of a conventional model, or a model that combines some structural elements of conventional models, but is not limited to these.

[0103] Furthermore, the monocular 3D object detection model 630 may also be a model that generates both 2D information, which is the result of 2D object detection, and 3D information, which is the result of 3D object detection, as a result of detecting objects on the input image. The 2D information and 3D information will be described later.

[0104] A method for training a monocular 3D object detection model 630 using training data generated through 3D auto-labeling with a learning device 600 according to one embodiment of the present invention configured as described above is as follows.

[0105] First, the 3D auto-labeling device performs the following steps: (i) When the original image is acquired, (i-1) 2D object detection is performed on the original image through a 2D object detection model to acquire 2D bounding box information (including the center point coordinates of the 2D bounding box and the width and height information of the 2D bounding box) and class information corresponding to at least one object on the original image; (i-2) Object orientation information is acquired based on the center point of the 2D bounding box through a direction estimation model; (i-3) Object shape information corresponding to the class information is acquired by referring to pre-set shape information for each class; and the 2D bounding box information and object shape information are combined to acquire 3D object size information corresponding to the class information; (ii) Depth estimation (iii) a process of generating a 3D bounding box in a pre-configured virtual 3D space by referring to depth information, direction information, and 3D object size information of the center point obtained by referring to a depth map of the original image generated through a fixed model, and (iii) a process of generating a 3D pseudo-label including the 3D bounding box and annotating the original image with the 3D pseudo-label, thereby generating first training data with pseudo-labels, the training device 600 can acquire at least one second training data, input the second training data into the monocular 3D object detection model 630, and generate 2D information and 3D information respectively as a result of detecting at least one training object on the training image contained in the second training data using the monocular 3D object detection model 630.

[0106] Here, the second training data may include the first training data and finely labeled training data, the first training data may be weakly labeled data roughly pseudo-3D labeled by a 3D auto-labeling device, and the finely labeled training data may be precisely labeled training data.

[0107] In other words, the monocular 3D object detection model 630 may be input with at least one second training data, which may be either the first training data or the finely labeled training data.

[0108] Furthermore, when the second training data corresponds to the first training data, the ground truth label of the monocular 3D object detection model 630 may be the label of the first training data, i.e., a 3D pseudo label. When the second training data corresponds to finely labeled training data, the ground truth label of the monocular 3D object detection model 630 may be a fine label.

[0109] Next, the learning device 600 can refer to the 2D and 3D information to train the monocular 3D object detection model 630 to minimize an integrated loss that includes at least some of the following: 2D-3D consistency loss, depth Kullback-Leibler divergence loss, region loss, and query diversity loss.

[0110] Here, the 2D information may include, but is not limited to, the information of the predicted 2D bounding box including the size information of the predicted 2D bounding box, the predicted class information of the learning object, the projected center information, and the diversity information.

[0111] Also, the 3D information may include, but is not limited to, the information of the predicted 3D bounding box including the size information of the predicted 3D bounding box, the predicted depth information, the depth Kullback-Leibler divergence information, and the predicted orientation information of the learning object.

[0112] At this time, the 2D-3D consistency loss can be generated by referring to the difference between the information obtained by converting the predicted 3D bounding box to the coordinates on the learning image and the coordinate information of the predicted 2D bounding box.

[0113] For example, the 2D-3D consistency loss may be generated by the following function, but is not limited thereto.

[0114] JPEG0007862902000028.jpg987

[0115] JPEG0007862902000029.jpg1492

[0116] Here, the B 3D is the coordinate information of the predicted 3D bounding box, and the B 2D is the coordinate information of the predicted 2D bounding box, K is the internal matrix of the camera, and π(K, B 3D) is information obtained by transforming the predicted 3D bounding box into coordinates on the training image by referring to K, and β may be the tolerance threshold. Note that the SmoothL1 function is already known conventionally, so a detailed explanation of the SmoothL1 function will be omitted here.

[0117] Furthermore, the depth Kullback-Leibler divergence loss can be generated using depth Kullback-Leibler divergence information, which can be generated by referencing the ground truth depth information and predicted depth information contained in the ground truth labels for the training images.

[0118] For example, the depth Kullback-Leibler divergence loss may be generated by, but is not limited to, the following function:

[0119] JPEG0007862902000030.jpg1489

[0120] Here, p is the ground truth depth distribution, q is the predicted depth distribution, σ is the standard deviation of the predicted depth distribution, ε is the standard deviation of the ground truth depth distribution, μ is the depth mean of the predicted depth distribution, and m may be the depth mean of the ground truth depth distribution. Note that the Kullback-Leibler divergence is already known, so a detailed explanation of the Kullback-Leibler divergence will be omitted here.

[0121] Furthermore, the region loss can be generated by using the predicted 3D bounding box and ground truth labels for the training images to generate the Dice loss and BCE (Binary Cross-entropy) loss, respectively, and then assigning predetermined weights to the Dice loss and BCE loss.

[0122] For example, the region loss may be generated by, but is not limited to, the following function.

[0123] JPEG0007862902000031.jpg880

[0124] JPEG0007862902000032.jpg1441

[0125] JPEG0007862902000033.jpg997

[0126] Here, p is a multiscale predicted probability map, g is a ground truth mask, and λ D λ is the weight of the die loss, BCE This may also be the weight of the BCE loss. Since the dice loss and BCE loss are already known, a detailed explanation of the dice loss and BCE loss will be omitted here.

[0127] Furthermore, query diversity loss can be generated using batch information from the monocular 3D object detection model 630 and 3D query information output through the 3D decoder of the monocular 3D object detection model 630.

[0128] For example, query diversity loss may be generated by, but is not limited to, the following function.

[0129] JPEG0007862902000034.jpg1880

[0130] Here, B is the batch size of the monocular 3D object detection model 630, and Q is the number of 3D queries output from the 3D decoder in batch B. b,iis the i-th query vector in the b-th batch, and q b,j This could be the j-th query vector in the b-th batch. Since query diversity loss is already well-known, a detailed explanation of query diversity loss will be omitted here.

[0131] Furthermore, the learning device 600 can be configured so that, when the second learning data input to the monocular 3D object detection model 630 corresponds to the first learning data, only the 3D object detection portion of the monocular 3D object detection model 630 is learned.

[0132] Furthermore, the learning device 600 can be configured to learn both the 2D object detection portion and the 3D object detection portion of the monocular 3D object detection model 630 if the second training data input to the monocular 3D object detection model 630 corresponds to finely labeled training data.

[0133] Furthermore, the learning device 600 can ensure the overall stability of learning by filtering out or L2 regularization to exclude training data with excessive loss values ​​(e.g., noisy pseudo-3D labels) or by reducing their weight.

[0134] Thus, in this invention, since the internal camera parameters (focal length, resolution, etc.) differ for each camera that captured the original image, a method for generalizing and processing these parameters is to define a single unified virtual camera coordinate system, which is a pre-defined virtual 3D space, and normalize it, thereby reducing the variation in quality among the generated pseudo-3D labeled training data.

[0135] Furthermore, this invention reduces domain gaps caused by differences in resolution and focal length of various sensors through such normalization, enabling AI models to learn data effectively. It can also be used for data augmentation by utilizing the conversion process between the original 2D image and a pre-defined virtual 3D space. For example, by simulating diverse visual conditions from the same original data through changes in virtual focal length (Δf) or virtual camera angle (Δθ), the learning data can be augmented more efficiently.

[0136] Furthermore, the present invention is characterized by its ability to generate training data for various environments, such as indoor factories, crowded shopping malls, logistics warehouses, ports, and aerial photography, which existing datasets have not been able to adequately handle, in 2D RGB images.

[0137] Furthermore, the present invention is characterized by its ability to solve problems such as the high cost and time-consuming manual labeling process and the need to install and operate additional sensors (e.g., LiDAR) by automatically performing pseudo-3D labeling through a 3D auto-labeling device.

[0138] Furthermore, the present invention is characterized by enabling the learning of a monocular 3D object detection model 630 using a 2D-3D consistency loss that maintains consistency between 3D information and 2D information, thereby effectively correcting errors that may occur during the generation process of pseudo-3D labels through a 3D auto-labeling device, and enabling stable learning of the monocular 3D object detection model.

[0139] Furthermore, the present invention is characterized in that, instead of training the monocular 3D object detection model 630 with only weak-labeled data that has been pseudo-3D labeled by a 3D auto-labeling device, the model is also trained with precisely labeled fine-labeled training data, thereby enabling the monocular 3D object detection model 630 to detect objects more accurately.

[0140] Furthermore, experimental results applying the method of the present invention confirmed that a monocular 3D object detection model using the learning method proposed in this invention showed a significant performance improvement compared to a model trained using limited data constructed through LiDAR sensors in autonomous driving scenarios such as existing KITTI data, using training data generated through automated pseudo-3D labeling without the use of external sensors such as LiDAR or manual labeling. This indicates that the method of the present invention simultaneously overcomes two obstacles in conventional approaches: dataset bias and sensor dependency.

[0141] The embodiments of the present invention described above are implemented in the form of program instructions that can be executed through various computer components and may be recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., individually or in combination. The program instructions recorded on the computer-readable recording medium may be specially designed and configured for the present invention, or they may be known and available to those skilled in the art in the field of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program instructions, such as ROMs, RAMs, and flash memory. Examples of program instructions include not only machine code, such as that produced by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices may be configured to operate as one or more software modules to perform the processing according to the present invention, and vice versa.

[0142] Although the present invention has been described above with specific details such as concrete components, and with limited embodiments and drawings, these are provided only to aid in a more overall understanding of the invention, and the invention is not limited to the above embodiments. A person with ordinary skill in the art to which the invention pertains can make various modifications and variations from this description.

[0143] Therefore, the concept of the present invention should not be limited to the embodiments described above, and all modifications equivalent to or equivalent to the claims described below shall also fall within the scope of the concept of the present invention.

Claims

1. In a method for generating training data for training a monocular 3D object detection model, (a) When the source image is acquired, the 3D auto-labeling device (i) performs 2D object detection on the source image through a 2D object detection model to acquire 2D bounding box information (the 2D bounding box information includes the center point coordinates of the 2D bounding box and the width and height information of the 2D bounding box) and class information corresponding to at least one object on the source image, (ii) acquires the orientation information of the object with respect to the center point of the 2D bounding box through a direction estimation model, (iii) acquires object shape information corresponding to the class information by referring to pre-set shape information for each class, and combines the 2D bounding box information and the object shape information to acquire 3D object size information corresponding to the class information. (b) The 3D auto-labeling device generates a 3D bounding box in a pre-configured virtual 3D space by referring to the depth information of the center point, the direction information and the 3D object size information obtained by referring to the depth map for the original image generated through the depth estimation model, (c) The 3D auto-labeling device generates a 3D pseudo-label including the 3D bounding box, and generates training data by annotating the original image with the 3D pseudo-label, A method that includes this.

2. In step (b) above, The method according to claim 1, wherein the 3D auto-labeling device refers to the depth information, inversely projects the center point onto the preset virtual 3D space according to the scale magnification, obtains the center 3D coordinates corresponding to the center point, and generates the 3D bounding box by referring to 3D object size adjustment information obtained by adjusting the 3D object size information according to the scale magnification and the direction information, with the center 3D coordinates as the center.

3. The scaling factor is a coordinate system transformation factor between the original image and the pre-set virtual 3D space, and the scaling factor includes sx, which is the x-coordinate scale factor, and sy, which is the y-coordinate scale factor, and sx and sy are determined as follows: And, And, The method according to claim 2, wherein Wv is the width of the pre-set virtual 3D space, Worig is the width of the original image, Hv is the height of the pre-set virtual 3D space, and Horig is the height of the original image.

4. The aforementioned pre-configured virtual 3D space is a perspective space which is a 3D space having a vanishing point, and the aforementioned pre-configured virtual 3D space is set as the virtual coordinate system of the virtual camera, and the virtual internal matrix Kv of the virtual camera is defined as follows: The method according to claim 3, wherein fv is the virtual focal length, cvx is the x-coordinate of the virtual principal point, cvy is the y-coordinate of the virtual principal point, cvx is the product of sx and cx, which is the x-coordinate of the principal point of the original image, and cvy is the product of sy and cy, which is the y-coordinate of the principal point of the original image.

5. The 3D auto-labeling device, in back-projecting the center point onto the preset virtual 3D space according to the scale magnification, obtains the center 3D coordinates (uv, vv, Zv) corresponding to the center point based on the following formula, and at that time, The method according to claim 4, wherein u and v are the coordinates of the center point, Zcam is the depth information of the center point, and fx is the focal length of the original image.

6. In step (c) above, The method according to claim 4, wherein the 3D auto-labeling device, in generating the 3D pseudo-label, includes in the 3D pseudo-label coordinate information obtained by converting the 3D bounding box to coordinates on the original image by referring to the scale magnification and the class information.

7. The 3D auto-labeling device converts the 3D bounding box to coordinates on the original image based on the following formula, and in doing so, The method according to claim 6, wherein Zcam is depth information on the original image, Xcam is the x-coordinate of the original image, Ycam is the y-coordinate of the original image, uv is the x-coordinate of any one point of the 3D bounding box, vv is the y-coordinate of any one point of the 3D bounding box, Zv is the z-coordinate of any one point of the 3D bounding box, and fx and fy are the focal lengths of the original image.

8. In step (b) above, When the 3D auto-labeling device generates the 3D bounding boxes in the preset virtual 3D space, if at least two of the generated 3D bounding boxes, namely the first 3D bounding box and the second 3D bounding box, overlap each other, the device performs a sub-process to adjust the position of the first 3D bounding box by referring to a first point which is any one point within a predetermined area centered on the first center point of the first 2D bounding box corresponding to the first 3D bounding box, and (ii) a sub-process to adjust the position of the first 3D bounding box by referring to the second center point of the second 2D bounding box corresponding to the second 3D bounding box. The method according to claim 1, comprising: (iii) performing one of the subprocesses for adjusting the position of the second 3D bounding box by referring to a second point which is any one point within a predetermined region; and (iii) performing one of the subprocesses for adjusting the positions of the first 3D bounding box and the second 3D bounding box by referring to a third point which is any one point within a predetermined region centered on the first center point of the first 2D bounding box corresponding to the first 3D bounding box, and a fourth point which is any one point within a predetermined region centered on the second center point of the second 2D bounding box corresponding to the second 3D bounding box.

9. In a method for training a monocular 3D object detection model using training data generated through 3D autolabeling, (a) When a 3D auto-labeling device acquires an original image, (i-1) through a 2D object detection model, 2D object detection is performed on the original image to acquire 2D bounding box information (the 2D bounding box information includes the center point coordinates of the 2D bounding box and the width and height information of the 2D bounding box) and class information corresponding to at least one object on the original image, (i-2) through a direction estimation model, the direction information of the object is acquired with respect to the center point of the 2D bounding box, (i-3) by referring to pre-set shape information for each class, object shape information corresponding to the class information is acquired, and the 2D bounding box information and the object shape information are combined to acquire 3D object size information corresponding to the class information, (ii) the original image generated through a depth estimation model (iii) a process of generating a 3D bounding box in a pre-configured virtual 3D space by referring to the depth information of the center point obtained by referring to a depth map for the center point, the direction information and the 3D object size information, and (iii) a process of generating a 3D pseudo-label including the 3D bounding box and annotating the original image with the 3D pseudo-label, thereby generating first training data with pseudo-labels, the learning device then acquires at least one second training data (the second training data includes the first training data and fine-labeled training data), inputs the second training data into a monocular 3D object detection model, and generates 2D information and 3D information respectively as a result of detecting at least one training object on the training image contained in the second training data using the monocular 3D object detection model. (b) The learning device, referring to the 2D information and the 3D information, calculates the 2D-3D consistency loss, the depth Kullback-Leibler divergence loss, the region loss, and the query divergence loss. The steps include training the monocular 3D object detection model to minimize the combined loss, which includes at least a portion of the versity loss, Methods that include...

10. The method according to claim 9, wherein the 2D information includes size information of the predicted 2D bounding box, predicted class information of the training object, projected center information and diversity information, and the 3D information includes size information of the predicted 3D bounding box, predicted depth information, depth Kullback-Leibler divergence information and predicted orientation information of the training object.

11. In step (b) above, The method according to claim 10, wherein the 2D-3D consistency loss is generated by referring to the difference between information obtained by converting the predicted 3D bounding box into coordinates on the training image and the coordinate information of the predicted 2D bounding box.

12. In step (b) above, The aforementioned depth Kullback-Leibler divergence loss is generated using the aforementioned depth Kullback-Leibler divergence information, and in doing so, The method according to claim 10, wherein the depth Kullback-Leibler divergence information is generated by referencing the ground truth depth information contained in the ground truth label for the training image and the predicted depth information.

13. In step (b) above, The method according to claim 10, wherein the region loss is generated by generating a Dice loss and a BCE (Binary Cross-entropy) loss, respectively, using the predicted 3D bounding box and the ground truth labels for the training image, and assigning predetermined weights to the Dice loss and the BCE loss, respectively.

14. In step (b) above, The method according to claim 10, wherein the query diversity loss is generated using batch information of the monocular 3D object detection model and 3D query information output through the 3D decoder of the monocular 3D object detection model.

15. In a 3D auto-labeling device that generates training data for training a monocular 3D object detection model, At least one memory to store instructions, The system includes at least one processor configured to execute the aforementioned instructions, and in doing so, The processor (i) when a source image is acquired, (i) performs 2D object detection on the source image through a 2D object detection model to acquire 2D bounding box information (the 2D bounding box information includes the center point coordinates of the 2D bounding box and the width and height information of the 2D bounding box) and class information corresponding to at least one object on the source image, (ii) acquires the orientation information of the object with respect to the center point of the 2D bounding box through a direction estimation model, and (iii) refers to pre-set shape information for each class to determine the object corresponding to the class information. A 3D auto-labeling device that performs the following processes: (II) acquiring shape information, combining the 2D bounding box information and the object shape information to acquire 3D object size information corresponding to the class information; (II) generating a 3D bounding box in a pre-set virtual 3D space by referring to the depth information of the center point, the direction information and the 3D object size information acquired by referring to a depth map for the original image generated through a depth estimation model; and (III) generating a 3D pseudo-label including the 3D bounding box, annotating the original image with the 3D pseudo-label to generate training data.

16. In the above process (II), The 3D auto-labeling apparatus according to claim 15, wherein the processor refers to the depth information, inversely projects the center point onto the preset virtual 3D space according to the scale factor, obtains the center 3D coordinates corresponding to the center point, and generates the 3D bounding box by referring to 3D object size adjustment information obtained by adjusting the 3D object size information according to the scale factor and the direction information, with the center 3D coordinates as the center.

17. The scaling factor is a coordinate system transformation factor between the original image and the pre-set virtual 3D space, and the scaling factor includes sx, which is the x-coordinate scale factor, and sy, which is the y-coordinate scale factor, and sx and sy are determined as follows: And, And, The 3D auto-labeling apparatus according to claim 16, wherein Wv is the width of the pre-set virtual 3D space, Worig is the width of the original image, Hv is the height of the pre-set virtual 3D space, and Horig is the height of the original image.

18. The aforementioned pre-configured virtual 3D space is a perspective space which is a 3D space having a vanishing point, and the aforementioned pre-configured virtual 3D space is set as the virtual coordinate system of the virtual camera, and the virtual internal matrix Kv of the virtual camera is defined as follows: The 3D auto-labeling apparatus according to claim 17, wherein fv is the virtual focal length, cvx is the x-coordinate of the virtual principal point, cvy is the y-coordinate of the virtual principal point, cvx is the product of sx and cx, which is the x-coordinate of the principal point of the original image, and cvy is the product of sy and cy, which is the y-coordinate of the principal point of the original image.

19. The processor, in backprojecting the center point onto the pre-set virtual 3D space according to the scale factor, obtains the center 3D coordinates (uv, vv, Zv) corresponding to the center point based on the following formula, and at that time, The 3D auto-labeling apparatus according to claim 18, wherein u and v are the coordinates of the center point, Zcam is the depth information of the center point, and fx is the focal length of the original image.

20. In the above (III) process, The 3D auto-labeling apparatus according to claim 18, wherein the processor, in generating the 3D pseudo-label, includes in the 3D pseudo-label coordinate information obtained by converting the 3D bounding box to coordinates on the original image by referring to the scale factor and the class information.

21. The processor converts the 3D bounding box to coordinates on the original image based on the following formula, and in doing so, The 3D auto-labeling apparatus according to claim 20, wherein Zcam is depth information on the original image, Xcam is the x-coordinate of the original image, Ycam is the y-coordinate of the original image, uv is the x-coordinate of any one point of the 3D bounding box, vv is the y-coordinate of any one point of the 3D bounding box, Zv is the z-coordinate of any one point of the 3D bounding box, and fx and fy are the focal lengths of the original image.

22. In the above process (II), When the processor generates the 3D bounding boxes in the pre-configured virtual 3D space, if at least two of the generated 3D bounding boxes, namely the first 3D bounding box and the second 3D bounding box, overlap each other, (i) a subprocess adjusts the position of the first 3D bounding box by referring to a first point which is any point within a predetermined region centered on the first center point of the first 2D bounding box corresponding to the first 3D bounding box, or (ii) any point within a predetermined region centered on the second center point of the second 2D bounding box corresponding to the second 3D bounding box 3D auto-labeling apparatus according to claim 15, comprising: (iii) performing one of the sub-processes for adjusting the position of the second 3D bounding box by referring to a second point which is any one of the following: (iii) performing one of the sub-processes for adjusting the positions of the first 3D bounding box and the second 3D bounding box by referring to a third point which is any one of the following points within a predetermined region centered on the first center point of the first 2D bounding box corresponding to the first 3D bounding box, and a fourth point which is any one of the following points within a predetermined region centered on the second center point of the second 2D bounding box corresponding to the second 3D bounding box.

23. In a learning device that learns a monocular 3D object detection model using training data generated through 3D auto-labeling, At least one memory to store instructions, The system includes at least one processor configured to execute the aforementioned instructions, and in doing so, The processor performs the following processes: (i) when a source image is acquired by a 3D auto-labeling device, (i-1) 2D object detection on the source image through a 2D object detection model to acquire 2D bounding box information (the 2D bounding box information includes the center point coordinates of the 2D bounding box and the width and height information of the 2D bounding box) and class information corresponding to at least one object on the source image; (i-2) direction information of the object with respect to the center point of the 2D bounding box through a direction estimation model; (i-3) object shape information corresponding to the class information by referring to pre-set shape information for each class; and a process of combining the 2D bounding box information and the object shape information to acquire 3D object size information corresponding to the class information; and (ii) a depth map for the source image generated through a depth estimation model. (ii) A process to generate a 3D bounding box in a preset virtual 3D space by referring to the obtained depth information of the center point, the direction information and the 3D object size information, and (iii) a process to generate a 3D pseudo-label including the 3D bounding box and annotate the original image with the 3D pseudo-label, thereby generating first training data with pseudo-labels, and at least one second training data (the second training data includes the first training data and fine-labeled training data), input the second training data into a monocular 3D object detection model, and generate 2D information and 3D information respectively as a result of detecting at least one training object on the training image contained in the second training data using the monocular 3D object detection model, and (ii) a process to generate 2D-3D consistency loss (2D-3D Consistency Loss, Depth Kullback-Leibler Divergence LossA learning device that performs a process of training the monocular 3D object detection model to minimize integrated losses, including Divergence Loss, Region Loss, and Query Diversity Loss.

24. The learning apparatus according to claim 23, wherein the 2D information includes size information of the predicted 2D bounding box, predicted class information of the learning object, projected center information, and diversity information, and the 3D information includes size information of the predicted 3D bounding box, predicted depth information, depth Kullback-Leibler divergence information, and predicted orientation information of the learning object.

25. In the above process (II), The learning device according to claim 24, wherein the 2D-3D consistency loss is generated by referring to the difference between information obtained by converting the predicted 3D bounding box into coordinates on the learning image and the coordinate information of the predicted 2D bounding box.

26. In the above process (II), The aforementioned depth Kullback-Leibler divergence loss is generated using the aforementioned depth Kullback-Leibler divergence information, and in doing so, The learning device according to claim 24, wherein the depth Kullback-Leibler divergence information is generated by referring to the ground truth depth information contained in the ground truth label for the learning image and the predicted depth information.

27. In the above process (II), The learning apparatus according to claim 24, wherein the region loss is generated by generating a Dice loss and a BCE (Binary Cross-entropy) loss, respectively, using the predicted 3D bounding box and the ground truth label for the learning image, and assigning predetermined weights to each of the Dice loss and the BCE loss.

28. In the above process (II), The learning device according to claim 24, wherein the query diversity loss is generated using batch information of the monocular 3D object detection model and 3D query information output through the 3D decoder of the monocular 3D object detection model.

Images