Method and apparatus for generating 3D ground live

Abstract

A method and apparatus for generating a three-dimensional (3D) ground live (GT) is disclosed. The method includes: obtaining a 3D GT of a target object by using data obtained by a lidar; obtaining a 2D frame of the obtained 3D GT by projecting the obtained 3D GT onto a 2D plane; comparing the 2D frame with a frame of the 2D GT of the target object obtained by the camera; and generating a final 3D GT of the target object by correcting the obtained 3D GT based on the comparison result.

Description

Technical Field

[0001] The present disclosure relates to a technology for generating three-dimensional (3D) ground truth (GT), and more particularly, to a method and apparatus for generating 3D GT capable of generating high-quality 3D GT by adjusting the synchronization of data obtained by a laser radar (LiDAR) and a camera. Background Art

[0002] In the mobile industry (e.g., autonomous driving) and the advanced sensor industry, reliability verification is very important. In order to take advantage of advanced driver assistance systems (ADAS) and sensor development, objects (e.g., people, vehicles, street trees, lanes, etc.) may need to be classified. In some implementations, ground truth (GT) labeling may be required for verification. For example, for autonomous driving, object recognition technology may be required to detect people, signals, and other vehicles. In order to build an object identifier, a set of learning data labeled with the shape and type of each object may be required. In some implementations, for example, all images or videos may need to be analyzed and interpreted in advance to identify objects, and this process is generally referred to as GT labeling. Labeled data can also be used as evaluation criteria for ADAS and autonomous driving algorithms.

[0003] 3D GT can obtain data by using a laser radar. In order to obtain 3D GT using two sensors (eg, a camera and a laser radar), synchronization (sync.) consistency of data obtained by the two sensors is an important issue.

[0004] However, because the LiDAR sensor is a physically rotating sensor, it may be difficult to synchronize the 0-degree and 360-degree data with the camera image. In particular, when the size of the target object is small or the relative speed of the target object is large, it is more difficult to synchronize the data from the LiDAR sensor with the camera image.

[0005] The description in this background section is provided to enhance understanding of the background of the present disclosure and includes description in addition to the prior art that is known to one of ordinary skill in the art to which this technology belongs. Summary of the invention

[0006] The following summary briefly summarizes certain features. The summary is not an extensive overview and is not intended to identify key or decisive elements.

[0007] One aspect of the present disclosure provides a method and apparatus for generating three-dimensional (3D) ground truth (GT), which is capable of generating high-quality 3D GT by adjusting (eg, correcting) synchronization of data obtained by a lidar and a camera.

[0008] Another aspect of the present disclosure provides a method and apparatus for generating 3D GT, which is capable of generating high-quality 3D GT by adjusting synchronization of data obtained by a laser radar and a camera using a 2DGT.

[0009] Yet another aspect of the present disclosure provides a method and apparatus for generating a 3D GT, which is capable of generating high-quality 3D GT by comparing a 2D frame as a result of projecting the 3D GT with a frame of the 2D GT and correcting the 3D GT.

[0010] Yet another aspect of the present disclosure provides a method and apparatus for generating a 3D GT, which can quickly generate high-quality 3D GT.

[0011] The technical problems to be solved by the present disclosure are not limited to the above-mentioned problems, and those skilled in the art to which the present disclosure belongs will clearly understand any other technical problems not mentioned herein through the following description.

[0012] A device may include: one or more processors; and a memory storing instructions, which, when executed by the one or more processors, cause the device to: obtain a three-dimensional (3D) ground truth (GT) of a target object based on sensing information of a sensor; obtain a two-dimensional (2D) frame of the obtained 3D GT by projecting the obtained 3D GT onto a 2D plane; compare the 2D frame with a frame of the 2D GT corresponding to the target object and obtained based on sensing information of a camera; generate an adjusted 3D GT of the target object by adjusting the obtained 3D GT based on a comparison result of the 2D frame with the frame of the 2D GT; and output a signal indicating the adjusted 3D GT.

[0013] The instructions, when executed by the one or more processors, may cause the apparatus to generate an adjusted 3D GT by minimizing differences between vertices of the 2D box and vertices of the box of the 2D GT.

[0014] The instructions, when executed by the one or more processors, may cause the device to: obtain multiple 3D GTs by reflecting a preset initial error value in the obtained 3DGTs; obtain a 2D box for each of the multiple 3D GTs; and generate a 3D GT in which the difference between the vertices of one of the 2D boxes of the multiple 3D GTs and the vertices of the box of the 2D GT is minimized, as an adjusted 3D GT.

[0015] The instructions, when executed by the one or more processors, may cause the device to: obtain a plurality of other 3D GTs by reflecting the first error value to the one 3D GT after changing the initial error value to a first error value smaller than the initial error value based on the one 3D GT; obtain 2D boxes for the plurality of other 3D GTs; and generate another 3D GT in which the difference between the vertices of one of the 2D boxes of the plurality of other 3D GTs and the vertices of the box of the 2D GT is minimized, as an adjusted 3D GT.

[0016] When executed by the one or more processors, the instructions may cause the apparatus to iterate the steps of obtaining the 2D frame and generating the adjusted 3D GT a specified number of times to generate the adjusted 3D GT.

[0017] At least one of the initial error value and the first error value may be determined based on a distance to the target object.

[0018] When executed by the one or more processors, the instructions may cause the apparatus to obtain the 2D frame by projecting the obtained 3DGT onto the 2D plane in a rectangular parallelepiped form.

[0019] When executed by the one or more processors, the instructions may cause the device to obtain the 2D frame by changing the obtained 3DGT into coordinate information of eight vertices of a cuboid in a 2D coordinate system and projecting the coordinate information of the eight vertices onto an image.

[0020] The instructions, when executed by the one or more processors, may cause the apparatus to: determine a position of the target object based on the adjusted 3DGT; and generate a signal for a vehicle based on the position of the target object.

[0021] The instructions, when executed by the one or more processors, may cause the apparatus to: control autonomous driving of a vehicle based on at least one of the adjusted 3DGT and the position of the target object.

[0022] A method may include: obtaining, by a processor, a three-dimensional (3D) ground truth (GT) of a target object based on sensing information of a sensor; obtaining, by the processor, a two-dimensional (2D) frame of the obtained 3D GT by projecting the obtained 3D GT onto a 2D plane; comparing, by the processor, the 2D frame with a frame of the 2D GT corresponding to the target object and obtained based on sensing information of a camera; generating, by the processor, an adjusted 3D GT of the target object by adjusting the obtained 3D GT based on the comparison result; and outputting a signal indicating the adjusted 3D GT.

[0023] The method may further perform one or more operations described herein.

[0024] These and other features and advantages are described in more detail below.The features briefly summarized above regarding the present disclosure are merely exemplary aspects of the detailed description of the present disclosure described below and do not limit the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The above and other objects, features and advantages of the present disclosure will become more apparent through the following detailed description in conjunction with the accompanying drawings:

[0026] Figure 1 is a schematic diagram illustrating the synchronization problem between the lidar and the camera;

[0027] Figure 2 is a flow chart illustrating a method of generating a 3D GT;

[0028] Figure 3 shows a schematic diagram illustrating 3D GT and 2D GT obtained by a sensor;

[0029] Figure 4 is a schematic diagram illustrating an operation of comparing a 3D GT obtained by a lidar with a 2D GT obtained by a camera;

[0030] Figure 5 is a schematic diagram illustrating the operation of generating the final 3D GT through a rule-based approach;

[0031] Figure 6 shows a graph illustrating the operation of generating the final 3D GT by the Newton-Raphson method and the Euler method;

[0032] Figure 7 is a block diagram illustrating an apparatus for generating a 3D GT; and

[0033] Figure 8 is a block diagram illustrating a computing system for executing a method of generating a 3D GT. DETAILED DESCRIPTION

[0034] Hereinafter, various examples of the inventive concept will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement the inventive concept. However, the inventive concept is not limited to the examples described herein and can be variously modified in many different forms.

[0035] When describing an example of the present specification, when the detailed description of the related art is considered to obscure the subject of the features of the present specification, the detailed description will be omitted. In the accompanying drawings, parts irrelevant to the description will not be shown in order to make the present disclosure clear.

[0036] It should be understood that when an element is referred to as being "connected" or "coupled" to another element, it may be directly connected to the other element or indirectly connected to the other element. In addition, when some parts "include" or "have" certain elements, unless explicitly described to the contrary, it means that other elements may be further included without excluding them.

[0037] Expressions such as "first" or "second" etc. can express their elements regardless of their priority or importance, and can be used to distinguish one element from another element, but are not limited to these components. Therefore, without departing from the scope of the present disclosure, a first component of one example can be referred to as a second component of another example. Similarly, a second component of one example can be referred to as a first component of another example.

[0038] In the present disclosure, components that are distinguished from each other are only used to clearly describe the characteristics and do not mean that the components must be separated. That is, multiple components can be integrated to form a single hardware or software unit, or a component can be distributed to form multiple hardware or software units. Accordingly, such integrated or distributed components / elements are included in the scope of the present disclosure, even if not mentioned separately.

[0039] In the present disclosure, the components described in the present disclosure do not necessarily mean necessary components, and some components may be optional components. Therefore, an implementation consisting of a subset of the components described in the exemplary implementation is also included in the scope of the present disclosure. In addition, in addition to the components described in the various examples, one or more implementations that also include other components are also included in the scope of the present disclosure.

[0040] In the present disclosure, for the convenience of description, expressions of positional relationships used herein are described, such as up, down, left, right, etc. When the drawings shown in this specification are viewed in reverse, the positional relationships described in the specification may be interpreted in an opposite manner.

[0041] As used herein, each of phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "at least one of A, B or C" may include any or all possible combinations of the items listed together in the corresponding one phrase.

[0042] In order to generate 3D GT using two sensors (e.g., LiDAR and camera), it may be important to synchronize the camera and LiDAR. Figure 1 As shown in , because LiDAR is a physically rotating sensor, it can be difficult to match data from 0 and 360 degree viewpoints with the camera image. Figure 1As shown, the 0.9s LiDAR data can be synchronized with the camera image, but the 0.1s LiDAR data may not be synchronized with the camera image.

[0043] According to the present disclosure, the 2D GT obtained by the camera can be used to adjust (e.g., correct) the 3D GT of the target object obtained by the lidar, so that the lidar data is synchronized with the camera data, thereby generating high-quality 3D GT for the target object.

[0044] According to the present disclosure, a 2D frame for a lidar can be obtained by projecting the 3D GT of a target object obtained by a lidar onto a 2D plane in the form of a rectangle, and the obtained 2D frame can be compared with the frame of the 2D GT obtained by a camera so that the lidar data is synchronized with the camera data, thereby generating a synchronized or adjusted 3D GT of the target object through synchronization.

[0045] According to the present disclosure, the 2D box can be adjusted to minimize the difference between the vertices of the 2D box projected to a coordinate system (e.g., a world coordinate system) and the vertices of the 2D GT box, thereby generating a final 3D GT of the target object. In this case, the operation of minimizing the difference between the vertices of the present disclosure can be repeated a specified number of times, and the operation of minimizing the difference by using various objective functions can be repeated a specified number of times.

[0046] Reference Figures 1 to 7 A method and apparatus for generating 3D GT according to the present disclosure is described.

[0047] Figure 2 is a flowchart illustrating a method of generating a 3D GT according to the present disclosure, which illustrates operations performed by an apparatus for generating a 3D GT.

[0048] refer to Figure 2 , the method of generating 3D GT includes S210: obtaining 3D GT of the target object by using lidar data about the target object obtained by a lidar, and obtaining 2D GT of the target object by using image data about the target object obtained by a camera.

[0049] In this case, at S210 , the 3D GT obtained by the lidar may be the 3D GT of the lidar coordinate system, the 2D GT obtained by the camera may be the 2D GT of the world coordinate system, and the 2D GT obtained by the camera may be the accurately labeled 2D GT because the 2D GT is the GT obtained from the image.

[0050] The method according to an example of the present disclosure is provided to solve the synchronization problem that occurs when the 3D GT is moved to the world coordinate system in order to generate the final 3D GT by using the 3D GT obtained by the lidar and the 2D GT obtained by the camera, and uses the accurately marked 2D GT obtained by the camera.

[0051] For example, in S210, if Figure 3 As shown in part (a) of FIG. 3 , the lidar data can be used to obtain the 3D GT 310 of the lidar coordinate system of the target object, and as shown in FIG. Figure 3 As shown in part (b), the image data obtained by the camera can be used to obtain the 2D GT 320 of the world coordinate system of the target object.

[0052] If the 3D GT of the laser radar and the 2D GT of the camera are obtained in S210 , the 3D GT of the laser radar may be projected onto a 2D plane of the world coordinate system in S220 to obtain a 2D frame of the 3D GT of the laser radar in S220 .

[0053] In this case, since the 3D GT of the laser radar is a laser radar coordinate system and includes the coordinates (x, y, z), width, height and length of the center point of the target object, in S220, by changing the 3D GT of the laser radar coordinate system of the target object into a world coordinate system in the form of a cuboid, 8 vertices can be projected onto the image to obtain a 2D box of the 3D GT in the world coordinate system. For example, in S220, the coordinates (x, y, z), width, height and length can be converted into coordinate information (x0, y0, z0), (x1, y1, z1), ..., (x7, y7, z7) of eight vertices, and by projecting such converted coordinate information onto the image, a 2D box can be obtained.

[0054] For example, in S220, if Figure 4 As shown, the 3D GT 410 changed into a rectangular parallelepiped shape of the world coordinate system of the target object 400 may be projected onto the image, thereby generating a 2D frame 430 of the 3D GT 410 .

[0055] If a 2D frame in the world coordinate system is obtained in S220, then in S230 and S240, the 2D frame of the 3D GT obtained by the lidar is compared with the 2D GT frame obtained by the camera, and the 3D GT is adjusted (e.g., corrected) based on the result of comparing the two frames to generate a final 3D GT.

[0056] In the example, Figure 4As shown, the vertices of the 2D box 430 of the 3D GT 410 may be compared with the vertices of the 2D GT box 420 obtained by the camera at S230 , and the final 3D GT may be generated using an approximation or optimization technique that minimizes the difference between the vertices of the two boxes 420 and 430 at S240 .

[0057] In this case, operations S220 to S240 may be repeated a designated number of times to generate a high-quality final 3D GT.

[0058] In an example, in S220 to S240, the two boxes may be compared with each other by using at least one of a rule-based method, a Newton / Newton-Raphson method, and an Euler method, and the 3D GT may be adjusted to minimize the difference between four vertices of the two boxes, thereby generating a final 3D GT of the target object.

[0059] The objective function of the approximation technique for generating the final three-dimensional GT at S240 may include three variables x, y and z to adjust the 3D GT. However, considering the motion characteristics of the vehicle, since the problem usually only occurs on the variable x (for example, the variable x as the vehicle's driving direction is the main variable), the objective function may be defined as a function of the variable x.

[0060] When reference Figure 5 When describing the operation of generating the final 3D GT using the rule-based method, a plurality of 3D GTs are obtained by reflecting a preset initial error value (e.g., an initial error value of 1 m) in the 3D GT obtained for the target object using the lidar data, and a 2D box is obtained for each 3D GT. Figure 5As shown, when t (x-axis, y-axis, z-axis) of the obtained 3D GT is (5m, 2m, 1m), after generating 3D GT with initial error values ​​(e.g., maximum error value of 1m) for +1m and -1m, that is, 3D GT of (6m, 2m, 1m) and (4m, 2m, 1m), and after projecting the three 3D GTs onto the image in a rectangular shape, 2D boxes with minimum and maximum values ​​are generated or obtained, for example, 2D box 510 of the obtained 3D GT (existing 3D GT), 2D box 530 of the 3D GT for +1m, and 2D box 520 of the 3D GT for -1m. Then, the loss L1 between each of the 2D boxes 510, 520, and 530 and the 2D GT box 540 is calculated, that is, the loss L1 is calculated using the four vertices of each of the 2D boxes 510, 520, and 530 and the 2D GT box 540, so that the 3D GT with the lowest loss L1 is determined as the new 3D GT. When the new 3D GT is determined, the initial error value is changed to a first error value less than the initial error value, for example, 50% (0.5m) of the previous error value, and the first error value of 0.5m (+0.5m, -0.5m) is reflected in the new 3D GT to generate a plurality of new 3D GTs based on the new 3D GT, and then a 2D box is generated for each of the plurality of new 3D GTs. Then, by calculating the loss L1 between the newly generated 2D box and the 2D GT, the 3D GT with the lowest loss L1 is again determined as the new 3D GT. By repeating the above process for a specified number of times (eg, after reducing the maximum error value to 0.25 m, 0.125 m, etc.), the final 3D GT of the target object can be generated.

[0061] In an example, the method of the present disclosure may determine the initial error value and the first error value based on the distance from the target object in a rule-based method. For example, in the method of the present disclosure, depending on the distance from the target object, the initial error value may be set to be greater than or less than 1m, and the scheme of changing from the initial error value to the first error value may determine the initial error value differently by reflecting the distance from the target object instead of 50% of the previous error value. In another example, if the initial error value, the first error value, the second error value, and the third error value are changed sequentially, the method of the present disclosure changes the initial error value to a second or third error value in addition to the first error value according to the distance from the target object, and the number of repetitions may be changed. Of course, the change method may be determined by the individual or business operator who provides the technology disclosed herein.

[0062] Reference Figure 6 Describe the method by using the Newton / Newton-Raphson method ( Figure 6 Part (a)) and Euler's method ( Figure 6Part (b)) generates the operation of the final 3D GT, uses the projection matrix and the four vertices of the 2D GT to generate the equation f(x) for each technology, and uses each technology to approximate the solution of f(x). In this case, f(x) = (u1-u1') + (v1-v1') + ... (u4-u4') = 0, u1 to u4 can refer to the x-axis coordinate values ​​of the four vertices of the 2D box, u1' to u4' can refer to the x-axis coordinate values ​​of the four vertices of the 2D GT box, and x can be the center value x. "f(x)" may include equations approximated by Newton's law and / or Euler's law.

[0063] The initial value can use the center value "x" of the 3D GT previously obtained, and the final GT can be generated by approximating the solution of f(x). Because the solution of approximating f(x) can converge quickly by significantly changing the value when the difference between the 3D cuboid and the 2D GT is large, and slowly changing the value when the difference is small, it can converge faster than simply changing the value little by little. Therefore, the final 3D GT of the target object can be quickly generated. The Newton / Newton-Raphson method and the Euler method can be repeated a specified number of times, for example three times, to generate the final three-dimensional GT. Since the Newton / Newton-Raphson method and the Euler method are known to those skilled in the art, a detailed description is omitted.

[0064] As described above, the method of generating 3D GT according to the present disclosure can generate high-quality 3D GT by adjusting the synchronization of data obtained by the laser radar and the camera.

[0065] The method of generating 3D GT according to the present disclosure can generate high-quality 3D GT by adjusting the 3D GT of the target object obtained by the laser radar using the 2D GT obtained by the camera.

[0066] According to the method of generating 3D GT of the present disclosure, a 2D frame obtained by projecting the 3D GT obtained by the laser radar into the world coordinate system in a rectangular parallelepiped shape can be compared with the 2D GT frame obtained by the camera to adjust the 3D GT, thereby generating high-quality 3D GT.

[0067] The method for generating 3D GT according to the present disclosure can adjust the synchronization problem caused by moving the 3D GT obtained by the lidar to the world coordinate system through the 2D GT obtained by the camera, and repeat the adjustment operation a specified number of times using the approximation technology, thereby generating high-quality 3D GT.

[0068] Figure 7 is a block diagram illustrating an apparatus for generating a 3D GT, which illustrates the execution Figures 1 to 6 The method of the device.

[0069] refer to Figure 7 , the apparatus 700 for generating 3D GT may include a receiving device 710, an obtaining device 720, a comparing device 730, a generating device 740, and a storage 750. One or more of the receiving device 710, the obtaining device 720, the comparing device 730, and / or the generating device 740 may be implemented by one or more processors. At least a part of the receiving device 710 and / or the obtaining device 720 may be implemented by one or more communication interfaces and / or sensors.

[0070] Storage 750 may be a component for storing all or part of the data related to the technology disclosed herein, and may store data obtained by a lidar, data obtained by a camera, information about a target object, 3D GT of the target object obtained by a lidar, 2D GT of the target object obtained by a camera, an algorithm for generating a final 3D GT (e.g., an approximation technique algorithm), and data about the final 3D GT of the target object.

[0071] The receiving device 710 receives lidar data obtained for the target object from the lidar, and receives image data obtained for the target object from the camera.

[0072] The obtaining device 720 obtains 3D GT by using lidar data obtained by a lidar for a target object, and obtains 2D GT by using image data obtained by a camera.

[0073] In this case, the 3D GT obtained by the lidar can be the 3D GT of the lidar coordinate system, and the 2D GT obtained by the camera can be the 2D GT of the world coordinate system, and because the 2D GT obtained by the camera is obtained from the image, the 2D GT can be accurately marked.

[0074] The acquisition device 720 can obtain the 2D frame of the 3D GT of the laser radar by projecting the 3D GT of the laser radar onto the 2D plane of the world coordinate system.

[0075] In this case, because the 3D GT of the lidar is in the lidar coordinate system and includes the coordinates (x, y, z), width, height and length of the center point of the target object, the acquisition device 720 can change the 3D GT of the world coordinate system into a 3D GT of the world coordinate system in the shape of a rectangular parallelepiped, so that the 2D box of the 3D GT in the world coordinate system can be obtained by projecting 8 vertices onto the image.

[0076] According to an example, when generating a final 3D GT using a rule-based method, the obtaining device 720 may reflect a preset initial error value in the 3D GT to obtain a plurality of 3D GTs, and obtain a 2D box for each of the plurality of 3D GTs.

[0077] According to the example, in order to generate a final 3D GT, when the generating device determines any one of a plurality of 3D GTs, after changing the initial error value to a first error value smaller than the initial error value based on the any one 3D GT, the obtaining device 720 may reflect the first error value in one 3D GT to obtain a plurality of other 3D GTs, and obtain a 2D box of each of the plurality of other 3D GTs.

[0078] In this case, the initial error value and the first error value may be determined in consideration of the distance from the target object, and the number of repetitions may also be determined in consideration of the distance from the target object.

[0079] The comparison device 730 compares the 2D frame of the 3D GT obtained by the laser radar with the 2D GT frame obtained by the camera.

[0080] According to an example, the comparison device 730 may compare four vertices of the 2D box with four vertices of the 2D GT box.

[0081] According to an example, when the obtaining device 720 obtains a plurality of 2D boxes, the comparing device 730 may compare each of the plurality of 2D boxes with the 2D GT box and provide the comparison result to the generating device 740 .

[0082] The generating device 740 generates a final 3D GT of the target object by correcting the 3D GT obtained for the target object based on the comparison result of the comparing device 730 .

[0083] According to an example, the generating device 740 may generate the final 3D GT of the target object by using an approximation or optimization technique that minimizes the difference between the vertices of the two boxes compared by the comparing device 730 .

[0084] In this case, the generating device 740 may adjust the 3D GT of the target object by using at least one of a rule-based method, a Newton / Raphson method, and an Euler method to minimize the difference between four vertices of the two boxes, thereby generating a final 3D GT of the target object.

[0085] As an example, when the final 3D GT is generated by using a rule-based method, the generating device 740 may generate another 3D GT as the final 3D GT based on the comparison result of the comparing device 730, in which the difference between the vertices of the 2D box of each of the plurality of other 3D GTs and the vertices of the box of the 2D GT is minimized. In this case, the generating device may repeat the operation a specified number of times to generate one 3D GT that minimizes the loss L1 of the comparing device as the final 3D GT.

[0086] Although reference Figure 7 In a device according to another example of the present disclosure, repeated description is omitted, but a device according to another example of the present disclosure may include Figures 1 to 6 Everything described in the method.

[0087] According to one aspect of the present disclosure, a method for generating a three-dimensional (3D) ground truth (GT) includes: obtaining a 3D GT of a target object by using data obtained by a lidar; obtaining a two-dimensional (2D) frame of the obtained 3D GT by projecting the obtained 3D GT onto a 2D plane; comparing the 2D frame with a frame of the 2D GT of the target object obtained by a camera; and generating a final 3D GT of the target object by correcting the obtained 3D GT based on the comparison result.

[0088] According to an example, generating the final 3D GT may include generating the final 3D GT by correcting the obtained 3D GT by minimizing a difference between vertices of the 2D box and vertices of the box of the 2D GT.

[0089] According to an example, obtaining a 2D box may include: obtaining multiple 3D GTs by reflecting a preset initial error value in the obtained 3D GT; and obtaining a 2D box for each of the multiple 3D GTs, and generating a final 3D GT may include: generating a 3D GT in which the difference between the vertices of the 2D box of each of the multiple 3D GTs and the vertices of the box of the 2D GT is minimized, as the final 3D GT.

[0090] According to an example, obtaining a 2D box may include: obtaining multiple other 3D GTs by reflecting the first error value to one 3D GT after changing the initial error value to a first error value smaller than the initial error value based on one 3D GT; and obtaining a 2D box for each of the multiple other 3D GTs, and generating a final 3D GT may include: generating another 3D GT in which the difference between the vertices of the 2D box of each of the multiple other 3D GTs and the vertices of the box of the 2D GT is minimized, as the final 3D GT.

[0091] According to an example, the step of obtaining the 2D box and the step of generating the final 3D GT may be performed for a specified number of iterations to generate the final 3D GT.

[0092] According to an example, at least one of the initial error value and the first error value may be determined based on a distance to the target object.

[0093] According to an example, obtaining the 2D frame may include obtaining the 2D frame by projecting the obtained 3D GT onto a 2D plane in a cuboid form.

[0094] According to an example, obtaining the 2D frame may include obtaining the 2D frame by changing the obtained 3D GT into coordinate information of eight vertices of a cuboid in a world coordinate system and projecting the coordinate information of the eight vertices onto the image.

[0095] According to another aspect of the present disclosure, an apparatus for generating three-dimensional (3D) ground truth (GT) includes: an acquisition device, which obtains a 3D GT of a target object by using data obtained by a lidar, and obtains a two-dimensional (2D) frame of the obtained 3D GT by projecting the obtained 3D GT onto a 2D plane; a comparison device, which compares the 2D frame with the frame of the 2D GT of the target object obtained by a camera; and a generation device, which generates a final 3D GT of the target object by correcting the obtained 3D GT based on the comparison result.

[0096] According to an example, the generating apparatus may generate the final 3D GT by correcting the obtained 3D GT by minimizing the difference between the vertices of the 2D box and the vertices of the box of the 2D GT.

[0097] According to the example, the obtaining device can obtain multiple 3D GTs by reflecting a preset initial error value in the obtained 3D GT, and obtain a 2D box for each of the multiple 3D GTs, and the generating device can generate a 3D GT in which the difference between the vertices of the 2D box of each of the multiple 3D GTs and the vertices of the box of the 2D GT is minimized, as the final 3DGT.

[0098] According to the example, the obtaining device can obtain multiple other 3D GTs by reflecting the first error value to one 3DGT after changing the initial error value to a first error value smaller than the initial error value based on one 3D GT, and obtain a 2D box for each of the multiple other 3D GTs, and the generating device can generate another 3D GT in which the difference between the vertices of the 2D box of each of the multiple other 3D GTs and the vertices of the box of the 2D GT is minimized, as the final 3D GT.

[0099] According to an example, the obtaining device and the generating device may obtain the 2D box and generate the final 3DGT at a specified number of iterations to generate the final 3D GT.

[0100] According to an example, at least one of the initial error value and the first error value may be determined based on a distance to the target object.

[0101] According to an example, the obtaining device may obtain the 2D frame by projecting the obtained 3D GT onto a 2D plane in the form of a cuboid.

[0102] According to an example, the obtaining device may obtain the 2D frame by changing the obtained 3D GT into coordinate information in a world coordinate system of eight vertices of a cuboid and projecting the coordinate information of the eight vertices onto the image.

[0103] Figure 8 is a block diagram illustrating a computing system for executing a method of generating a 3D GT.

[0104] refer to Figure 8 , the above method of generating 3D GT can be implemented by a computing system 1000. The computing system 1000 may include at least one processor 1100, a memory 1300, a user interface input device 1400, a user interface output device 1500, a storage 1600, and a network interface 1700 connected via a system bus 1200.

[0105] The processor 1100 may be a central processing device (CPU) or a semiconductor device for processing instructions stored in the memory 1300 and / or the storage 1600. The memory 1300 and the storage 1600 may include various types of volatile or non-volatile storage media. For example, the memory 1300 may include a ROM (Read Only Memory) 1310 and a RAM (Random Access Memory) 1320.

[0106] Accordingly, the process of the method or algorithm described in one or more embodiments of the present disclosure can be directly implemented by hardware, software modules or a combination thereof executed by the processor 1100. The software module can reside in a storage medium (i.e., memory 1300 and / or storage 1600), such as RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disk, solid state drive (SSD), removable disk or CD-ROM. An exemplary storage medium is coupled to the processor 1100, and the processor 1100 can read information from the storage medium and can write information in the storage medium. In another method, the storage medium can be integrated with the processor 1100. The processor 1100 and the storage medium can reside in an application specific integrated circuit (ASIC). The ASIC can reside in a user terminal. In another method, the processor 1100 and the storage medium can reside in a user terminal as separate components.

[0107] According to one or more aspects of the present disclosure, high-quality 3D GT may be generated by correcting the synchronization of data obtained by a lidar and a camera.

[0108] According to one or more aspects of the present disclosure, high-quality 3D GT may be generated by correcting synchronization of data obtained by a lidar and a camera using a 2D GT.

[0109] According to one or more aspects of the present disclosure, high-quality 3D GT may be generated by comparing a 2D box obtained by projection of the obtained 3D GT with a box of the 2D GT and correcting the 3D GT.

[0110] According to one or more aspects of the present disclosure, high-quality 3D GT can be quickly generated by correcting a synchronization problem caused by converting 3D GT obtained by a lidar into a world coordinate system using 2D GT obtained by a camera and repeating the correction operation a specified number of times.

[0111] Effects obtained by various features of the present disclosure may not be limited to the above, and other effects will be clearly understood by a person of ordinary skill in the art from the following disclosure.

[0112] Although various examples of the present disclosure are described for illustrative purposes, it will be understood by those skilled in the art that various modifications, additions and substitutions may be made without departing from the scope and spirit of the present disclosure. Therefore, one or more exemplary embodiments disclosed in the present disclosure are provided for the purpose of description rather than to limit the technical concept of the present disclosure, and it should be understood that such one or more exemplary embodiments are not intended to limit the scope of the technical concept of the present disclosure. The attached claims should understand the scope of protection of the present disclosure, and all technical concepts within the equivalent scope should be interpreted as within the scope of the rights of the present disclosure.

Claims

1. A method comprising the following steps: A processor obtains a three-dimensional (3D) ground truth (GT) of the target object based on the sensing information of the sensor; Obtaining, by the processor, a two-dimensional (2D) box of the obtained 3D GT by projecting the obtained 3D GT onto a 2D plane; The processor compares the 2D frame with a frame of a 2DGT corresponding to the target object and obtained based on sensing information of a camera; generating, by the processor, an adjusted 3D GT of the target object by adjusting the obtained 3D GT based on the comparison result; and A signal indicative of the adjusted 3D GT is output.

2. The method according to claim 1, wherein: The steps to generate the adjusted 3D GT include: Minimize the difference between the vertices of the 2D box and the vertices of the 2D GT's box.

3. The method according to claim 1, wherein: The steps of obtaining the 2D frame include: obtaining a plurality of 3D GTs by reflecting a preset initial error value in the obtained 3D GTs; and obtain a 2D box for each of the plurality of 3D GTs, and The step of generating the adjusted 3D GT includes: A 3D GT in which a difference between vertices of one of the 2D boxes of the plurality of 3D GTs and vertices of the box of the 2D GT is minimized is generated as an adjusted 3D GT.

4. The method according to claim 3, wherein: The steps of obtaining the 2D frame include: After changing the initial error value to a first error value smaller than the initial error value based on the one 3D GT, obtaining a plurality of other 3D GTs by reflecting the first error value to the one 3D GT; and obtain 2D boxes for the other 3D GTs, and The step of generating the adjusted 3D GT includes: Another 3D GT in which a difference between vertices of one of the 2D boxes of the plurality of other 3D GTs and vertices of the box of the 2D GT is minimized is generated as an adjusted 3D GT.

5. The method according to claim 4, wherein: The steps of obtaining the 2D box and generating the adjusted 3D GT are performed for a specified number of iterations to generate the adjusted 3D GT.

6. The method according to claim 4, wherein: At least one of the initial error value and the first error value is determined based on a distance to the target object.

7. The method according to claim 1, wherein: The steps of obtaining the 2D frame include: The 2D frame is obtained by projecting the obtained 3D GT onto the 2D plane in a rectangular parallelepiped form.

8. The method according to claim 7, wherein: The steps of obtaining the 2D frame include: The 2D frame is obtained by changing the obtained 3D GT into coordinate information in a 2D coordinate system of eight vertices of a rectangular parallelepiped and projecting the coordinate information of the eight vertices onto an image.

9. The method according to claim 1, further comprising the steps of: Determining the position of the target object based on the adjusted 3D GT; and Based on the position of the target object, a signal is generated for the vehicle.

10. An apparatus comprising: one or more processors; and a memory storing instructions that, when executed by the one or more processors, cause the apparatus to: Based on the sensing information of the sensor, a three-dimensional (3D) ground truth (GT) of the target object is obtained; Obtain a two-dimensional (2D) box of the obtained 3D GT by projecting the obtained 3D GT onto a 2D plane; Comparing the 2D frame with a frame of the 2D GT corresponding to the target object and obtained based on sensing information of a camera; generating an adjusted 3D GT of the target object by adjusting the obtained 3D GT based on a comparison result of the 2D box and the box of the 2D GT; and A signal indicative of the adjusted 3D GT is output.

11. The device according to claim 10, wherein: The instructions, when executed by the one or more processors, cause the apparatus to: The adjusted 3DGT is generated by minimizing the difference between the vertices of the 2D box and the vertices of the box of the 2D GT.

12. The device according to claim 10, wherein: The instructions, when executed by the one or more processors, cause the apparatus to: Obtaining a plurality of 3D GTs by reflecting a preset initial error value in the obtained 3D GTs; obtaining a 2D box for each of the plurality of 3D GTs; and A 3D GT in which a difference between vertices of one of the 2D boxes of the plurality of 3D GTs and vertices of the box of the 2D GT is minimized is generated as an adjusted 3D GT.

13. The device according to claim 12, wherein: The instructions, when executed by the one or more processors, cause the apparatus to: After changing the initial error value to a first error value smaller than the initial error value based on the one 3D GT, obtaining a plurality of other 3D GTs by reflecting the first error value to the one 3D GT; Obtaining 2D boxes for the plurality of other 3D GTs; as well as Another 3D GT in which a difference between vertices of one of the 2D boxes of the plurality of other 3D GTs and vertices of the box of the 2D GT is minimized is generated as an adjusted 3D GT.

14. The device according to claim 13, wherein: The instructions, when executed by the one or more processors, cause the apparatus to: The steps of obtaining the 2D frame and generating the adjusted 3D GT are iterated a specified number of times to generate the adjusted 3D GT.

15. The device according to claim 13, wherein: At least one of the initial error value and the first error value is determined based on a distance to the target object.

16. The device according to claim 10, wherein: The instructions, when executed by the one or more processors, cause the apparatus to: The 2D frame is obtained by projecting the obtained 3D GT onto the 2D plane in a rectangular parallelepiped form.

17. The device according to claim 16, wherein: The instructions, when executed by the one or more processors, cause the apparatus to: The 2D frame is obtained by changing the obtained 3D GT into coordinate information in a 2D coordinate system of eight vertices of a rectangular parallelepiped and projecting the coordinate information of the eight vertices onto an image.

18. The device according to claim 10, wherein: The instructions, when executed by the one or more processors, cause the apparatus to: Determining the position of the target object based on the adjusted 3D GT; and Based on the position of the target object, a signal is generated for the vehicle.

Images