Method and device for generating depth map

Abstract

The invention discloses a method and a device for generating a depth map. The method comprises the following steps: acquiring LiDAR point cloud generated by LiDAR; obtaining a 3D bounding box of at least a portion of the LiDAR point cloud; generating a first vehicle mask corresponding to the 3D bounding box from an image point cloud obtained by projecting coordinates of points of the LiDAR point cloud into an image coordinate system; and generating a depth map by removing far points not included in the 3D bounding box among the points included in the first vehicle mask.

Description

[0001] Related Application

[0002] This application claims priority to Korean Patent Application No. 10-2023-0171461, filed on November 30, 2023, the entire contents of which are incorporated herein by reference for all purposes. Technical Field

[0003] The present disclosure relates to a method and apparatus for generating a depth map, and more particularly, to a method and apparatus for generating a depth map by removing point clouds passing through the glass of an adjacent vehicle from a light detection and ranging (LiDAR) point cloud. Background Art

[0004] In recent years, as computer vision technologies based on deep neural networks have developed in autonomous driving technologies, various artificial intelligence models such as object detection, semantic segmentation, depth map estimation, and lane detection have been studied.

[0005] For example, depth map estimation has been variously utilized to recognize surrounding situations and spaces, such as peripheral objects and free space, by using a camera in an autonomous driving scenario. Generally, in order to train a network for generating a depth map, a large amount of labeled and accurate learning data (e.g., depth maps for learning) is required.

[0006] A depth map for learning can be generated based on a LiDAR point cloud, but due to the limitations of a LiDAR that emits laser pulses, distant points passing through the window of an adjacent vehicle can be projected onto the depth map, thereby generating an inaccurate depth map. Accordingly, there may be a need to provide a method for generating a more accurate depth map for learning in which distant points from the window of an adjacent vehicle are not projected onto the depth map.

[0007] The information included in this background of the present disclosure is only for enhancing the understanding of the general background of the present disclosure and may not be regarded as an admission or any form of suggestion that this information forms the prior art known to those skilled in the art. Summary of the Invention

[0008] Aspects of the present disclosure are directed to providing a method and apparatus for generating a depth map by filtering a portion of a LiDAR point cloud generated by a LiDAR of an autonomous vehicle.

[0009] Another aspect of the present invention provides a method and apparatus for generating a depth map, the method and apparatus being configured to remove distant points such that distant points are not projected from the window of an adjacent vehicle onto the depth map.

[0010] Another aspect of the present invention provides a method and apparatus for generating a depth map, which are configured to increase the accuracy of the learning depth map for training a depth map estimation network.

[0011] The technical problems to be solved by the present disclosure are not limited to the above problems, and those skilled in the art to which the present disclosure pertains will clearly understand any other technical problems not mentioned herein from the following description.

[0012] According to an aspect of the present disclosure, a method for generating a depth map includes: obtaining a LiDAR point cloud generated by a LiDAR; obtaining a 3D bounding box for at least a part of the LiDAR point cloud; generating a first vehicle mask corresponding to the 3D bounding box from an image point cloud obtained by projecting the coordinates of the points of the LiDAR point cloud into an image coordinate system; and generating a depth map by removing the far points not included in the 3D bounding box from the points included in the first vehicle mask.

[0013] According to an exemplary embodiment of the present disclosure, obtaining the 3D bounding box may include: extracting a close-range point cloud by removing the point cloud at a predetermined distance from the LiDAR point cloud; and obtaining the 3D bounding box based on the close-range point cloud.

[0014] According to an exemplary embodiment of the present disclosure, extracting the close-range point cloud may include: determining the predetermined distance based on at least one of the focal length of the image point cloud and the distortion degree of the spacing between the points of the image point cloud.

[0015] According to an exemplary embodiment of the present disclosure, generating the first vehicle mask may include: generating the first vehicle mask by using a gift warping algorithm or an alpha shape scheme for the points included in the 3D bounding box in the image point cloud.

[0016] According to an exemplary embodiment of the present disclosure content, generating the first vehicle mask may include: determining the second vehicle mask generated by the alpha shape scheme as the first vehicle mask in response to a value obtained by dividing the area of the second vehicle mask generated by the alpha shape scheme by the area of the third vehicle mask generated by the gift warping algorithm being less than a threshold.

[0017] According to an exemplary embodiment of the present disclosure, generating the first vehicle mask may include: generating a camera point cloud that projects the coordinates of the points of the LiDAR point cloud into a camera coordinate system by multiplying the coordinates of the points of the LiDAR point cloud by an extrinsic matrix; and generating an image point cloud by changing the focal length of the camera point cloud.

[0018] According to an exemplary embodiment of the present disclosure, generating an image point cloud may include: generating an image point cloud by converting a coordinate plane of points of a camera point cloud into an image plane having an arbitrary value in an image plane where the focal length is normalized to 1.

[0019] According to an exemplary embodiment of the present disclosure, generating an image point cloud may include: generating an image point cloud by multiplying coordinates of points of a camera point cloud by an intrinsic matrix.

[0020] According to an aspect of the present disclosure, an apparatus for generating a depth map includes: a camera, a LiDAR that generates a LiDAR point cloud, and a processor that obtains a 3D bounding box for at least a portion of the LiDAR point cloud, generates a first vehicle mask corresponding to the 3D bounding box from an image point cloud obtained by projecting coordinates of points of the LiDAR point cloud into an image coordinate system, and generates a depth map by removing far points that are not included in the 3D bounding box from the points included in the first vehicle mask.

[0021] According to an exemplary embodiment of the present disclosure, the processor is further configured to extract a close-range point cloud by removing point clouds at a predetermined distance from the LiDAR point cloud, and generate a 3D bounding box based on the close-range point cloud.

[0022] According to an exemplary embodiment of the present disclosure, the processor is further configured to determine the predetermined distance based on at least one of a focal length of the image point cloud and a degree of distortion of a spacing between points of the image point cloud.

[0023] According to an exemplary embodiment of the present disclosure content, the processor is further configured to generate a first vehicle mask for points included in the 3D bounding box in the image point cloud by using a gift wrapping algorithm or an alpha shape scheme.

[0024] According to an exemplary embodiment of the present disclosure, the processor is further configured to determine the second vehicle mask generated by the alpha shape scheme as the first vehicle mask in response to a value obtained by dividing an area of the second vehicle mask generated by the alpha shape scheme by an area of the third vehicle mask generated by the gift wrapping algorithm being less than a threshold.

[0025] According to an exemplary embodiment of the present disclosure, the processor is further configured to generate a camera point cloud by converting a coordinate reference point of points of the LiDAR point cloud from the LiDAR to the camera.

[0026] According to an exemplary embodiment of the present disclosure, the processor is further configured to generate a camera point cloud by multiplying coordinates of points of the LiDAR point cloud by an extrinsic matrix.

[0027] According to an exemplary embodiment of the present disclosure, the processor is further configured to generate an image point cloud by converting the coordinate plane of the points of the camera point cloud into an image plane with an arbitrary value of the focal length included in the image plane where the focal length is normalized to 1.

[0028] According to an exemplary embodiment of the present disclosure, the processor is further configured to generate an image point cloud by multiplying the coordinates of the points of the camera point cloud by the intrinsic matrix.

[0029] According to an exemplary embodiment of the present disclosure, the LiDAR may be disposed at a position higher than the camera in the vehicle.

[0030] According to an exemplary embodiment of the present disclosure, the LiDAR may be disposed on the vehicle.

[0031] According to an exemplary embodiment of the present disclosure, the camera may be disposed on the vehicle.

[0032] The features briefly summarized above regarding the present disclosure are merely exemplary aspects of the detailed description of the present disclosure described below and do not limit the scope of the present disclosure.

[0033] The methods and apparatuses of the present disclosure have other features and advantages that will be apparent from or more particularly set forth in the accompanying drawings incorporated herein and the following detailed description, which together are used to explain certain principles of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a diagram showing a method of generating a depth map according to an exemplary embodiment of the present disclosure;

[0035] Figure 2 is a flowchart showing a method of generating a depth map according to an exemplary embodiment of the present disclosure;

[0036] Figure 3A is a diagram showing a point cloud in the LiDAR coordinate system according to an exemplary embodiment of the present disclosure;

[0037] Figure 3B is a diagram showing a point cloud projected onto the camera coordinate system according to an exemplary embodiment of the present disclosure;

[0038] Figure 3C is a diagram showing a point cloud projected onto the image coordinate system according to an exemplary embodiment of the present disclosure;

[0039] Figure 4 is a diagram showing a process of projecting a point cloud onto the image coordinate system according to an exemplary embodiment of the present disclosure.

[0040] Figure 5is a diagram showing a 3D bounding box according to an exemplary embodiment of the present disclosure;

[0041] Figure 6A is a diagram showing a far point according to an exemplary embodiment of the present disclosure;

[0042] Figure 6B is a diagram showing a far point according to an exemplary embodiment of the present disclosure;

[0043] Figure 6C is a diagram showing a far point according to an exemplary embodiment of the present disclosure;

[0044] Figure 7 is a flowchart showing a method for determining a vehicle mask according to an exemplary embodiment of the present disclosure;

[0045] Figure 8 is a block diagram showing an apparatus for generating a depth map according to an exemplary embodiment of the present disclosure; and

[0046] Figure 9 is a block diagram showing a computing system for performing a method of generating a depth map according to an exemplary embodiment of the present disclosure.

[0047] It can be understood that the drawings are not necessarily drawn to scale and present a somewhat simplified representation of various features illustrating the basic principles of the present disclosure. Specific design features of the present disclosure as included herein (including, for example, specific dimensions, orientations, positions, and shapes) will be determined in part by the specific intended application and the use environment.

[0048] In the drawings, throughout several views of the drawings, reference numerals refer to the same or equivalent parts of the present disclosure. Detailed Description of the Embodiments

[0049] Now, various embodiments of the present disclosure will be described in detail. Examples thereof are shown in the drawings and described below. Although the present disclosure will be described in conjunction with the exemplary embodiments of the present disclosure, it should be understood that this specification is not intended to limit the present disclosure to those exemplary embodiments. On the other hand, the present disclosure is intended to cover not only the exemplary embodiments of the present disclosure, but also various alternatives, modifications, equivalents, and other embodiments that may be included within the spirit and scope of the present disclosure as defined by the appended claims.

[0050] Hereinafter, various embodiments of the inventive concept will be described in detail with reference to the drawings, so that those skilled in the art can easily implement the inventive concept. However, the inventive concept is not limited to the embodiments set forth herein and can be variously modified in many different forms.

[0051] In describing the exemplary embodiments of the present specification, in response to the specific description of the related art being considered to obscure the subject matter of the exemplary embodiments of the present specification, the detailed description will be omitted. In the drawings, in order to make the present disclosure clear, parts not related to the description will not be shown.

[0052] It will be understood that in response to an element being referred to as "connected" or "coupled" to another element, it may be directly connected or indirectly connected to the other element. In addition, in response to some parts "including" or "having" some elements, unless explicitly described to the contrary, it means that other elements may be further included but not excluded.

[0053] Expressions such as "first" or "second" may represent their elements, regardless of their priority or importance, and may be used to distinguish one element from another, but are not limited to these components. Therefore, without departing from the scope of the present disclosure, the first component of various exemplary embodiments of the present disclosure may be referred to as the second component of another exemplary embodiment of the present disclosure. Similarly, the second component of various exemplary embodiments of the present disclosure may be referred to as the first component of another exemplary embodiment of the present disclosure.

[0054] In the exemplary embodiments of the present disclosure, the components distinguished from each other are only used to clearly describe the features, and do not mean that the components must be separated. That is, a plurality of components may be integrated to form a single hardware or software unit, or a single component may be distributed to form a plurality of hardware or software units. Therefore, such integrated or distributed embodiments are included within the scope of the present disclosure even if not separately mentioned.

[0055] In the exemplary embodiments of the present disclosure, the components described in each embodiment are not necessarily essential components, and some may be optional components. Therefore, exemplary embodiments constituted by a subset of the components described in the exemplary embodiments of the present disclosure are also included within the scope of the present disclosure. In addition, various exemplary embodiments including other components in addition to the components described in the various exemplary embodiments of the present disclosure are also included within the scope of the present disclosure.

[0056] In the exemplary embodiments of the present disclosure, for the sake of convenience of description, expressions of positional relationships used herein, such as up, down, left, right, etc. are described. When the drawings shown in this specification are viewed in the opposite way, the positional relationships described in this specification can be interpreted in the opposite way.

[0057] As used herein, each of the phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one or all possible combinations of the items enumerated together in the respective phrase.

[0058] Hereinafter, various exemplary embodiments of the present disclosure will be described in detail with reference to Figures 1 to 9 FIGs.

[0059] Figure 1 is a diagram illustrating a method of generating a depth map according to an exemplary embodiment of the present disclosure.

[0060] Referring to Figure 1 , the LiDAR 130 may emit laser pulses 131 and 132 and measure the time it takes for the laser pulses reflected by an object within the range to reach the receiver to estimate the distance and direction of the object. However, since the laser pulses 131 and 132 pass through an object through which light passes without being reflected, it may be difficult to detect an object such as glass using the LiDAR 130. For example, the laser pulse 132 may be reflected by the oncoming vehicle 120 around the LiDAR, but the laser pulse 131 may be emitted in the direction of the window of the oncoming vehicle 120 and penetrate the window. The laser pulse 131 passing through the window may reach a tree 110 farther than the oncoming vehicle 120 and be reflected. Therefore, an inconsistency may occur where a part of the tree 110 outside the window of the oncoming vehicle 120 is visible in the depth map generated by the LiDAR 130, but this part is not easily visible in the image using the camera. Due to the far points appearing on the window of the oncoming vehicle 120, the distance information of the oncoming vehicle 120 may not be constant, and thus, the depth map may not be suitable as a deep learning DB. In addition, due to the far points, the hidden point removal filter may not operate smoothly. Therefore, there is a need to provide a method of generating a depth map that is configured to solve the difference between the image generated by the camera and the depth map generated by the LiDAR 130. The method of generating a depth map according to an exemplary embodiment of the present disclosure may generate a depth map suitable for supervised learning by removing the far points that appear on the window of the oncoming vehicle 120 at a short distance.

[0061] Figure 2 is a flowchart illustrating a method of generating a depth map according to an exemplary embodiment of the present disclosure.

[0062] Figure 3A , Figure 3B and Figure 3Care diagrams respectively showing a LiDAR point cloud, a camera point cloud, and an image point cloud according to an exemplary embodiment of the present invention. In this case, the camera point cloud may represent a point cloud obtained by projecting the coordinates of points in the LiDAR point cloud onto a camera coordinate system. In addition, the image point cloud may refer to a point cloud in which the coordinates of points in the camera point cloud are projected onto an image coordinate system.

[0063] Figure 4 is a diagram showing an image point cloud according to an exemplary embodiment of the present disclosure. Hereinafter, reference will be made to Figure 3A , Figure 3B and Figure 3C and Figure 4 to describe Figure 2 .

[0064] Referring to Figure 2 , a method for generating a depth map according to an exemplary embodiment of the present disclosure may obtain a LiDAR point cloud generated by a LiDAR in operation S210. For example, the LiDAR point cloud generated based on the LiDAR may be similar to the picture shown in Figure 3A . Since the laser pulse of the LiDAR penetrates the vehicle window, the LiDAR point cloud does not include LiDAR points of the window of the oncoming vehicle within a short distance, but includes far points of objects farther than the window (e.g., the tree in Figure 1 ).

[0065] According to the method for generating a depth map, in operation S220, a 3D bounding box for at least a part of the LiDAR point cloud may be obtained.

[0066] For example, the method for generating a depth map may extract a close-range point cloud by removing point clouds beyond a predetermined distance from the LiDAR point cloud. In addition, the method for generating a depth map may include generating a 3D bounding box based on the close-range point cloud. The 3D bounding box may refer to a box-shaped label that distinguishes the categories of close-range point clouds in a 3D coordinate system. The 3D bounding box may be obtained by a network that receives the close-range point cloud and outputs a 3D bounding box for the oncoming vehicle. In response to the presence of a GT label for the 3D bounding box, the network may be omitted. The detailed format of the 3D bounding box will be described below.

[0067] As described above, the close-range point cloud may be a point cloud obtained by removing point clouds beyond a predetermined distance from the LiDAR point cloud. In the case where the window of the oncoming vehicle is at a predetermined distance or greater, since the window point cloud cannot be represented at a certain distance due to point cloud occlusion caused by the window frame or the like, for the close-range point cloud, it is only necessary to remove the window point cloud of the oncoming vehicle. The predetermined distance may be determined based on the focal length of the image point cloud, the distortion degree of the interval between points in the image point cloud, etc. The details of the image point cloud will be described below.

[0068] According to the method for generating a depth map, in operation S230, a first vehicle mask corresponding to the 3D bounding box can be generated from the image point cloud, where, in the image point cloud, the coordinates of the points of the LiDAR point cloud are projected onto the image coordinate system.

[0069] In a vehicle, the camera and the LiDAR can be located at different positions. For example, the LiDAR can be installed at a higher position than the camera (e.g., on the vehicle) to identify objects in the environment around the vehicle. Therefore, the LiDAR point cloud generated by the LiDAR and the image generated by the camera may not correspond to each other. Since the LiDAR point cloud is a point cloud generated based on the LiDAR as a coordinate reference point (origin), and the camera image is an image generated based on the camera as a coordinate reference point, in order to correspond the LiDAR point cloud with the camera image, it is necessary to convert the coordinate reference point of the LiDAR point cloud from the LiDAR to the camera. Among them, the point cloud whose coordinate reference point of the LiDAR point cloud is converted from the LiDAR to the camera can be called a camera point cloud. For example, Figure 3B the picture shown in Figure 3A shows the camera point cloud generated by converting the coordinate reference point of the LiDAR point cloud shown in

[0070] from the LiDAR to the camera.

[0071]

Equation 1

[0072]

[0073] The matrix including r 11 to r 33 and t 1 to t 3 as components corresponds to the extrinsic matrix. Among the components of the extrinsic matrix, r 11 to r 33 can correspond to the components of the rotation matrix, and t 1 to t 3 can correspond to the components of the translation matrix. In addition, X, Y, and Z can represent the coordinates of the points of the LiDAR point cloud. The coordinates (0, 0, 0) as the coordinate reference point of the LiDAR point cloud can indicate the position of the LiDAR. By multiplying the coordinates of the LiDAR point cloud and the extrinsic matrix, a camera point cloud with the position of the camera as the reference point can be generated.

[0074] In addition, the image point cloud can be generated by changing the focal length of the camera point cloud. For example, Figure 3B the camera point cloud in Figure 3CThe image point cloud in

[0075] Specifically, referring to Figure 4 , the coordinates of the points in the camera point cloud can be converted from the coordinates on the normalized image plane to the coordinates on the image plane. In the current case, the normalized image plane can refer to a virtual space with a focal length of 1. Therefore, the image point cloud can be generated by changing the focal length of the camera point cloud. In other words, the method for generating a depth map can include generating an image point cloud by converting the coordinate plane of the points in the camera point cloud from an image plane with a focal length normalized to 1 to an image plane with a focal length of an arbitrary value f.

[0076] In addition, the coordinates corresponding to the normalized image plane can be converted to the coordinates on the non-normalized image plane by multiplying by the intrinsic matrix. For example, as Figure 4 shown, in response to the coordinates of the points in the camera point cloud being (Xc, Yc, Zc), which can be referred to as the points (x’, y’) on the normalized image plane. In this case, x’ = Xc / Zc and y’ = Yc / Zc. In addition, in response to the points on the non-normalized image plane being (u, v), the following equation can be established.

[0077] [Equation 2]

[0078]

[0079] In this case, the matrix including f x 、f y 、o x and o y as components can be the intrinsic matrix. In the current case, f x and f y can represent the coordinates of the focal points, and o x and o y can represent the coordinates of the principal points.

[0080] The coordinate systems of the LiDAR point cloud and the camera point cloud can be 3D coordinate systems. The coordinates of the image point cloud can be normalized in the depth direction to correspond to 2D coordinates.

[0081] The gift wrapping algorithm or the alpha shape scheme can be used for the points included in the 3D bounding box in the image point cloud to generate the first vehicle mask. However, the scheme for generating the first vehicle shielding cover is not limited to the above methods.

[0082] The method for generating a depth map can include: in response to the value obtained by dividing the area of the second vehicle mask generated by the alpha shape scheme by the area of the third vehicle mask generated by the gift wrapping algorithm being less than a threshold, determining the second vehicle mask generated by the alpha shape scheme as the first vehicle mask. The specific algorithm for determining the first vehicle mask will be described later.

[0083] A method of generating a depth map may include generating a depth map in operation S240 by removing far points included in the first vehicle mask that are not included in the 3D bounding box. Among the points included in the first vehicle mask, the far points not included in the 3D bounding box may pass through the window of the other vehicle (near vehicle) and correspond to points generated by laser pulses reflected on an object (e.g., Figure 1 a tree) that is farther than the other vehicle. By removing the far points within the first vehicle mask, a more accurate depth map can be generated, from which the points generated through the window are removed. Examples related to far point removal will be described later.

[0084] Figure 5 is a diagram showing a 3D bounding box according to an exemplary embodiment of the present disclosure.

[0085] Referring to Figure 5 , the 3D bounding box can be generated in a free format. Specifically, in response to being able to determine whether a specific point is included in the 3D bounding box, the format of the 3D bounding box can be free.

[0086] For example, similar to the 3D bounding box 510, a 3D bounding box can be generated by eight vertices C1 to C8.

[0087] In addition, like the 3D bounding box 520, the 3D bounding box can be generated by the distance h 1 from the ground plane and h 2 and four vertices C1 to C4.

[0088] In addition, similar to the 3D bounding box 530, the 3D bounding box can be generated by the lengths (w, l) of the edges of the bottom portion of the bounding box and the midpoint (x, y, z) and the height h.

[0089] The above formats of the 3D bounding box are examples for helping understanding, and the 3D bounding box according to the exemplary embodiments of the present disclosure is not limited to the above.

[0090] Figure 6A is a diagram showing a far point according to an exemplary embodiment of the present disclosure. Figure 6A is a diagram showing an image point cloud, where the darker the color of the point, the farther the point is from the camera.

[0091] Referring to Figure 6A, it can be understood that the dark far point 610 appears on the front window of the oncoming vehicle 600 in the image point cloud. The far point 610 can be a point generated in response to a laser pulse of the LiDAR passing through the glass of the oncoming vehicle (close vehicle) 600 and reflecting on an object farther than the oncoming vehicle 600. In the case of a depth map including the far point 610, since the distance information of the oncoming vehicle 600 is not constant, it may not be suitable for deep learning DB. Therefore, it is necessary to remove the far point 610.

[0092] Figure 6B is a diagram showing a far point according to an exemplary embodiment of the present disclosure. Figure 6B is a diagram showing an image point cloud, where the darker the color of the point, the farther the point is from the camera.

[0093] The vehicle mask 620 can be a vehicle mask for the oncoming vehicle 600. Specifically, the vehicle mask 620 can be a vehicle mask corresponding to the 3D bounding box of the oncoming vehicle 600 in the image point cloud. Therefore, the far point 610 is included in the vehicle mask 620, but may not be included in the 3D bounding box for the oncoming vehicle 600 because the far point 610 is a point for an object farther than the vehicle. Therefore, a more accurate depth map can be generated by removing the far point 610 included in the vehicle mask 620 of the oncoming vehicle 600 but not included in the 3D bounding box of the oncoming vehicle 600.

[0094] Figure 6C is a diagram showing a far point according to an exemplary embodiment of the present disclosure. Figure 6C is a diagram showing an image point cloud, where the darker the color of the point, the farther the point is from the camera.

[0095] Figure 6C The shown image point cloud is Figure 6A and Figure 6B the image point cloud with the far point 610 removed as shown. Therefore, it can be understood that the points corresponding to the front window of the oncoming vehicle 600 are removed, as indicated by the region 630. Therefore, the distance information for the oncoming vehicle 600 can be constant, and the difference between the depth map generated by the LiDAR and the camera image can be eliminated.

[0096] Figure 7 is a flowchart showing a method for determining a vehicle mask according to an exemplary embodiment of the present disclosure. Figure 7 The flowchart of can be a flowchart of a method embodying Figure 2 the operation S230 of.

[0097] A method for generating a depth map according to an exemplary embodiment of the present disclosure may include generating a second vehicle mask by using an alpha shape scheme in operation S231.

[0098] In addition, the method of generating a depth map may include generating a third vehicle mask in operation S232 by using a gift wrapping algorithm.

[0099] In addition, the method of generating a depth map may include determining, in operation S233, whether a value obtained by dividing the area of the second vehicle mask by the area of the third vehicle mask is less than a threshold.

[0100] In addition, the method of generating a depth map may include: in operation S234, when the value obtained by dividing the area of the second vehicle mask by the area of the third vehicle mask is less than the threshold, determining the second vehicle mask as the first vehicle mask. In other words, the second vehicle mask generated using the alpha shape scheme may be determined as the first vehicle mask.

[0101] In addition, according to the method of generating a depth map, in operation S235, when the value obtained by dividing the area of the second vehicle mask by the area of the third vehicle mask is greater than or equal to the threshold, the third vehicle mask may be determined as the first vehicle mask. In other words, the third vehicle mask generated using the gift wrapping algorithm may be determined as the first vehicle mask.

[0102] Figure 8 FIG. is a block diagram of an apparatus for generating a depth map according to an exemplary embodiment of the present disclosure.

[0103] An apparatus 800 for generating a depth map according to an exemplary embodiment of the present disclosure may include a LiDAR 810, a camera 820, and a processor 830. Figure 8 The LiDAR 810 disclosed in Figure 1 may be the same as the LiDAR 130 disclosed in

[0104] The LiDAR 810 may be configured to generate LiDAR point clouds.

[0105] The LiDAR 810 may be disposed at a position higher than the camera 820 in the vehicle. In addition, the LiDAR 110 may be disposed on the vehicle, but the exemplary embodiments of the present disclosure are not limited thereto.

[0106] In addition, the camera 820 may be disposed on the vehicle, but the exemplary embodiments of the present disclosure are not limited thereto.

[0107] The processor 830 may be configured to generate a camera point cloud by changing a coordinate reference point of points of the LiDAR point cloud based on a positional difference between the LiDAR 810 and the camera 820. For example, the processor 830 may be configured to generate a camera point cloud by changing the coordinate reference point of points of the LiDAR point cloud from the LiDAR 810 to the camera 820. Additionally, the processor 830 may be configured to generate a camera point cloud by multiplying the coordinates of points of the LiDAR point cloud by an extrinsic matrix.

[0108] Additionally, the processor 830 may be configured to generate an image point cloud by changing a focal length of the camera point cloud. Specifically, the processor 830 may be configured to generate an image point cloud by converting a coordinate plane of points of the camera point cloud from an image plane in which the focal length is normalized to 1 to an image plane in which the focal length has an arbitrary value. For example, the processor 830 may be configured to generate an image point cloud by multiplying the coordinates of points in the camera point cloud by an intrinsic matrix.

[0109] Additionally, the processor 830 may be configured to obtain a 3D bounding box of the LiDAR point cloud. For example, the processor 830 may be configured to convert the LiDAR point cloud into a close-range point cloud by removing point clouds in the LiDAR point cloud that exceed a predetermined distance, and generate a 3D bounding box based on the close-range point cloud. Additionally, the processor 830 may be configured to determine the predetermined distance based on the focal length.

[0110] Additionally, the processor 830 may be configured to generate a first vehicle mask corresponding to the 3D bounding box in the image point cloud. For example, the processor 830 may be configured to generate the first vehicle mask by using a gift wrapping algorithm or an alpha shape scheme for points included in the 3D bounding box in the image point cloud. Specifically, when a value obtained by dividing an area of a second vehicle mask generated by the alpha shape scheme by an area of a third vehicle mask generated by the gift wrapping algorithm is less than a threshold, the processor 830 may be configured to determine the second vehicle mask generated by the alpha shape scheme as the first vehicle mask.

[0111] Additionally, the processor 830 may be configured to generate a depth map by removing far points that are not included in the 3D bounding box among points included in the first vehicle mask.

[0112] Figure 9 is a block diagram of a computing system for performing a method for determining fail-safe of camera image recognition according to various exemplary embodiments of the present disclosure.

[0113] Reference Figure 9, the method for determining the fail-safe of camera image recognition according to the above exemplary embodiments of the present disclosure can be implemented by a computing system 1000. The computing system 1000 may include at least one processor 1100, a memory 1300, a user interface input device 1400, a user interface output device 1500, a storage device 1600, and a network interface 1700 connected via a system bus 1200.

[0114] The processor 1100 may be a central processing unit (CPU) or a semiconductor device that processes instructions stored in the memory 1300 and / or the storage device 1600. The processor 1100 may correspond to Figure 8 the processor 830.

[0115] The memory 1300 and the storage device 1600 may include various types of volatile or non-volatile storage media. For example, the memory 1300 may include a read-only memory (ROM) 1310 and a random access memory (RAM) 1320.

[0116] Therefore, the processes of the methods or algorithms described in connection with the exemplary embodiments of the present disclosure may be directly implemented by hardware, software modules, or a combination thereof executed by the processor 1100. The software modules may reside in a storage medium (i.e., the memory 1300 and / or the storage device 1600), such as RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disk, solid state drive (SSD), removable disk, or CD-ROM. The exemplary storage medium is coupled to the processor 1100, and the processor 1100 can read information from the storage medium and can write information in the storage medium. In another method, the storage medium may be integrated with the processor 1100. The processor 1100 and the storage medium may reside in an application specific integrated circuit (ASIC). The ASIC may reside in a user terminal. In another method, the processor 1100 and the storage medium may reside as separate components in a user terminal.

[0117] According to an exemplary embodiment of the present disclosure, an accurate learning depth map for training a network for generating a depth map can be generated.

[0118] According to an exemplary embodiment of the present disclosure, an accurate depth map can be generated by removing far points from the windows of nearby opposing vehicles so that the far points are not projected onto the depth map.

[0119] The effects obtained through various embodiments of the present disclosure may not be limited to the above, and other effects will be clearly understood by those of ordinary skill in the art from the following disclosure.

[0120] In various exemplary embodiments of the present disclosure, each of the above operations may be performed by a control device, and the control device may be configured by a plurality of control devices or an integrated single control device.

[0121] In various exemplary embodiments of the present disclosure, the memory and the processor may be provided as one chip or as separate chips.

[0122] In different exemplary embodiments of the present disclosure, the scope of the present disclosure includes software or machine-executable commands (e.g., operating systems, applications, firmware, programs, etc.) for enabling the operations of methods according to different embodiments to be performed on a device or a computer, and non-transitory computer-readable media storing such software or commands and executable on the device or the computer.

[0123] In various exemplary embodiments of the present disclosure, the control device may be implemented in the form of hardware or software, or may be implemented in a combination of hardware and software.

[0124] In addition, terms such as "unit" and "module" included in the specification refer to units for processing at least one function or operation, which may be implemented by hardware, software, or a combination thereof.

[0125] In an exemplary embodiment of the present disclosure, a vehicle may be referred to based on the concept including various transportation means. In some cases, a vehicle may be interpreted as based on not only various land transportation means (such as cars, motorcycles, trucks, and buses) traveling on roads but also various transportation means such as airplanes, drones, ships, etc.

[0126] For the convenience of description and to accurately define the appended claims, referring to the positions of such features shown in the drawings, terms such as "upper", "lower", "inner", "outer", "upward", "downward", "up", "down", "front", "rear", "back", "inner", "outer", "inward", "outward", "inner", "outer", "inner", "outer", "forward", and "backward" are used to describe the features of the exemplary embodiments. It should be further understood that the term "connected" or its derivatives refer to direct connection and indirect connection.

[0127] The term "and / or" may include combinations of multiple related listed items or any of the multiple related listed items. For example, "A and / or B" includes all three cases, namely "A", "B", and "A and B".

[0128] In this specification, unless otherwise stated, singular expressions include plural expressions, unless the context clearly indicates otherwise.

[0129] In the exemplary embodiments of the present disclosure, it should be understood that terms such as "including" or "having" are intended to specify the presence of the features, numbers, steps, operations, elements, components, or combinations thereof described in the specification, and do not preclude the possibility of adding or the presence of one or more other features, numbers, steps, operations, elements, components, or combinations thereof.

[0130] According to an exemplary embodiment of the present invention, components may be combined with each other to be implemented as one, or some components may be omitted.

[0131] For purposes of illustration and description, the foregoing description of specific exemplary embodiments of the present disclosure has been presented. They are not intended to be exhaustive or to limit the present disclosure to the precise forms disclosed, and obviously many modifications and variations are possible in light of the above teachings. To illustrate certain principles of the present invention and its practical applications, exemplary embodiments have been selected and described so that others skilled in the art can make and utilize the various exemplary embodiments of the present disclosure and their various alternatives and modifications. The scope of the present disclosure is intended to be defined by the appended claims and their equivalents.

Claims

1. A method for generating a depth map, the method comprising: obtaining, by a processor, a lidar point cloud generated by a lidar operatively connected to the processor; obtaining, by the processor, a 3D bounding box for at least a portion of the lidar point cloud; generating, by the processor, a first vehicle mask corresponding to the 3D bounding box from an image point cloud obtained by projecting coordinates of points of the lidar point cloud into an image coordinate system; as well as The depth map is generated by the processor by removing distant points that are not included in the 3D bounding box from among the points included in the first vehicle mask.

2. The method according to claim 1, wherein: Obtaining the 3D bounding box includes: Extracting a close-range point cloud by removing point clouds beyond a predetermined distance from the lidar point cloud; and The 3D bounding box is obtained based on the close-range point cloud.

3. The method according to claim 2, wherein: Extracting the close-range point cloud includes determining the predetermined distance based on at least one of a degree of distortion of intervals between points of the image point cloud and a focal length of the image point cloud.

4. The method according to claim 1, wherein: Generating the first vehicle mask includes generating the first vehicle mask by using a gift wrapping algorithm or an alpha shape scheme for points in the image point cloud that are contained in the 3D bounding box.

5. The method according to claim 4, wherein: Generating the first vehicle mask includes determining the second vehicle mask generated by the alpha shape scheme as the first vehicle mask in response to a value obtained by dividing an area of ​​the second vehicle mask generated by the alpha shape scheme by an area of ​​a third vehicle mask generated by the gift wrapping algorithm being less than a threshold.

6. The method according to claim 1, wherein: Generating the first vehicle mask includes: Generate a camera point cloud by projecting the coordinates of the points of the laser radar point cloud to a camera coordinate system by multiplying the coordinates of the points of the laser radar point cloud by an external parameter matrix; and The image point cloud is generated by changing the focal length of the camera point cloud.

7. The method according to claim 1, wherein: Generating the first vehicle mask includes generating the image point cloud by converting a coordinate plane of points of the camera point cloud obtained by projecting coordinates of points of the lidar point cloud onto a camera coordinate system from an image plane whose focal length is normalized to 1 to an image plane whose focal length has an arbitrary value.

8. The method according to claim 7, wherein: Generating the image point cloud includes: generating the image point cloud by multiplying the coordinates of the points of the camera point cloud by an intrinsic parameter matrix.

9. A device for generating a depth map, the device comprising: camera; A lidar configured to generate a lidar point cloud; as well as A processor is operatively connected to the camera and the lidar, and the processor is configured to obtain a 3D bounding box for at least a portion of the lidar point cloud, generate a first vehicle mask corresponding to the 3D bounding box from an image point cloud in which coordinates of points of the lidar point cloud are projected into an image coordinate system, and generate the depth map by removing far points that are not included in the 3D bounding box from points included in the first vehicle mask.

10. The device according to claim 9, wherein: The processor is further configured to extract a close-range point cloud by removing point clouds exceeding a predetermined distance from the lidar point cloud, and obtain the 3D bounding box based on the close-range point cloud.

11. The device according to claim 10, wherein: The processor is further configured to determine the predetermined distance based on at least one of a degree of distortion of intervals between points of the image point cloud and a focal length of the image point cloud.

12. The device according to claim 9, wherein: The processor is further configured to generate the first vehicle mask by using a gift wrapping algorithm or an alpha shape scheme for points in the image point cloud that are contained in the 3D bounding box.

13. An apparatus according to claim 12, wherein the processor is further configured to: in response to a value obtained by dividing the area of ​​the second vehicle mask generated by the alpha shape scheme by the area of ​​the third vehicle mask generated by the gift wrapping algorithm being less than a threshold, determine the second vehicle mask generated by the alpha shape scheme as the first vehicle mask.

14. The device according to claim 9, wherein: The processor is further configured to generate a camera point cloud by converting coordinate reference points of points of the lidar point cloud from the lidar to the camera.

15. The device according to claim 14, wherein: The processor is further configured to generate the camera point cloud by multiplying the coordinates of the points of the lidar point cloud by an external parameter matrix.

16. The device according to claim 9, wherein: The processor is further configured to generate the image point cloud by converting a coordinate plane of points of the camera point cloud obtained by projecting the coordinates of points of the lidar point cloud onto a camera coordinate system from an image plane whose focal length is normalized to 1 to an image plane whose focal length has an arbitrary value.

17. The device according to claim 16, wherein: The processor is further configured to generate the image point cloud by multiplying coordinates of points of the camera point cloud by an intrinsic parameter matrix.

18. The device according to claim 9, wherein: The laser radar is arranged at a higher position in the vehicle than the camera.

19. The device according to claim 9, wherein: The laser radar is arranged on the vehicle.

20. The device according to claim 9, wherein: The camera is arranged on the vehicle.

Images