Apparatus and method for generating depth maps using volumetric features

Abstract

To provide an apparatus and method for generating a depth map using volumetric features.SOLUTION: The method includes encoding and post-processing a base image included in the surround view image to generate a single feature map of the base image, encoding the single feature map with depth information and then projecting the encoding result into a three dimensional space to generate a volumetric feature, and decoding the volumetric feature using a depth decoder to generate a depth map of the surround view image.SELECTED DRAWING: Figure 11

Description

[Technical Field]

[0001] The present invention relates to an apparatus and method for generating a depth map using volumetric features. [Background technology]

[0002] There has been active research into methods for estimating 3D depth information, which is used in various fields such as robot vision, human computer interface, intelligent visual surveillance, 3D image acquisition, etc. In particular, in autonomous driving systems, research on highly accurate depth estimation is required to control the vehicle by recognizing and judging various driving environments, including the distance between the vehicle and objects detected in collected image data.

[0003] There are two methods for estimating depth information: one that uses signals such as infrared, ultrasound, laser, and light scattering characteristics, and one that analyzes images. Methods that use infrared or ultrasound to estimate distance have the disadvantage that the reflected signal can be affected by the state of the object. Using signals such as laser is highly accurate, but has the disadvantage of requiring expensive additional equipment.

[0004] On the other hand, estimating depth information using a camera does not require any additional equipment. Depth information can also be generated based on disparity calculations using a stereo camera. However, accurate depth estimation requires precise calibration of the two cameras, and disparity calculations take a lot of time. Therefore, there is a growing need for research into methods for accurately estimating depth information using a monocular camera.

[0005] The above-mentioned background art is technical information that the inventor possessed in order to derive the present invention or that he acquired in the process of deriving the present invention, and is not necessarily publicly known art that was disclosed to the general public prior to the filing of the present invention. Summary of the Invention [Problem to be solved by the invention]

[0006] The present invention provides an apparatus and method for generating a depth map using volumetric features. The problems to be solved by the present invention are not limited to those described above, and other problems and advantages of the present invention that are not described will be understood from the following description and will become more clearly understood by embodiments of the present invention. It will also be understood that the problems to be solved and advantages of the present invention can be achieved by the means and combinations thereof set forth in the claims. [Means for solving the problem]

[0007] As a technical means for solving the above-mentioned technical problem, a first aspect of the present disclosure can provide a method for generating a depth map using volumetric features, the method including the steps of acquiring a surround view image using one or more monocular cameras, encoding basic images included in the surround view image to generate multi-scale image features, resizing and concatenating each of the multi-scale image features based on a predetermined resolution to generate a single feature map of the basic image, encoding the single feature map together with depth information and then projecting the encoding result into three-dimensional space to generate volumetric features, and decoding the volumetric features using a depth decoder to generate a depth map of the surround view image.

[0008] A second aspect of the present disclosure can provide an apparatus for generating a depth map using volumetric features, the apparatus including: a memory having at least one program stored therein; and a processor for executing the at least one program, wherein the processor acquires a surround-view image using one or more monocular cameras; encodes base images included in the surround-view image to generate multi-scale image features; resizes and concatenates each of the multi-scale image features based on a predetermined resolution to generate a single feature map for the base image; encodes the single feature map together with depth information; and then projects the encoded result into three-dimensional space to generate volumetric features; and decodes the volumetric features using a depth decoder to generate a depth map for the surround-view image.

[0009] A third aspect of the present disclosure can provide a method for acquiring pose information of a monocular camera using volumetric features, the method including the steps of acquiring a surround view image using one or more monocular cameras, encoding basic images included in the surround view image to generate multi-scale image features, resizing and concatenating each of the multi-scale image features based on a predetermined resolution to generate a single feature map of the basic image, encoding the single feature map and then projecting the encoding result into three-dimensional space to generate volumetric features, and decoding the volumetric features using a pose decoder to collectively acquire pose information of the monocular camera at a next point in time.

[0010] Further, a method can be provided in which the step of collectively acquiring pose information of the monocular cameras includes a step of acquiring canonical camera motion indicating the amount of change between the camera pose at the current time point and the camera pose at the next time point in the reference monocular camera by decoding the volumetric features using a pose decoder, and a step of collectively acquiring pose information of the other monocular cameras at the next time point based on the external parameter relationship between the reference monocular camera and the other monocular cameras and the canonical camera motion.

[0011] Further, a method can be provided in which the step of obtaining the standard camera motion includes the steps of flattening the volumetric features into bird's eye view (BEV) features and decoding the bird's eye view features using the pose decoder.

[0012] Additionally, the method may further include generating a depth map for the surround-view image by decoding the volumetric features using a depth decoder.

[0013] Furthermore, the method may further include training the neural network using the surround-view images as input data for the neural network and using the depth map and the standard camera motion as output data.

[0014] Furthermore, a method can be provided in which the neural network is trained to minimize image reconstruction loss and depth synthesis loss.

[0015] Furthermore, the method can be provided, wherein the image reconstruction loss includes temporal loss, spatial loss, and spatio-temporal loss.

[0016] Further, a method can be provided in which the depth compositing loss includes a depth consistency loss.

[0017] A fourth aspect of the present disclosure provides an apparatus for generating a depth map using a monocular camera, the apparatus including: a memory having at least one program stored therein; and a processor for executing the at least one program, wherein the processor acquires a surround-view image using one or more monocular cameras; encodes base images included in the surround-view image to generate multi-scale image features; resizes and concatenates each of the multi-scale image features based on a predetermined resolution to generate a single feature map of the base image; encodes the single feature map together with depth information; and then projects the encoded result into three-dimensional space to generate volumetric features; and collectively acquires pose information of the monocular camera at a next point in time based on the volumetric features.

[0018] A fifth aspect of the present disclosure can provide a computer-readable recording medium having recorded thereon a program for causing a computer to execute the method according to the first or third aspect.

[0019] In addition, other methods and systems for realizing the present invention, and computer-readable recording media storing computer programs for executing the methods can also be provided.

[0020] Further aspects, features, and advantages will become apparent from the accompanying drawings, the claims, and the following detailed description of the invention. [Effects of the Invention]

[0021] According to the above-described means for solving the problem of the present disclosure, the present disclosure can generate a smooth depth map even when the camera pose is arbitrarily modulated rather than using an actually captured image. [Brief explanation of the drawings]

[0022] [Figure 1] FIG. 1 is a diagram illustrating an autonomous driving method according to an embodiment. [Figure 2] FIG. 1 is a diagram illustrating an autonomous driving method according to an embodiment. [Figure 3] FIG. 1 is a diagram illustrating an autonomous driving method according to an embodiment. [Figure 4A] 1 is an exemplary diagram illustrating a method for acquiring a surround view image according to an embodiment. [Figure 4B] 1 is an exemplary diagram illustrating a method for acquiring a surround view image according to an embodiment. [Figure 5] 1 is a diagram illustrating a method for generating a single feature map for a surround-view image using an image encoder according to an embodiment; [Figure 6A] FIG. 1 is an exemplary diagram illustrating a method for generating volumetric features using a single feature map according to an embodiment. [Figure 6B] FIG. 1 is an exemplary diagram illustrating a method for generating volumetric features using a single feature map according to an embodiment. [Figure 6C] FIG. 1 is an exemplary diagram illustrating a method for generating volumetric features using a single feature map according to an embodiment. [Figure 7] 1 is an exemplary diagram illustrating a method for generating a depth map of a surround view image according to an embodiment; [Figure 8A]10A and 10B are exemplary diagrams illustrating a method for acquiring a depth map according to an arbitrary rotational view according to an embodiment; [Figure 8B] 10A and 10B are exemplary diagrams illustrating a method for acquiring a depth map according to an arbitrary rotational view according to an embodiment; [Figure 8C] 10A and 10B are exemplary diagrams illustrating a method for acquiring a depth map according to an arbitrary rotational view according to an embodiment; [Figure 8D] 10A and 10B are exemplary diagrams illustrating a method for acquiring a depth map according to an arbitrary rotational view according to an embodiment; [Figure 8E] 10A and 10B are exemplary diagrams illustrating a method for acquiring a depth map according to an arbitrary rotational view according to an embodiment; [Figure 9] 1 is an exemplary diagram illustrating a method for predicting standard motion according to an embodiment; [Figure 10] FIG. 1 is an exemplary diagram illustrating a neural network using self-supervised learning according to an embodiment. [Figure 11] 1 is a flowchart illustrating a method for generating a depth map using a monocular camera according to an embodiment. [Figure 12] 1 is a block diagram of a depth map generator according to one embodiment; DETAILED DESCRIPTION OF THE INVENTION

[0023] The advantages and features of the present invention, as well as methods for achieving them, will become more apparent by reference to the embodiments described in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments presented below, and can be realized in various different forms, and it should be understood that the present invention includes all modifications, equivalents, and alternatives within the spirit and technical scope of the present invention. The embodiments presented below are provided to provide a complete disclosure of the present invention and to enable those skilled in the art to fully understand the scope of the invention. In describing the present invention, if a detailed description of related publicly known technology is considered to obscure the gist of the present invention, such a detailed description will be omitted.

[0024] The terms used in this application are merely used to describe specific embodiments and are not intended to limit the present invention. The singular expressions include the plural expressions unless the context clearly dictates otherwise. In this application, the terms "comprise" and "have" and the like specify the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, and should be understood not to preclude the presence or additional possibility of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0025] Some embodiments of the present disclosure may be illustrated using functional block configurations and various processing steps. Some or all of these functional blocks may be implemented using various hardware and / or software components that perform specific functions. For example, the functional blocks of the present disclosure may be implemented using one or more microprocessors or circuitry for a given function. For example, the functional blocks of the present disclosure may be implemented using various programming or scripting languages. The functional blocks may be implemented using algorithms executed by one or more processors. Furthermore, the present disclosure may employ conventional techniques for electronic configuration, signal processing, and / or data processing. Terms such as "mechanism," "element," "means," and "component" may be used broadly and are not limited to mechanical and physical components.

[0026] It should be noted that the connecting lines or connecting members between components shown in the drawings are merely exemplary of functional and / or physical connections or circuit connections, and that in an actual device, connections between components may be represented by various alternative or additional functional, physical, or circuit connections.

[0027] Hereinafter, the term "vehicle" may refer to any type of transport means having a motor and used to move people or goods, such as a car, bus, bike, scooter or truck.

[0028] The present disclosure will now be described in detail with reference to the accompanying drawings.

[0029] 1 to 3 are diagrams for explaining an autonomous driving method according to an embodiment.

[0030] Referring to FIG. 1 , an autonomous driving device according to an embodiment of the present invention may be mounted on a vehicle to realize an autonomous driving vehicle 10. The autonomous driving device mounted on the autonomous driving vehicle 10 may include various sensors for collecting information about surrounding conditions. As an example, the autonomous driving device may detect the movement of a leading vehicle 20 traveling ahead using an image sensor and / or an event sensor mounted on the front of the autonomous driving vehicle 10. The autonomous driving device may further include sensors for detecting other vehicles 30 traveling on adjacent lanes as well as in front of the autonomous driving vehicle 10, pedestrians around the autonomous driving vehicle 10, and the like.

[0031] At least one of the sensors for collecting situational information about the autonomous vehicle may have a predetermined field of view (FoV), as shown in Fig. 1. As an example, if a sensor mounted on the front of the autonomous vehicle 10 has an FoV as shown in Fig. 1, information detected at the center of the sensor may have a relatively high importance. This is because the information detected at the center of the sensor contains most of the information corresponding to the movement of the leading vehicle 20.

[0032] The autonomous driving device processes information collected by sensors of the autonomous vehicle 10 in real time to control the movement of the autonomous vehicle 10, while at least a portion of the information collected by the sensors can be stored in a memory device.

[0033] 2, the autonomous driving device 40 may include a sensor unit 41, a processor 46, a memory system 47, a vehicle control module 48, etc. The sensor unit 41 includes a plurality of sensors 42 to 45, which may include an image sensor, an event sensor, an illuminance sensor, a GPS device, an acceleration sensor, etc.

[0034] The data collected by the sensors 42 to 45 may be transmitted to the processor 46. The processor 46 stores the data collected by the sensors 42 to 45 in a memory system 47 and controls a body control module 48 to determine the movement of the vehicle based on the data collected by the sensors 42 to 45. The memory system 47 may include two or more memory devices and a system controller for controlling the memory devices. Each of the memory devices may be provided as a single semiconductor chip.

[0035] In addition to the system controller of memory system 47, each of the memory devices included in memory system 47 may include a memory controller, and the memory controller may include an artificial intelligence (AI) calculation circuit such as a neural network. The memory controller can generate calculation data by applying a predetermined weight to data received from sensors 42 to 45 or processor 46, and store the calculation data in a memory chip.

[0036] 3 is a diagram showing an example of image data acquired by a sensor of an autonomous vehicle equipped with an autonomous driving device. Referring to FIG. 3, image data 50 may be data acquired by a sensor attached to the front of the autonomous vehicle. Therefore, image data 50 may include a front portion 51 of the autonomous vehicle, a preceding vehicle 52 on the same road as the autonomous vehicle, vehicles 53 traveling around the autonomous vehicle, a background 54, and lanes 55 and 56.

[0037] 3, the data in the area showing the front part 51 of the autonomous vehicle and the background 54 may be data that is unlikely to affect the operation of the autonomous vehicle. In other words, the front part 51 of the autonomous vehicle and the background 54 may be considered to be data with a relatively low level of importance.

[0038] On the other hand, the distance to the preceding vehicle 52, the lane-changing behavior of the moving vehicle 53, and the like can be very important factors in the safe operation of the autonomous vehicle. Therefore, in the image data 50, data of an area that includes the preceding vehicle 52, the moving vehicle 53, and the like can be relatively important in the operation of the autonomous vehicle.

[0039] The memory device of the autonomous driving device can store image data 50 received from the sensor by assigning different weights to different areas. For example, a high weight may be assigned to data of an area including a leading vehicle 52 and a moving vehicle 53, and a low weight may be assigned to data of an area showing the front part 51 of the autonomous driving vehicle and the background 54.

[0040] 4A and 4B are exemplary diagrams illustrating a method for acquiring a surround view image according to an embodiment.

[0041] A device that generates a depth map (hereinafter referred to as a "depth map generating device") can acquire images of a road on which a vehicle is traveling using one or more monocular cameras.

[0042] The one or more cameras may be mounted on a vehicle and oriented in different directions to capture omnidirectional images of the vehicle, each of the one or more cameras having intrinsic parameters and extrinsic parameters.

[0043] The intrinsic parameters may include a focal length, a principal point, a skew coefficient, etc. The extrinsic parameters are parameters that describe the transformation relationship between the camera coordinate system and the world coordinate system, and may include rotation and translation transformation information between the two coordinate systems.

[0044] Referring to FIG. 4A, six cameras may be installed on vehicle 400, each of which may be configured to capture the front, rear, front-right, rear-right, front-left, and rear-left of vehicle 400.

[0045] The images acquired by the cameras installed in the vehicle 400 are I i For example, the cameras installed in FIG. 4A may be designated I1, I2, . . . I6.

[0046] Comparing FIG. 4A and FIG. 4B, FIG. 4A shows six images taken by each camera at time t, and each image is TIFF2026010196000002.tif518. Also, FIG. 4B shows six images taken by each camera at time t+1 after time t, and each image is This can be shown in TIFF2026010196000003.tif530.

[0047] Hereinafter, an image set made up of a plurality of basic images captured at a given point in time by one or more monocular cameras installed in the vehicle 400 as described above will be referred to as a "surround view image."

[0048] FIG. 5 is a diagram illustrating a method for generating a single feature map for a surround-view image using an image encoder according to an embodiment.

[0049] The depth map generator may include an image encoder 500. The image encoder 500 can use a plurality of base images included in a surround-view image as input data.

[0050] The image encoder 500 can generate multi-scale image features 510 of the base image and can resize the multi-scale image features 510 based on a predetermined resolution and then concatenate them to generate a single feature map 520 of the base image.

[0051] 5, the image encoder 500 may use a convolution neural network (CNN) to reduce the resolution of a base image by half to generate multi-scale image features 510. For example, the image encoder 500 may use ResNet18 to generate multi-scale image features 510 with resolutions of 1 / 2, 1 / 4, 1 / 8, and 1 / 16 of the base image (H×W resolution). Alternatively, the image encoder 500 may set "H / 8×W / 8" as a reference resolution, up-sample the multi-scale image features 510, resize them based on "H / 8×W / 8", and then concatenate the multi-scale image features 510 to generate a single feature map 520.

[0052] That is, the depth map generator uses the image encoder 500 to generate a plurality of base images I included in a surround-view image in the above-described manner. i Each single feature map F i For example, the depth map generator may generate a single feature map F1, F2, ... F6 for each of multiple base images I1, I2, ... I6 included in the surround-view image.

[0053] 6A to 6C are exemplary diagrams illustrating a method for generating volumetric features using a single feature map according to an embodiment.

[0054] A depth map generator can generate volumetric features by encoding a single feature map with depth information and then projecting the encoding result into three-dimensional space.

[0055] Referring to FIG. 6A, a basic image I i A single feature map F generated from i For p, the depth map generator selects a set of voxels along a ray from pixel p to the center of a given camera and assigns the pixels to a single feature map F i (p) can be set, and the depth map generator can achieve this with a mapping function such as backward-warping using the intrinsic and extrinsic parameters of a given camera.

[0056] A single feature map F for each pixel p i (p)610 has F i Since the high-level information along the ray of (p)610 pixels is included, referring to FIG. 6B, the depth map generator can i (p) 610 can be concatenated with depth information of voxel coordinates 620 and encoded. The depth map generator can also extract local 3D features of each voxel coordinate (x, y, z) by projecting the encoded results into 3D space.

[0057] On the other hand, due to spatial overlap in the views of multiple cameras capturing surround view images, some voxels in the three-dimensional volumetric space may be associated with multiple single feature maps.

[0058] Referring to FIG. 6B, it can be seen that some voxels in the three-dimensional volumetric space overlap, while other voxels do not.

[0059] The depth map generator may generate volumetric features by inputting the local 3D features to a volumetric encoder. In one embodiment, the depth map generator may assign voxels to different multi-layer perceptrons (MLPs) depending on whether overlap occurs for each voxel.

[0060] Referring to FIG. 6C, it can be seen that after the local 3D features are input into the volumetric encoder, voxels where overlap occurs and voxels where no overlap occurs are assigned to different MLPs.

[0061] The depth map generator may fuse the results of voxels assigned to different MLPs and encoded, to generate volumetric feature 640. In the present disclosure, voxels are assigned to different MLPs depending on whether overlap occurs for each voxel, and encoded, and then the results are fused to generate volumetric feature 640, so that each voxel of volumetric feature 640 can encode a high-dimensional feature related to 3D geometry.

[0062] FIG. 7 is an exemplary diagram illustrating a method for generating a depth map of a surround-view image according to an embodiment.

[0063] The depth map generator can convert the volumetric features generated in Fig. 6C into projected image features with a predetermined resolution based on the internal and external parameters of one or more cameras installed in the vehicle. Here, the internal parameters may include a focal length, a principal point, a skew coefficient, etc. The external parameters are parameters that describe the transformation relationship between the camera coordinate system and the world coordinate system, and may include rotation and translation transformation information between the two coordinate systems.

[0064] The depth map generator can also generate a depth map for the surround-view image by decoding the projected image features using a depth decoder.

[0065] Specifically, referring to FIG. 7, the depth map generator calculates a camera-specific intrinsic parameter K i and the external parameter E i to obtain a projected image feature 730 having a predetermined resolution of "H / 8 x W / 8". For example, the depth map generator may generate a projected image feature 730 by applying camera-specific extrinsic parameters E1, E2, ..., E6 to the volumetric feature 710V. TIFF2026010196000005.tif621 can be generated.

[0066] The depth map generator uses a depth decoder 740 to generate projected image features 730 with resolution "H / 8 x W / 8". By decoding TIFF2026010196000006.tif64, a depth map with a resolution of H×W can be generated. For example, the depth map generator may generate a projected image feature 730 with a resolution of H / 8×W / 8. TIFF2026010196000007.tif621 can be upsampled to produce depth maps D1, D2, ... D6.

[0067] The depth decoder 740 may consist of, but is not limited to, three convolutional layers for upsampling and one convolutional layer for depth output.

[0068] In the present disclosure, the six camera images I1, I2, ... I6 captured as surround view images in Figure 4a can be converted into depth maps D1, D2, ... D6 in Figure 7 using the method described above in Figures 5 to 7.

[0069] 8A to 8E are exemplary diagrams illustrating a method for acquiring a depth map according to an arbitrary rotational view according to an embodiment.

[0070] As described above in FIG. 7, the depth map generator can convert volumetric features into projected image features with a predetermined resolution based on the extrinsic parameters of one or more cameras installed on the vehicle.

[0071] Without loss of generality in the process, the present disclosure can transform volumetric features into projected image features using a projection matrix from a canonical coordinate system for a desired view, instead of maintaining the camera pose as is. In particular, the present disclosure can obtain a new camera pose by modulating the product of canonical motion and the desired camera pose. That is, the depth map generator can determine a projection matrix for transforming volumetric features into projected image features with a predetermined resolution, and modulate the depth map using the projection matrix that matches the desired view.

[0072] Also, since spatial overlaps in camera views share common volumetric features, consistent metric scaled depth maps can be predicted in this disclosure.

[0073] The method of the present disclosure can generate a smooth depth map by varying not only the focal length but also the yaw, roll, and pitch angles, i.e., the present disclosure can generate a smooth depth map even when the camera pose is arbitrarily modulated instead of the actual captured image.

[0074] FIG. 8A shows images acquired from three cameras installed on a vehicle, which respectively capture the front-left, front, and front-right of the vehicle.

[0075] Figure 8B is a depth map obtained by modulating the camera focus at magnifications of x1, x1 / 2, and x1 / 3 using a method according to the present disclosure, Figure 8C is a view-interpolation depth map obtained by modulating the yaw angle based on the left front and right front images using a method according to the present disclosure, Figure 8D is a depth map obtained by modulating the roll angle of the front image by ±10° using a method according to the present disclosure, and Figure 8E is a depth map obtained by modulating the pitch angle of the front image by ±3° using a method according to the present disclosure.

[0076] FIG. 9 is an example diagram illustrating a method for predicting standard motion according to an embodiment.

[0077] The depth map generator can flatten the volumetric features generated in FIG. 6C into bird's-eye view (BEV) features. The depth map generator can also obtain canonical camera motion, which indicates the amount of change between the camera pose at a current time point and the camera pose at a next time point, of the reference monocular camera by decoding the bird's-eye view features using a pose decoder. Here, "motion" refers to the amount of change between the pose at a current time point and the pose at a next time point of the reference monocular camera. The depth map generator can collectively obtain pose information of the other monocular cameras at a next time point based on the external parameter relationship between the reference monocular camera and the other monocular cameras and the canonical camera motion.

[0078] Specifically, referring to FIG. 9, the depth map generator includes a volumetric feature 910 Collapse the Z axis of TIFF2026010196000008.tif728 by the channel dimension C (i.e., the 3D tensor TIFF2026010196000009.tif732) and apply 2D convolution to the flattened bird's-eye view feature 920F BEV (i.e., TIFF2026010196000010.tif832) can be generated.

[0079] The depth map generator also uses a pose decoder 930 to generate standard camera motion. You can get TIFF2026010196000011.tif513. Standard camera motion TIFF2026010196000012.tif513 indicates the amount of change between the camera pose of a reference monocular camera (e.g., a front monocular camera) at the current time point t and the camera pose at the next time point t+1.

[0080] The depth map generating device can collectively acquire pose information of the other monocular cameras at the next time point based on the external parameter relationship between the reference monocular camera and the other monocular cameras and the standard camera motion, which is expressed as in the following Equation 1. In the following Equation 1, E1 means the external parameters of the reference monocular camera.

[0081]

number

[0082] Referring to FIG. 9, when there are N monocular cameras installed on a vehicle, the depth map generating device calculates pose information of the other monocular cameras other than the reference monocular camera at the next time point t+1 according to Equation 1. TIFF2026010196000014.tif545 can be obtained in bulk.

[0083] On the other hand, in addition to the forward camera, a monocular camera oriented in another direction can also be set as the reference monocular camera.

[0084] FIG. 10 is an exemplary diagram illustrating a neural network using self-supervised learning according to an embodiment.

[0085] The depth map generator can train a neural network using self-supervised learning, using surround-view images as input data for the neural network and depth maps and standard camera motion as output data.

[0086] In one embodiment, the neural network calculates the image reconstruction loss according to Equation 2: TIFF2026010196000015.tif68 and depth synthesis loss It can be learned in a direction that minimizes TIFF2026010196000016.tif611.

[0087]

number

[0088]

number

[0089] In addition, the depth synthesis loss is calculated as the depth consistency loss L cоns The depth consistency loss L cоns denotes the penalty for the depth difference between the synthesized depth at the new view and the known depth at each camera view i.

[0090]

number

[0091] FIG. 11 is a flowchart illustrating a method for generating a depth map using a monocular camera according to one embodiment.

[0092] Referring to FIG. 11, in step 1110, a processor may acquire surround view images with one or more monocular cameras.

[0093] In step 1120, the processor may encode the base images included in the surround-view image to generate multi-scale image features.

[0094] In step 1130, the processor may resize and concatenate each of the multi-scale image features relative to a predetermined resolution to generate a single feature map of the base image.

[0095] In step 1140, the processor may generate volumetric features by encoding the single feature map with depth information and then projecting the encoding result into three-dimensional space.

[0096] In one embodiment, the processor can extract local 3D features in a 3D volumetric space by concatenating and encoding a single feature map with the depth information, and then projecting the encoded result into the 3D space, and the processor can generate the volumetric features using the local 3D features.

[0097] In one embodiment, the processor may input the local 3D features to a volumetric encoder to generate volumetric features.

[0098] In one embodiment, the volumetric encoder may be configured with multiple multi-layer perceptrons. The processor may assign voxels to different multi-layer perceptrons depending on whether overlap occurs for each voxel in the 3D volumetric space. The processor may fuse encoding results from different multi-layer perceptrons to generate volumetric features.

[0099] In step 1150, the processor may generate a depth map for the surround view image by decoding the volumetric features using a depth decoder.

[0100] In one embodiment, the processor can convert volumetric features into projected image features having the predetermined resolution based on intrinsic and extrinsic parameters of one or more monocular cameras, and the processor can generate a depth map of the surround-view image by decoding the projected image features using a depth decoder.

[0101] In one embodiment, the processor can determine a projection matrix for transforming the volumetric features into projected image features having a predetermined resolution, and the processor can modulate the depth map with the projection matrix to match the desired view.

[0102] In one embodiment, the processor can flatten volumetric features into bird's-eye view (BEV) features. The processor can obtain canonical camera motion indicating the amount of change between the camera pose of the reference monocular camera at a current time point and the camera pose at a next time point by decoding the bird's-eye view features using a pose decoder. The processor can collectively obtain pose information of the other monocular cameras at a next time point based on the external parameter relationship between the reference monocular camera and the other monocular cameras and the canonical camera motion.

[0103] In one embodiment, the processor can train a neural network using the surround-view images as input data to the neural network and the depth map and standard camera motion as output data.

[0104] The neural network can be trained to minimize image reconstruction loss and depth synthesis loss. Image reconstruction loss can include temporal loss, spatial loss, and spatio-temporal loss. Depth synthesis loss can include depth consistency loss.

[0105] FIG. 12 is a block diagram of a depth map generating device according to one embodiment.

[0106] 12, a depth map generating device 1200 may include a communication unit 1210, a processor 1220, and a DB 1230. Only components related to the embodiment are shown in the depth map generating device 1200 of Fig. 12. Therefore, a person skilled in the art would understand that the depth map generating device 1200 may further include other general-purpose components in addition to the components shown in Fig. 12.

[0107] The communication unit 1210 may include one or more components that enable wired / wireless communication with an external server or device. For example, the communication unit 1210 may include at least one of a short-range communication unit (not shown), a mobile communication unit (not shown), and a broadcast receiving unit (not shown).

[0108] The DB 1230 is hardware that stores various data processed within the depth map generating device 1200, and can store programs for processing and controlling the processor 1220.

[0109] DB1230 may include random access memory (RAM), such as dynamic random access memory (DRAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM, Blu-ray or other optical disc storage, hard disk drive (HDD), solid state drive (SSD), or flash memory.

[0110] The processor 1220 controls the overall operation of the depth map generating device 1200. For example, the processor 1220 may execute a program stored in the DB 1230 to overall control an input unit (not shown), a display (not shown), a communication unit 1210, the DB 1230, etc. The processor 1220 may control the operation of the depth map generating device 1200 by executing a program stored in the DB 1230.

[0111] The processor 1220 can control at least a part of the operations of the depth map generation device 1200 described above in Figures 1 to 11. The depth map generation device 1200 and the autonomous driving device 40 may be the same device, and at least a part of the operations performed by each device may be the same.

[0112] The processor 1220 may be implemented using at least one of application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, and other electrical units for performing functions.

[0113] In one embodiment, the depth map generating device 1200 may be a mobile electronic device. For example, the depth map generating device 1200 may be implemented in a smartphone, tablet PC, PC, smart TV, personal digital assistant (PDA), laptop, media player, navigation system, camera-equipped device, or other mobile electronic device. The depth map generating device 1200 may also be implemented in a wearable device, such as a watch, glasses, a headband, or a ring, that has communication and data processing capabilities.

[0114] In another embodiment, the depth map generator 1200 may be an electronic device that is integrated into a vehicle, for example, the depth map generator 1200 may be an electronic device that is inserted into a vehicle after the manufacturing process by tuning.

[0115] In yet another embodiment, depth map generator 1200 may be a server located outside the vehicle. The server may be implemented as a computer device or multiple computer devices that communicate over a network to provide instructions, code, files, content, services, etc. The server may receive data necessary for generating the depth map from a device installed in the vehicle and generate the depth map based on the received data.

[0116] In yet other embodiments, the processes performed by the depth map generator 1200 may be performed at least in part by a mobile electronic device, an electronic device integrated into a vehicle, and a server located outside the vehicle.

[0117] Embodiments of the present invention may be realized in the form of a computer program executable by various components on a computer, and such a computer program may be recorded on a computer-readable medium, including magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory.

[0118] On the other hand, the computer program may be one specially designed and constructed for the present invention, or it may be one that is well known and available to those skilled in the art of computer software. Examples of computer programs include not only machine language code such as that produced by a compiler, but also high-level language code that is executed by a computer using an interpreter, etc.

[0119] According to one embodiment, methods according to various embodiments of the present disclosure can be provided in a computer program product. The computer program product can be traded as a commodity between a seller and a buyer. The computer program product can be distributed in the form of a machine-readable storage medium (e.g., a compact disc read only memory (CD-ROM)) or can be distributed through an application store (e.g., the Play Store). TM ) or directly between two user devices. In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored on or temporarily generated by a machine-readable storage medium, such as the memory of a manufacturer's server, an application store server, or an intermediary server.

[0120] Unless explicitly stated or stated to the contrary, steps constituting a method according to the present invention may be performed in any suitable order. The present invention is not necessarily limited to the order of steps described above. The use of all examples or exemplary terms (such as, for example, etc.) in the present invention is merely for the purpose of describing the present invention in detail, and the scope of the present invention is not limited by these examples or exemplary terms unless otherwise limited by the claims. Furthermore, those skilled in the art will understand that various modifications, combinations, and variations can be made within the scope of the claims or their equivalents, depending on design conditions and factors.

[0121] Therefore, the concept of the present invention should not be limited to the above-described embodiments, and not only the scope of the appended claims, but also all scopes equivalent to or modified equivalently from the scope of the claims, can be said to fall within the scope of the concept of the present invention.

Claims

1. 1. A method for generating a depth map using volumetric features, comprising: capturing a surround view image with one or more monocular cameras; encoding base images included in the surround-view image to generate multi-scale image features; resizing and concatenating each of the multi-scale image features relative to a predetermined resolution to generate a single feature map of the base image; generating volumetric features by encoding the single feature map with depth information and then projecting the encoded result into three-dimensional space; generating a depth map for the surround-view image by decoding the volumetric features with a depth decoder.

2. The step of generating volumetric features comprises: extracting local 3D features in a 3D volumetric space by concatenating the single feature map with the depth information, encoding the concatenated feature map, and then projecting the encoded result into the 3D space; and generating the volumetric features using the local three-dimensional features.

3. The step of generating volumetric features comprises: The method of claim 2 , comprising inputting the local 3D features into a volumetric encoder to generate volumetric features.

4. The volumetric encoder is composed of a plurality of multi-layer perceptrons, The step of generating volumetric features comprises: assigning voxels to different multi-layer perceptrons depending on whether overlap occurs for each voxel in the 3D volumetric space; and fusing encoding results from the different multi-layer perceptrons to generate the volumetric features.

5. The step of generating the depth map comprises: transforming the volumetric features into projected image features having the predetermined resolution based on intrinsic and extrinsic parameters of the one or more monocular cameras; and generating a depth map for the surround-view image by decoding the projected image features with a depth decoder.

6. The method comprises: determining a projection matrix for transforming the volumetric features into projected image features having the predetermined resolution; The method of claim 5 , further comprising the step of: modulating the depth map with a projection matrix that matches a desired view.

7. 1. An apparatus for generating a depth map using a monocular camera, comprising: a memory having at least one program stored therein; a processor for executing said at least one program; The processor: capturing a surround view image using one or more monocular cameras; encoding a base image included in the surround-view image to generate multi-scale image features; resizing and concatenating each of the multi-scale image features relative to a predetermined resolution to generate a single feature map of the base image; generating volumetric features by encoding the single feature map with depth information and then projecting the encoded result into three-dimensional space; The apparatus generates a depth map for the surround-view image by decoding the volumetric features with a depth decoder.

8. A computer-readable recording medium having recorded thereon a program for executing the method of claim 1 on a computer.

Images