Methods and apparatus for depth image generation

Abstract

A method and apparatus for generating depth images are provided. The method may include: receiving an input image; generating a first low-resolution image having a resolution lower than that of the input image; obtaining a first depth residual image corresponding to the input image by using a first generative model based on a first neural network; generating a first low-resolution depth image corresponding to the first low-resolution image by using a second generative model based on a second neural network; and generating a target depth image corresponding to the input image based on the first depth residual image and the first low-resolution depth image.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2019-0142886, filed on November 8, 2019, with the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference for all purposes. Technical Field

[0002] The following description relates to image processing techniques for generating depth images. Background Technology

[0003] The use of three-dimensional (3D) information can be important for image recognition or scene understanding. By adding depth information to two-dimensional (2D) spatial information, the spatial distribution of objects can be effectively predicted. Typically, depth information is only obtained when a depth camera is used to acquire a depth image, and the quality of the depth image that can be acquired from a depth camera varies depending on the camera's performance. For example, the noise level or resolution of the acquired depth image can vary depending on the depth camera's performance. Because the accuracy of depth information has a significant impact on the quality of depth-based results, acquiring high-quality depth images is crucial. Summary of the Invention

[0004] The present invention is provided in a simplified form to describe the choice of concepts further described in the following detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to help determine the scope of the claimed subject matter.

[0005] In one general aspect, a method for generating a depth image includes: receiving an input image; generating a first low-resolution image having a resolution lower than that of the input image; obtaining a first depth residual image corresponding to the input image by using a first generative model based on a first neural network; generating a first low-resolution depth image corresponding to the first low-resolution image by using a second generative model based on a second neural network; and generating a target depth image corresponding to the input image based on the first depth residual image and the first low-resolution depth image.

[0006] The steps for generating a target depth image may include: upsampling a first low-resolution depth image to the resolution of an input image; and generating the target depth image by combining the depth information of the upsampled first low-resolution depth image with the depth information of a first depth residual image.

[0007] The steps of generating a first low-resolution depth image may include: using a second generative model to obtain a second depth residual image corresponding to the first low-resolution image; generating a second low-resolution image with a resolution lower than that of the first low-resolution image; using a third generative model based on a third neural network to obtain a second low-resolution depth image corresponding to the second low-resolution image; and generating the first low-resolution depth image based on the second depth residual image and the second low-resolution depth image.

[0008] The step of generating a second low-resolution image may include downsampling the first low-resolution image to generate the second low-resolution image.

[0009] The steps of generating a first low-resolution depth image may include: upsampling a second low-resolution depth image to the resolution of a second depth residual image; and generating the first low-resolution depth image by combining the depth information of the upsampled second low-resolution depth image and the depth information of the second depth residual image.

[0010] The resolution of the second low-resolution depth image can be lower than that of the first low-resolution depth image.

[0011] Compared to the second low-resolution depth image, the second depth residual image may include depth information of high-frequency components.

[0012] Compared to the first depth residual image, the first low-resolution depth image may include depth information of low-frequency components.

[0013] The steps for generating the first low-resolution image may include: downsampling the input image to generate the first low-resolution image.

[0014] The input image can be a color image or an infrared image.

[0015] The input image may include a color image and an input depth image. In acquiring the first depth residual image, the first generative model may use the pixel values ​​of the color image and the pixel values ​​of the input depth image as input, and output the pixel values ​​of the first depth residual image.

[0016] The input image may include an infrared image and an input depth image. In acquiring the first depth residual image, the first generative model may use the pixel values ​​of the infrared image and the pixel values ​​of the input depth image as input, and output the pixel values ​​of the first depth residual image.

[0017] In another general aspect, a non-transitory computer-readable storage medium stores instructions that, when executed by a processor, cause the processor to perform the methods described above.

[0018] In another general aspect, a method for generating a depth image includes: receiving an input image; obtaining a first depth residual image and a first low-resolution depth image by using a generative model based on a neural network that uses the input image as input; and generating a target depth image corresponding to the input image based on the first depth residual image and the first low-resolution depth image.

[0019] The steps of obtaining a first depth residual image and a first low-resolution depth image may include: using a generative model to obtain a second depth residual image and a second low-resolution depth image; and generating a first low-resolution depth image based on the second depth residual image and the second low-resolution depth image.

[0020] The generative model can take an input image as input and output a first depth residual image, a second depth residual image, and a second low-resolution depth image.

[0021] Generative models can include single neural network models.

[0022] In another general aspect, a method for generating depth images includes: receiving an input image; acquiring intermediate depth images of the same size using a generative model based on a neural network that uses the input image as input; and generating a target depth image by combining the acquired intermediate depth images, wherein the intermediate depth images include depth information of varying precision.

[0023] In another general aspect, an apparatus for depth image generation includes a processor configured to: receive an input image; generate a first low-resolution image having a resolution lower than that of the input image; acquire a first depth residual image corresponding to the input image using a first generative model based on a first neural network; generate a first low-resolution depth image corresponding to the first low-resolution image using a second generative model based on a second neural network; and generate a target depth image corresponding to the input image based on the first depth residual image and the first low-resolution depth image.

[0024] The processor can also be configured to: upsample the first low-resolution depth image to the resolution of the input image; and generate a target depth image by combining the depth information of the upsampled first low-resolution depth image with the depth information of the first depth residual image.

[0025] The step of combining the depth information of the upsampled first low-resolution depth image and the depth information of the first depth residual image may include: calculating the weighted sum or total of the depth values ​​of corresponding pixel positions in the first depth residual image and the upsampled first low-resolution depth image.

[0026] The processor can also be configured to: use a second generative model to obtain a second depth residual image corresponding to the first low-resolution image; generate a second low-resolution image with a resolution lower than that of the first low-resolution image; use a third generative model based on a third neural network to obtain a second low-resolution depth image corresponding to the second low-resolution image; and generate a first low-resolution depth image based on the second depth residual image and the second low-resolution depth image.

[0027] The processor can also be configured to: upsample the second low-resolution depth image to the resolution of the second depth residual image; and generate the first low-resolution depth image by combining the depth information of the upsampled second low-resolution depth image and the depth information of the second depth residual image.

[0028] The step of combining the depth information of the upsampled second low-resolution depth image and the depth information of the second depth residual image may include: calculating a weighted sum or total of the depth values ​​of corresponding pixel positions in the second depth residual image and the upsampled second low-resolution depth image.

[0029] The resolution of the first low-resolution depth image may be higher than that of the second low-resolution depth image. Compared to the second low-resolution depth image, the second depth residual image may include depth information of high-frequency components.

[0030] The processor can also be configured to downsample the input image to generate a first low-resolution image.

[0031] The input image may include a color image and an input depth image. In acquiring the first depth residual image, the first generative model may use the pixel values ​​of the color image and the pixel values ​​of the input depth image as input, and output the pixel values ​​of the first depth residual image.

[0032] The input image may include an infrared image and an input depth image. In acquiring the first depth residual image, the first generative model may use the pixel values ​​of the infrared image and the pixel values ​​of the input depth image as input, and output the pixel values ​​of the first depth residual image.

[0033] The device may further include: a sensor configured to acquire an input image, wherein the input image includes any one or both of a color image and an infrared image.

[0034] In another general aspect, an apparatus for depth image generation includes a processor configured to: receive an input image; acquire a first depth residual image and a first low-resolution depth image by using a generative model based on a neural network that uses the input image as input; and generate a target depth image corresponding to the input image based on the first depth residual image and the first low-resolution depth image.

[0035] The processor can also be configured to: acquire a second depth residual image and a second low-resolution depth image using a generative model; and generate a first low-resolution depth image based on the second depth residual image and the second low-resolution depth image.

[0036] The first low-resolution depth image may have a resolution lower than that of the input image. The second low-resolution depth image may have a resolution lower than that of the first low-resolution depth image.

[0037] In another general aspect, an apparatus for generating depth images includes a processor configured to: receive an input image; acquire intermediate depth images of the same size by using a generative model based on a neural network that uses the input image as input; and generate a target depth image by combining the acquired intermediate depth images, wherein the intermediate depth images include depth information of varying precision.

[0038] The steps of combining the acquired intermediate depth images include: calculating the weighted sum or total of the depth values ​​of corresponding pixel positions in the acquired intermediate depth images.

[0039] Other features and aspects will become clear from the following detailed description, the accompanying drawings, and the claims. Attached Figure Description

[0040] Figure 1 An example showing a summary of a depth image generation device.

[0041] Figure 2 This is a flowchart illustrating an example of a depth image generation method.

[0042] Figure 3 This is a flowchart illustrating an example of generating a first low-resolution depth image.

[0043] Figure 4 An example of the processing used to generate a depth image is shown.

[0044] Figure 5 An example of training processing is shown.

[0045] Figure 6 An example of the processing used to generate a depth image is shown.

[0046] Figure 7 An example of training processing is shown.

[0047] Figures 8 to 10 An example of generating a depth image is shown.

[0048] Figure 11 An example configuration of a depth image generation device is shown.

[0049] Figure 12 An example of a computing device configuration is shown.

[0050] Throughout the accompanying drawings and detailed embodiments, unless otherwise described or provided, the same reference numerals will be understood to denote the same elements, features, and structures. The drawings may not be to scale, and for clarity, illustration, and convenience, the relative dimensions, scale, and depiction of elements in the drawings may be exaggerated. Detailed Implementation

[0051] The following detailed embodiments are provided to aid the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but may be changed as will become clear upon understanding this disclosure, except for operations that must occur in a specific order. Furthermore, for clarity and conciseness, descriptions of features known in the art may be omitted.

[0052] The features described herein may be implemented in different forms and should not be construed as limited to the examples described herein. Rather, the examples described herein have been provided merely to illustrate some of the many feasible ways of implementing the methods, apparatus, and / or systems described herein that will be clear upon understanding the disclosure of this application.

[0053] It should be noted here that the use of the term "may" (e.g., what an example or embodiment may include or implement) with respect to examples or embodiments indicates that there exists at least one example or embodiment that includes or implements such a feature, but all examples and embodiments are not limited thereto.

[0054] Throughout this specification, when an element (such as a layer, region, or substrate) is described as being "on" another element, "connected to," or "bonded to" another element, the element may be directly "on" said other element, directly "connected to," or directly "bonded to" said other element, or one or more other elements may be present in between. Conversely, when an element is described as being "directly on" another element, directly "connected to," or "bonded to" another element, no other elements may be present in between. As used herein, the term "and / or" includes any one of the associated listed items and any combination of any two or more.

[0055] Although terms such as “first,” “second,” and “third” may be used herein to describe various components, assemblies, regions, layers, or parts, these components, assemblies, regions, layers, or parts are not limited by these terms. Rather, these terms are used only to distinguish one component, assembly, region, layer, or part from another. Thus, without departing from the teaching of the examples described herein, the first component, first assembly, first region, first layer, or first part referred to as the first component, first assembly, first region, first layer, or first part may also be referred to as the second component, second assembly, second region, second layer, or second part.

[0056] The terminology used herein is for the purpose of describing various examples only and is not intended to limit disclosure. Unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. The terms “comprising,” “including,” and “having” indicate the presence of the described features, quantities, operations, components, elements, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, quantities, operations, components, elements, and / or combinations thereof.

[0057] As will be clear upon understanding the disclosure of this application, the features of the examples described herein can be combined in various ways. Furthermore, although the examples described herein have various configurations, other configurations are possible, as will be clear upon understanding the disclosure of this application.

[0058] Figure 1 An example showing the outline of the depth image generation device 100.

[0059] Reference Figure 1 The depth image generation device 100 is a device for generating a depth image based on an input image. According to examples, the depth image generation device 100 may generate a depth image based on a color image or an infrared image instead of using a separate depth image, or it may generate a depth image with relatively high resolution or high quality from a depth image with relatively low resolution or low quality.

[0060] In one example, the depth image generation device 100 may generate a depth image based on a color image sensed by image sensor 110 or an infrared image sensed by infrared sensor 120. In another example, the depth image generation device 100 may generate a depth image with a higher resolution than the depth image sensed by depth sensor 130, based on the color image sensed by image sensor 110 and the depth image sensed by depth sensor 130. In yet another example, the depth image generation device 100 may generate a depth image with a higher resolution than the depth image sensed by depth sensor 130, based on the infrared image sensed by infrared sensor 120 and the depth image sensed by depth sensor 130. In the foregoing examples, the color image, infrared image, and depth image may be images representing the same scene and corresponding to each other.

[0061] Image sensor 110 is, for example, a sensor configured to acquire a color image representing color information of an object, and includes, for example, a complementary metal-oxide-semiconductor (CMOS) image sensor, a charge-coupled device (CCD) image sensor, or a stereo camera. Infrared sensor 120 is a sensor configured to sense infrared light emitted from an object or infrared reflected light reflected by the object and generate an infrared image. Depth sensor 130 is a means configured to acquire a depth image representing depth information of an object, and may include, for example, a Kinect, a time-of-flight (TOF) depth camera, or a 3D scanner. In the example where image sensor 110 is a stereo camera, a stereo image including a left image and a right image can be acquired from the stereo camera, and a depth image can be obtained from the stereo image using a known stereo matching scheme.

[0062] A depth image is an image that represents depth information, which is information about the depth or distance from the shooting location to an object. Depth images can be used for object recognition (such as 3D face recognition) or for processing photographic effects (such as out-of-focus effects). For example, depth images can be used to understand a scene that includes objects. Depth images can determine the geometric relationships between objects or provide 3D geometric information to help improve the performance of visual object recognition.

[0063] Using physical sensors (e.g., depth sensor 130) to acquire depth images can increase costs, limit depth measurement distances, introduce measurement errors, and lead to vulnerability to external light. Depth image generation device 100 can generate depth images from color or infrared images using a deep learning-based generative model, enabling depth image acquisition even without the use of depth sensor 130, thus overcoming these limitations. For example, based on the depth image generated by depth image generation device 100, distributions in 3D space can be predicted using a single color or infrared image, increasing object recognition accuracy and enabling robust scene understanding even in the presence of occlusion.

[0064] To maximize the utilization of depth images, it is important to use depth images with high resolution or high quality. To obtain the desired results based on depth images, it is crucial to acquire depth images that accurately represent depth features (e.g., depth features of object edges). The depth image generation device 100 can generate high-resolution and high-quality depth images using a multi-scale-based depth image generation method, as described below. That is, the depth image generation device 100 can use a multi-scale-based depth image generation method to estimate depth information more accurately and precisely, distinguishing between global and local information in the depth image and estimating both global and local information.

[0065] Furthermore, the depth image generation device 100 can generate a high-quality depth image by processing a depth image acquired, for example, by the depth sensor 130. The operation by which the depth image generation device 100 generates a higher-quality depth image by processing the depth image provided as input can correspond to depth image calibration. For example, the depth image generation device 100 generates a depth image with finer depth information than the depth information provided as input, based on information included in a color image or infrared image. In this example, the depth image generation device 100 can use a multi-scale-based depth image generation method to generate a high-quality depth image.

[0066] In the following description, the method of generating a depth image by the depth image generating device 100 will be further described with reference to the accompanying drawings.

[0067] Figure 2 This is a flowchart illustrating an example of a depth image generation method.

[0068] Reference Figure 2In operation 210, the depth image generating device receives an input image. The input image may include, for example, a color image including RGB color information and / or an infrared image, and may be a single image. The depth image generating device receives the input image from an image acquiring device, and the image acquiring device may include an image sensor and / or an infrared sensor.

[0069] In operation 220, the depth image generation device uses a first generative model based on a first neural network to acquire a first depth residual image corresponding to the input image. Pixel values ​​of the input image are input to the first generative model, and the first generative model outputs a first depth residual image corresponding to the scale of the input image. The first generative model is a model trained through a training process to output a depth residual image based on input information. The first depth residual image includes depth information, such as high-frequency components, and is an image that can relatively accurately represent the edge components of an object. In this disclosure, the terms "scale" and "resolution" are used interchangeably.

[0070] In operation 230, the depth image generating device generates a first low-resolution image with a resolution lower than that of the input image. In one example, the depth image generating device may downsample the input image to generate the first low-resolution image. For example, the depth image generating device may generate a first low-resolution image corresponding to half the scale of the input image. In an example where the input image is a color image, the first low-resolution image may be a color image with reduced resolution. In an example where the input image is an infrared image, the first low-resolution image may be an infrared image with reduced resolution.

[0071] In operation 240, the depth image generation device uses a second generative model based on a second neural network to generate a first low-resolution depth image corresponding to the first low-resolution image. Compared to the first depth residual image generated in operation 220, the first low-resolution depth image includes depth information, for example, low-frequency components. The second generative model is also a model trained through a training process.

[0072] In one example, a depth image generation device estimates depth information of both high-frequency and low-frequency components, and combines the estimated high-frequency and low-frequency depth information to generate a depth image. In this example, pixel values ​​from a first low-resolution image are input to a second generation model, and the second generation model outputs a first low-resolution depth image corresponding to the scale or resolution of the first low-resolution image. The second generation model is trained to output the first low-resolution depth image based on the input information. The first depth residual image includes depth information of the high-frequency components, and the first low-resolution depth image includes depth information of the low-frequency components.

[0073] In another example, the depth image generation device estimates depth information of high-frequency components, mid-frequency components, and low-frequency components, and combines the estimated depth information of the high-frequency components, mid-frequency components, and low-frequency components to generate a depth image. In this example, a third generative model based on a third neural network can be used in conjunction with a second generative model. (Refer to...) Figure 3 Let me describe this example further.

[0074] Reference Figure 3 In operation 310, the depth image generation device uses a second generation model to acquire a second depth residual image corresponding to the first low-resolution image. The pixel values ​​of the first low-resolution image are input into the second generation model, and the second generation model outputs the second depth residual image corresponding to the first low-resolution image. The second depth residual image may include depth information of the intermediate frequency component. The second generation model is a model trained to output the second depth residual image based on the input information. Figure 3 The second generative model in Figure 2 The second generative model can be the same generative model.

[0075] In operation 320, the depth image generating device generates a second low-resolution image with a resolution lower than that of the first low-resolution image. For example, the depth image generating device may downsample the first low-resolution image to generate the second low-resolution image. For example, the depth image generating device may generate a second low-resolution image corresponding to half the scale of the first low-resolution image.

[0076] In operation 330, the depth image generation device uses a third generation model to acquire a second low-resolution depth image corresponding to the second low-resolution image. The second low-resolution depth image has a lower resolution than the first low-resolution depth image and includes depth information, such as low-frequency components. The third generation model is a model trained to output the second low-resolution depth image based on the input information.

[0077] In operation 340, the depth image generating device generates a first low-resolution depth image based on a second depth residual image and a second low-resolution depth image. Compared to the second low-resolution depth image, the second depth residual image may include depth information of high-frequency components. In one example, the depth image generating device upsamples the second low-resolution depth image to the resolution of the second depth residual image, and combines the depth information from the upsampled second low-resolution depth image and the depth information from the second depth residual image to generate the first low-resolution depth image.

[0078] Return to reference Figure 2In operation 250, the depth image generation device generates a target depth image corresponding to the input image based on a first depth residual image and a first low-resolution depth image. In one example, the depth image generation device upsamples the first low-resolution depth image to the resolution of the input image and combines the depth information of the upsampled first low-resolution depth image with the depth information of the first depth residual image to generate the target depth image.

[0079] As described above, the depth image generation device can generate depth images based on a structure that is progressively refined through scale subdivision of depth information. The device can be configured with input images of various scales, allowing each scale's input image to be fed into a corresponding generation model, and can acquire images from each model that include depth information with different frequency components. The device can generate a final target depth image by combining the acquired images containing depth information with different frequency components. Therefore, the depth image generation device can obtain high-quality depth images from color or infrared images instead of using a separate depth sensor or initial depth image.

[0080] In one example, the depth image generation device can generate a higher-quality depth image by calibrating the input depth image provided as input. In this example, Figure 2 The method for generating the depth image can be modified. In operation 210, the input depth image can be provided to the depth image generating device along with one of a color image and an infrared image. The input depth image can be a depth image acquired by a depth sensor or a depth image generated through image processing (e.g., a stereo matching scheme).

[0081] In an example where the input image includes a color image and an input depth image, in operation 220, the depth image generation device can use a first generation model to obtain a first depth residual image. The first generation model uses the pixel values ​​of the color image and the pixel values ​​of the input depth image as input and outputs the pixel values ​​of the first depth residual image. In operation 230, the depth image generation device can generate a first low-resolution input depth image with a resolution lower than that of the input depth image, and a first low-resolution image with a resolution lower than that of the color image. In operation 240, the depth image generation device can use a second generation model to obtain the first low-resolution depth image. The second generation model uses the pixel values ​​of the first low-resolution image and the pixel values ​​of the first low-resolution input depth image as input and outputs the pixel values ​​of the first low-resolution depth image.

[0082] In another example, the depth image generation device can also use... Figure 3Similar processing methods can be used to obtain a first low-resolution depth image. For example, in operation 310, the depth image generation device can use a second generation model to obtain a second depth residual image, the second generation model using the pixel values ​​of the first low-resolution image and the pixel values ​​of the first low-resolution input depth image as input and outputting the pixel values ​​of the second depth residual image. In operation 320, the depth image generation device can generate a second low-resolution input depth image with a resolution lower than that of the first low-resolution input depth image and a second low-resolution image with a resolution lower than that of the first low-resolution image. In operation 330, the depth image generation device can use a third generation model to obtain a second low-resolution depth image, the third generation model using the pixel values ​​of the second low-resolution image and the pixel values ​​of the second low-resolution input depth image as input and outputting the pixel values ​​of the second low-resolution depth image. In operation 340, the depth image generation device can generate a first low-resolution depth image by combining the second depth residual image and the second low-resolution depth image.

[0083] Similar to the above description, in operation 250, the depth image generation device generates a target depth image corresponding to the input image based on the first depth residual image and the first low-resolution depth image. Unlike the example above, when the input image includes an infrared image and an input depth image, the depth image generation device can generate the target depth image based on the process in which the infrared image replaces the color image in the above processing. Even if the depth information is not fine or even if a low-quality (e.g., noisy) depth image is provided as the input depth image, the depth image generation device can generate a higher-quality depth image based on the multi-scale depth image generation structure described above.

[0084] Figure 4 An example of the processing used to generate a depth image is shown.

[0085] As described above, a depth image generation device can generate a depth image from an input image using a multi-scale depth estimation structure. Even when depth information is not provided as input, the depth image generation device can estimate depth information from a color image or an infrared image using a multi-scale depth estimation structure. A multi-scale depth estimation structure decomposes the input image into frequency components and estimates and processes depth information corresponding to each frequency component. For example, Figure 4 The multi-scale-based depth estimation structure decomposes the input image 410 into high-frequency and low-frequency components, estimates the depth information corresponding to each frequency component, combines the estimated depth information, and generates a depth image. Using this multi-scale-based depth estimation structure, the depth information is sequentially refined for each scale to generate the final target depth image.

[0086] Reference Figure 4 The depth image generation device receives input image 410. Input image 410 may include, for example, a color image or an infrared image, and may be a single image. In one example, an image obtained by concatenating a color image and an infrared image may be provided as input image 410. Although the following describes an example where input image 410 is a color image, the following processing also applies to examples where input image 410 is another image.

[0087] The depth image generation device uses a first generative model 420 based on a first neural network to acquire a first depth residual image 430 corresponding to the input image 410. The pixel values ​​of the input image 410 are input to the first generative model 420, and the first generative model 420 outputs the pixel values ​​of the first depth residual image 430. The first depth residual image 430 has a resolution or scale corresponding to the resolution or scale of the input image 410, and includes depth information of high-frequency components, including edge detail components of objects.

[0088] The depth image generation device generates a first low-resolution image 440 by downsampling the input image 410. For example, the depth image generation device may downsample the input image 410, perform blurring (e.g., Gaussian smoothing), and generate the first low-resolution image 440. Compared to the input image 410, the first low-resolution image 440 includes color information of low-frequency components.

[0089] The depth image generation device uses a second generation model 450 based on a second neural network to generate a first low-resolution depth image 460 corresponding to the first low-resolution image 440. Pixel values ​​of the first low-resolution image 440 are input to the second generation model 450, and the second generation model 450 outputs the pixel values ​​of the first low-resolution depth image 460. The first low-resolution depth image 460 has a resolution or scale corresponding to that of the first low-resolution image 440, and includes depth information with low-frequency components compared to the first depth residual image 430.

[0090] The first generative model 420 and the second generative model 450 are models trained to output a first depth residual image 430 and a first low-resolution depth image 460, respectively, based on input information. Image-to-image transformation schemes using generative adversarial networks (GANs) (e.g., Pix2Pix, GycleGAN, and DiscoGAN) can be used to implement the first generative model 420 and the second generative model 450.

[0091] The depth image generation device upsamples a first low-resolution depth image 460 and generates an upsampled first low-resolution depth image 470. For example, the depth image generation device upsamples the first low-resolution depth image 460 to generate a first low-resolution depth image 470 with a scale corresponding to the scale of the first depth residual image 430. In operation 480, the depth image generation device combines the first depth residual image 430 and the upsampled first low-resolution depth image 470 to generate a target depth image 490 corresponding to the input image 410. For example, operation 480 corresponds to a weighted sum or total of the depth values ​​of corresponding pixel positions in the first depth residual image 430 and the upsampled first low-resolution depth image 470. In one example, the first depth residual image 430 includes depth information of a residual component obtained by removing the depth information of the upsampled first low-resolution depth image 470 from the target depth image 490.

[0092] As described above, in the generation of depth information by combining global and local information of depth information, the depth image generation device can guide the estimation of different frequency components of depth information in each of the multiple steps in a multi-scale structure to generate a depth image with higher resolution. The depth image generation device can also guide the processing of residual components corresponding to depth information not estimated in each step in another step, thus allowing the estimation of depth information of frequency components to be separated and independent in each step. Even without using a separate depth sensor, the depth image generation device can generate complex depth images from color or infrared images, and can also generate multiple depth images from a single input image.

[0093] In one example, the depth image generation device uses... Figure 4A multi-scale depth estimation structure is used to calibrate the depth information of the input depth image to generate a depth image of higher quality than the input depth image. In this example, the input depth image and a color image or an infrared image can be provided as input image 410. The depth image generation device uses a first generation model 420 to obtain a first depth residual image 430, which uses the pixel values ​​of the color image and the pixel values ​​of the input depth image as input and outputs the pixel values ​​of the first depth residual image 430. The depth image generation device generates a first low-resolution input depth image with a resolution lower than that of the input depth image and a first low-resolution image 440 with a resolution lower than that of the color image. The depth image generation device uses a second generation model 450 to obtain a first low-resolution depth image 460, which uses the pixel values ​​of the first low-resolution image 440 and the pixel values ​​of the first low-resolution input depth image as input and outputs the pixel values ​​of the first low-resolution depth image 460. Subsequent operations can be the same as those described above, and the target depth image 490 generated based on the above processing can include more refined and accurate depth information than the depth information of the input depth image provided as input.

[0094] Figure 5 An example of training processing is shown.

[0095] Reference Figure 5 A training device for performing training processing trains a first generative model 515 and a second generative model 535. The training device downsamples a depth image 580, which serves as the target image, to generate a depth image 590 with a reduced scale. Depth image 580 may include depth information of high-frequency components, and depth image 590 may include depth information of low-frequency components. Each of depth images 580 and 590 is used as a reference image for calculating the error value of the output of each of the first generative model 515 and the second generative model 535.

[0096] When training image 510 is provided, the training device uses a first generative model 515 based on a first neural network to generate a first depth residual image 520 corresponding to training image 510. Training image 510 may include, for example, a color image, an infrared image, or an image obtained by concatenating a color image and an infrared image. First depth residual image 520 may include depth information of high-frequency components.

[0097] The training device downsamples the training image 510 to generate a first low-resolution image 530. The training device uses a second generative model 535 based on a second neural network to generate a first low-resolution depth image 540 corresponding to the first low-resolution image 530. The first low-resolution depth image 540 includes depth information from low-frequency components.

[0098] The training device upsamples the first low-resolution depth image 540 to generate an upsampled first low-resolution depth image 550 with the same scale as the first depth residual image 520, and in operation 560, the upsampled first low-resolution depth image 550 and the first depth residual image 520 are combined to generate a result depth image 570.

[0099] The above processing of the depth image 570 generated by the training device corresponds to... Figure 4 The example shows the process of generating a target depth image 490 based on the input image 410.

[0100] The training device calculates the difference between the resulting depth image 570 and the ground truth depth image 580, which corresponds to the depth information of high-frequency components, by comparing the resulting depth image 570 and the depth image 580. The training device adjusts the values ​​of parameters (e.g., the parameters of the first neural network of the first generative model 515) to reduce the difference between the resulting depth image 570 and the depth image 580. For example, the training device may find optimal parameter values ​​that minimize the value of the loss function that defines the difference between the resulting depth image 570 and the depth image 580. In this example, the loss function may be defined in various forms based on a classification scheme or a regression scheme. The scheme for adjusting the parameter values ​​or calibrating the processing of the depth information used to generate the depth image 580 may be changed depending on how the loss function is defined. Furthermore, the training device calculates the difference between the first low-resolution depth image 540 and the ground truth depth image 590, which corresponds to the depth information of low-frequency components, by comparing the first low-resolution depth image 540 and the depth image 590. The training device adjusts the parameters of the second generative model 535 to reduce the difference between the first low-resolution depth image 540 and the depth image 590. The training device can find the optimal values ​​of the parameters of each of the first generative model 515 and the second generative model 535 by repeatedly performing the above processing on a large number of training images.

[0101] As a result, through training, the first generative model 515 is trained to output a first depth residual image, which includes residual components obtained by subtracting a depth image generated by upsampling a depth image 590 at the scale of the depth image 580 from the depth image 580, and the second generative model 535 is trained to output a downsampled depth image 590.

[0102] The training device can find the optimal values ​​of the parameters of each of the first generative model 515 and the second generative model 535 by repeatedly performing the above processing on a large number of training images. The training device trains the first generative model 515 and the second generative model 535 separately to estimate the depth information of each frequency component of the depth image, so that the depth information is estimated efficiently. In the multi-scale-based depth estimation architecture, the training device uses the depth estimation results of previous operations as guidance for the next training.

[0103] Figure 6 This shows another example of the processing used to generate depth images.

[0104] exist Figure 6 In this process, the depth image generation device uses a three-layer depth estimation structure to generate a depth image. The depth image generation device uses the depth estimation structure to estimate the depth information of high-frequency components, mid-frequency components, and low-frequency components separately, and combines the estimated depth information to generate the final target depth image.

[0105] Reference Figure 6 The depth image generating device receives an input image 610. The input image 610 may include, for example, a color image, an infrared image, or an image obtained by concatenating a color image and an infrared image. Although the following description is an example where the input image 610 is a color image, the following processing also applies to an example where the input image 610 is another image.

[0106] The depth image generation device uses a first generation model 615 based on a first neural network to acquire a first depth residual image 620 corresponding to an input image 610. The pixel values ​​of the input image 610 are input to the first generation model 615, and the first generation model 615 outputs the pixel values ​​of the first depth residual image 620. The first depth residual image 620 may have a resolution or scale corresponding to the resolution or scale of the input image 610, and may include depth information of high-frequency components.

[0107] The depth image generation device downsamples the input image 610 to generate a first low-resolution image 625. For example, the depth image generation device may downsample the input image 610, perform Gaussian smoothing, and generate the first low-resolution image 625. Compared to the input image 610, the first low-resolution image 625 may include color information of low-frequency components.

[0108] The depth image generation device uses a second generation model 630 based on a second neural network to acquire a second depth residual image 640 corresponding to the first low-resolution image 625. Pixel values ​​of the first low-resolution image 625 are input to the second generation model 630, and the second generation model 630 outputs pixel values ​​of the second depth residual image 640. The second depth residual image 640 may include depth information of the mid-frequency component, and compared to the first depth residual image 620, may include depth information of the low-frequency component.

[0109] The depth image generation device downsamples the first low-resolution image 625 to generate a second low-resolution image 645. For example, the depth image generation device may downsample the first low-resolution image 625, perform Gaussian smoothing, and generate the second low-resolution image 645. Compared to the first low-resolution image 625, the second low-resolution image 645 includes color information of low-frequency components.

[0110] The depth image generation device uses a third generation model 650 based on a third neural network to acquire a second low-resolution depth image 655 corresponding to the second low-resolution image 645. The pixel values ​​of the second low-resolution image 645 are input to the third generation model 650, and the third generation model 650 outputs the pixel values ​​of the second low-resolution depth image 655. The second low-resolution depth image 655 may include depth information of low-frequency components.

[0111] The first generative model 615, the second generative model 630, and the third generative model 650 are models trained to output a first depth residual image 620, a second depth residual image 640, and a second low-resolution depth image 655, respectively, based on input information. Image-to-image transformation schemes using GANs (e.g., Pix2Pix, GycleGAN, and DiscoGAN) can be used to implement the first generative model 615, the second generative model 630, and the third generative model 650.

[0112] The depth image generation device upsamples the second low-resolution depth image 655 to generate an upsampled second low-resolution depth image 660. For example, the depth image generation device may upsample the second low-resolution depth image 655 to generate a second low-resolution depth image 660 with a scale corresponding to the scale of the second depth residual image 640. In operation 665, the depth image generation device combines the second depth residual image 640 and the upsampled second low-resolution depth image 660 to generate a first low-resolution depth image 670. For example, operation 665 may correspond to a weighted sum or total of the depth values ​​of corresponding pixel positions in the second depth residual image 640 and the upsampled second low-resolution depth image 660. In one example, the second depth residual image 640 includes depth information of a residual component obtained by removing the depth information of the upsampled second low-resolution depth image 660 from the first low-resolution depth image 670.

[0113] The depth image generation device upsamples a first low-resolution depth image 670 to generate an upsampled first low-resolution depth image 675. For example, the depth image generation device may upsample the first low-resolution depth image 670 and generate a first low-resolution depth image 675 with a scale corresponding to the scale of the first depth residual image 620. In operation 680, the depth image generation device combines the first depth residual image 620 and the upsampled first low-resolution depth image 675 to generate a target depth image 685 corresponding to the input image 610. For example, operation 680 corresponds to a weighted sum or total of the depth values ​​at corresponding pixel locations in the first depth residual image 620 and the upsampled first low-resolution depth image 675. In one example, the first depth residual image 620 includes depth information of a residual component obtained by removing the depth information of the upsampled first low-resolution depth image 675 from the target depth image 685.

[0114] As described above, the depth image generation device combines global and local depth information through multiple steps based on a multi-scale depth estimation structure. The device extracts global depth information from the color image with the smallest scale, extracts local depth information from color images with other scales, and adds the extracted local depth information to the extracted global depth information to gradually refine the depth information.

[0115] Multi-scale-based depth estimation structures for generating depth images can have four or more layers. Figure 4 The example describes two layers and Figure 6 The example describes three layers.

[0116] In one example, the depth image generation device uses... Figure 6A multi-scale depth estimation structure is used to calibrate the input depth image to generate a depth image of higher quality than the input depth image. In this example, the input image 610 may include, for example, an input depth image and a color image or an infrared image. The depth image generation device uses a first generation model 615 to obtain a first depth residual image 620, which uses pixel values ​​from the color image and the input depth image as input and outputs pixel values ​​from the first depth residual image 620. The depth image generation device generates a first low-resolution input depth image with a resolution lower than that of the input depth image and a first low-resolution image 625 with a resolution lower than that of the color image. The depth image generation device uses a second generation model 630 to obtain a second depth residual image 640, which uses pixel values ​​from the first low-resolution image 625 and the first low-resolution input depth image as input and outputs pixel values ​​from the second depth residual image 640. The depth image generation device downsamples the first low-resolution image 625 and the first low-resolution input depth image to generate a second low-resolution image 645 and a second low-resolution input depth image, respectively. The depth image generation device uses a third generation model 650 to acquire the second low-resolution depth image 655. The third generation model 650 uses the pixel values ​​of the second low-resolution image 645 and the pixel values ​​of the second low-resolution input depth image as input and outputs the pixel values ​​of the second low-resolution depth image 655. Subsequent operations can be the same as described above, and the target depth image 685 generated based on the above processing can include more refined and accurate depth information than the depth information provided as input to the input depth image.

[0117] Figure 7 Another example of training processing is shown.

[0118] Reference Figure 7 The training device trains the first generative model 715, the second generative model 730, and the third generative model 750 through training processing. The training device decomposes the depth image 790, which serves as the target image, into three different frequency components, and trains the first generative model 715, the second generative model 730, and the third generative model 750 so that the depth information of each frequency component is effectively estimated based on the corresponding one of the depth images 790, 792, and 794 corresponding to each frequency component, respectively.

[0119] The training device downsamples depth image 790 to generate depth image 792 with a reduced scale, and downsamples depth image 792 to generate depth image 794 with a further reduced scale. Depth image 790 may include depth information of high-frequency components, depth image 792 may include depth information of mid-frequency components, and depth image 794 may include depth information of low-frequency components. Each of depth images 790, 792, and 794 is used as a reference image for calculating the error value of the output of each of the first generative model 715, the second generative model 730, and the third generative model 750.

[0120] When the training image 710 is provided, the training device uses a first generative model 715 based on a first neural network to generate a first depth residual image 720 corresponding to the training image 710. The training image 710 may include, for example, a color image, an infrared image, or an image obtained by concatenating a color image and an infrared image. The first depth residual image 720 may include depth information of high-frequency components.

[0121] The training device downsamples the training image 710 to generate a first low-resolution image 725. The training device uses a second generative model 730 based on a second neural network to generate a second depth residual image 740 corresponding to the first low-resolution image 725. The second depth residual image 740 may include depth information of the intermediate frequency components.

[0122] The training device downsamples the first low-resolution image 725 to generate a second low-resolution image 745. The training device uses a third generative model 750 based on a third neural network to generate a second low-resolution depth image 755 corresponding to the second low-resolution image 745. The second low-resolution depth image 755 may include depth information from low-frequency components.

[0123] The training device upsamples the second low-resolution depth image 755 to the scale of the second depth residual image 740 to generate an upsampled second low-resolution depth image 760. In operation 765, the training device combines the second depth residual image 740 and the upsampled second low-resolution depth image 760 to generate a first low-resolution depth image 770. The training device upsamples the first low-resolution depth image 770 to generate an upsampled first low-resolution depth image 775 with the same scale as the first depth residual image 720, and in operation 780, combines the upsampled first low-resolution depth image 775 and the first depth residual image 720 to generate a result depth image 785.

[0124] The above processing of the depth image 785 generated by the training device corresponds to Figure 5The example shows the process of generating a target depth image based on training image 510.

[0125] The training device calculates the difference between the resulting depth image 785 and the ground truth depth image 790 corresponding to the high-frequency components of the depth information, and adjusts the values ​​of the parameters of the first generative model 715 to reduce the difference between the resulting depth image 785 and the depth image 790. The training device calculates the difference between the first low-resolution depth image 770 and the ground truth depth image 792 corresponding to the mid-frequency components of the depth information, and adjusts the values ​​of the parameters of the second generative model 730 to reduce the difference between the first low-resolution depth image 770 and the depth image 792. Furthermore, the training device calculates the difference between the second low-resolution depth image 755 and the ground truth depth image 794 corresponding to the low-frequency components of the depth information, and adjusts the values ​​of the parameters of the third generative model 750 to reduce the difference between the second low-resolution depth image 755 and the depth image 794. The training device can find the optimal values ​​of the parameters of each of the first generative model 715, the second generative model 730, and the third generative model 750 by repeatedly performing the above processing on a large number of training images.

[0126] As a result, through training, the first generative model 715 is trained to output a first depth residual image, which includes a residual component obtained by subtracting a depth image generated by upsampling depth image 792 according to the scale of depth image 790 from depth image 790. The second generative model 730 is trained to output a second depth residual image, which includes a residual component obtained by subtracting a depth image generated by upsampling depth image 794 according to the scale of depth image 792 from depth image 792. Furthermore, the third generative model 750 is trained to output a downsampled depth image 794.

[0127] As described above, the training device decomposes the depth image 790 into multiple frequency components and trains the first generative model 715, the second generative model 730, and the third generative model 750 to estimate the depth information for each frequency component. In operations other than those using the third generative model 750, the training device allows learning only the depth residual components from previous operations to separate the characteristics of the depth information estimated in each operation and allows for the learning of these characteristics. The depth information estimated in previous operations is used to generate images for training in the next operation and to guide the next operation. The training device guides the residual components that were not estimated in each operation to be processed in the next operation, such that each of the first generative model 715, the second generative model 730, and the third generative model 750 effectively estimates the depth information of the frequency component corresponding to each of the first generative model 715, the second generative model 730, and the third generative model 750.

[0128] Figures 8 to 11 An example of generating a depth image is shown.

[0129] Reference Figure 8 The depth image generation device receives an input image 810 and uses a generative model 820 based on a neural network model to acquire a first depth residual image 830 and a first low-resolution depth image 840, instead of performing a process to transform the resolution or scale of the input image 810. The input image 810 may include, for example, a color image or an infrared image, and may be a single image. The generative model 820 corresponds to, for example, a single neural network model, and outputs the first depth residual image 830 and the first low-resolution depth image 840 based on the input image 810 through different output layers. The functionality of the generative model 820 is implemented through a process of training the generative model 820. The first depth residual image 830 and the first low-resolution depth image 840 may respectively correspond to... Figure 4 The first depth residual image 430 and the first low-resolution depth image 460.

[0130] and Figure 4 Similarly, the depth image generation device upsamples the first low-resolution depth image 840 to generate an upsampled first low-resolution depth image 850, and in operation 860, combines the first depth residual image 830 and the upsampled first low-resolution depth image 850 to generate a target depth image 870 corresponding to the input image 810. For example, operation 860 corresponds to a weighted sum or total of the depth values ​​at corresponding pixel positions in the first depth residual image 830 and the upsampled first low-resolution depth image 850.

[0131] Reference Figure 9 A depth image generation device receives an input image 910 and uses a generative model 920 based on a neural network model to acquire a first depth residual image 930, a second depth residual image 940, and a second low-resolution depth image 950, instead of performing a process that transforms the resolution or scale of the input image 910. The input image 910 may include, for example, a color image or an infrared image, and may be a single image. The generative model 920 corresponds to, for example, a single neural network model, and outputs the first depth residual image 930, the second depth residual image 940, and the second low-resolution depth image 950 based on the input image 910 through different output layers. The generative model 920 is a model trained to output the first depth residual image 930, the second depth residual image 940, and the second low-resolution depth image 950 based on input information. To implement the generative model 920, an image-to-image conversion scheme utilizing GANs can be used. The first depth residual image 930, the second depth residual image 940, and the second low-resolution depth image 950 may respectively correspond to... Figure 6 The first depth residual image 620, the second depth residual image 640, and the second low-resolution depth image 655.

[0132] and Figure 6 Similarly, the depth image generation device upsamples the second low-resolution depth image 950 to generate an upsampled second low-resolution depth image 960, and in operation 965, combines the second depth residual image 940 and the upsampled second low-resolution depth image 960 to generate a first low-resolution depth image 970. For example, operation 965 corresponds to a weighted sum or total of the depth values ​​at corresponding pixel positions in the second depth residual image 940 and the upsampled second low-resolution depth image 960.

[0133] The depth image generation device upsamples the first low-resolution depth image 970 to generate an upsampled first low-resolution depth image 975, and in operation 980, combines the first depth residual image 930 and the upsampled first low-resolution depth image 975 to generate a target depth image 990 corresponding to the input image 910. For example, operation 980 corresponds to a weighted sum or total of the depth values ​​of corresponding pixel positions in the first depth residual image 930 and the upsampled first low-resolution depth image 975.

[0134] Multi-scale-based depth estimation structures for generating depth images can have four or more layers, and so on. Figure 8 The example describes two layers and Figure 9 The example describes three layers.

[0135] Reference Figure 10The depth image generation device receives an input image 1010 and uses a generative model 1020 based on a neural network model that uses the input image 1010 as input to obtain intermediate depth images 1030, 1040, and 1050. For example, intermediate depth images 1030, 1040, and 1050 may have the same size but include depth information of different precision. The generative model 1020 outputs intermediate depth images 1030, 1040, and 1050, including depth information of different precision, through different output layers based on the input image 1010. The generative model 1020 is a single neural network model trained to output intermediate depth images 1030, 1040, and 1050 based on input information. For example, intermediate depth image 1030 includes depth information with relatively high precision, intermediate depth image 1050 includes depth information with relatively low precision, and intermediate depth image 1040 includes depth information with medium precision. Intermediate depth image 1030 includes, for example, local depth information or high-frequency depth information, and intermediate depth image 1040 includes, for example, intermediate frequency depth information. Intermediate depth image 1050 includes, for example, global depth information or low-frequency depth information.

[0136] In operation 1060, the depth image generating device combines the intermediate depth images 1030, 1040, and 1050 to generate a target depth image 1070 corresponding to the input image 1010. For example, operation 1060 corresponds to a weighted sum or total of the depth values ​​of corresponding pixel positions in the intermediate depth images 1030, 1040, and 1050. Through the above processing, the depth image generating device generates a high-quality depth image based on a color image or an infrared image.

[0137] Figure 11 An example configuration of the depth image generation device 1100 is shown.

[0138] Reference Figure 11 The depth image generation device 1100 includes, for example, a sensor 1110, a processor 1120, and a memory 1130. The sensor 1110, processor 1120, and memory 1130 communicate with each other via a communication bus 1140. In one example, the sensor 1110 may be located outside the depth image generation device.

[0139] Sensor 1110 may include any one or any combination of an image sensor, an infrared sensor, and a depth sensor, wherein the image sensor is configured to acquire a color image, the infrared sensor is configured to acquire an infrared image, and the depth sensor is configured to acquire a depth image. For example, sensor 1110 acquires an input image including any one or both of a color image and an infrared image. Sensor 1110 transmits the acquired input image to any one or both of processor 1120 and memory 1130.

[0140] Processor 1120 controls the depth image generation device and processes at least one operation associated with the depth image generation method described above. In one example, processor 1120 receives an input image including any one or both of a color image and an infrared image, and generates a first low-resolution image having a resolution lower than that of the input image. Processor 1120 downsamples the input image to generate the first low-resolution image. Processor 1120 uses a first generative model based on a first neural network to obtain a first depth residual image corresponding to the input image, and uses a second generative model based on a second neural network to generate a first low-resolution depth image corresponding to the first low-resolution image. Processor 1120 generates a target depth image corresponding to the input image based on the first depth residual image and the first low-resolution depth image. Compared to the first low-resolution depth image, the first depth residual image includes depth information, such as high-frequency components. Processor 1120 upsamples the first low-resolution depth image to the resolution of the input image, and combines the depth information of the upsampled first low-resolution depth image and the depth information of the first depth residual image to generate the target depth image.

[0141] In one example, to generate a target depth image in a three-layer multi-scale structure in processor 1120, in addition to the first and second generative models, processor 1120 may also use a third generative model based on a third neural network. In this example, processor 1120 uses the second generative model to obtain a second depth residual image corresponding to the first low-resolution image. Processor 1120 generates a second low-resolution image with a resolution lower than that of the first low-resolution image, and uses the third generative model to obtain a second low-resolution depth image corresponding to the second low-resolution image. Processor 1120 upsamples the second low-resolution depth image to the resolution of the second depth residual image, and combines the depth information of the upsampled second low-resolution depth image and the depth information of the second depth residual image to generate a first low-resolution depth image. Compared to the second low-resolution depth image, the second depth residual image includes depth information of high-frequency components. Processor 1120 combines the generated first low-resolution depth image and the first depth residual image to generate the target depth image.

[0142] In another example, processor 1120 performs processing to generate a high-quality depth image by calibrating an input depth image acquired by a depth sensor based on a color image or an infrared image. (The above has already been referenced.) Figure 2 This example is described.

[0143] In another example, processor 1120 receives an input image and uses a generative model based on a neural network using the input image as input to obtain a first depth residual image and a first low-resolution depth image. Processor 1120 generates a target depth image corresponding to the input image based on the first depth residual image and the first low-resolution depth image. To obtain the first low-resolution depth image, processor 1120 uses a generative model to obtain a second depth residual image and a second low-resolution depth image, and generates the first low-resolution depth image based on the second depth residual image and the second low-resolution depth image. (The above has been referred to...) Figure 8 and Figure 9 This example is described.

[0144] In yet another example, processor 1120 receives an input image and uses a generative model based on a neural network that uses the input image as input to acquire intermediate depth images of the same size. The intermediate depth images include depth information of varying precision. Processor 1120 combines the acquired intermediate depth images to generate a target depth image. (The above has already been referred to...) Figure 10 This example is described.

[0145] In addition, processor 1120 can execute the above references Figures 1 to 10 At least one of the operations described herein, and further descriptions of them will not be repeated here.

[0146] The memory 1130 stores information and result information used in the aforementioned processing for generating the depth image. Furthermore, the memory 1130 stores instructions that can be read from a computer. When the instructions stored in the memory 1130 are executed by the processor 1120, the processor 1120 processes at least one of the aforementioned operations.

[0147] Figure 12 An example configuration of computing device 1200 is shown.

[0148] The computing device 1200 is a device configured to perform the function of generating depth images, and performs... Figure 11 The operation of the depth image generation device. (Refer to...) Figure 12 The computing device 1200 includes, for example, a processor 1210, a memory 1220, a first camera 1230, a second camera 1235, a storage device 1240, an input device 1250, an output device 1260, a communication device 1270, and a communication bus 1280. Each component in the computing device 1200 exchanges data and / or information with another component via the communication bus 1280.

[0149] Processor 1210 performs functions and executes instructions within computing device 1200. For example, processor 1210 may process instructions stored in memory 1220 or storage device 1240. Processor 1210 executes the above-described instructions. Figures 1 to 11 At least one of the operations described.

[0150] Memory 1220 stores data and / or information. Memory 1220 includes a non-transitory computer-readable storage medium or computer-readable storage device. Memory 1220 may include, for example, random access memory (RAM), dynamic RAM (DRAM), static RAM (SRAM), or other types of non-volatile memory known in the art. Memory 1220 stores instructions to be executed by processor 1210, as well as information related to the execution of software or applications while they are being executed by computing device 1200.

[0151] The first camera 1230 can acquire any one or both of still images and video images as color images. The first camera 1230 corresponds to, for example, an image sensor described herein. The second camera 1235 can acquire infrared images. The second camera 1235 can capture infrared light emitted from or reflected from an object. The second camera 1235 corresponds to, for example, an infrared sensor described herein. In one example, the computing device 1200 may include any one or both of the first camera 1230 and the second camera 1235. In another example, the computing device 1200 may also include a third camera (not shown) configured to acquire depth images. In this example, the third camera may correspond to the depth sensor described herein.

[0152] Storage device 1240 includes a non-transitory computer-readable storage medium or a computer-readable storage device. Storage device 1240 can store a larger amount of information than memory 1220 and can store information for a relatively long period of time. Storage device 1240 may include, for example, magnetic hard disk, optical disk, flash memory, electrically erasable programmable read-only memory (EEPROM), or other types of non-volatile memory known in the art.

[0153] Input device 1250 receives input from a user via tactile input, video input, audio input, or touch input. For example, input device 1250 may detect input from a keyboard, mouse, touchscreen, microphone, or user, and may include other devices configured to transmit the detected input to computing device 1200.

[0154] Output device 1260 provides the output of computing device 1200 to a user using a visual, auditory, or tactile approach. For example, output device 1260 may include, for example, a liquid crystal display (LCD), a light-emitting diode (LED) display, a touch screen, a speaker, a vibration generator, or other devices configured to provide output to a user.

[0155] The communication device 1270 communicates with external devices via a wired or wireless network. For example, the communication device 1270 may use wired or wireless communication schemes (e.g., Bluetooth, Wi-Fi, 3G, or LTE) to communicate with external devices.

[0156] Figures 1 to 12The first generation models 420, 515, 615, and 715, the second generation models 450, 535, 630, and 730, the third generation model 650 and 750, processors 1120 and 1210, memory 1130 and 1220, communication buses 1140 and 1280, storage device 1240, input device 1250, output device 1260, communication device 1270, and other components and devices that perform the operations described in this application are implemented by hardware components configured to perform the operations described in this application. Examples of hardware components that can be used to perform the operations described in this application include, where appropriate, controllers, sensors, generators, drivers, memory, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components performing the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). A processor or computer may be implemented by one or more processing elements, such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field-programmable gate arrays, programmable logic arrays, microprocessors, or any other means or combination of means configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, the processor or computer includes or is connected to one or more memories storing instructions or software executed by the processor or computer. Hardware components implemented by the processor or computer may execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) for performing the operations described herein. The hardware components may also access, manipulate, process, create, and store data in response to the execution of instructions or software. For simplicity, the singular terms “processor” or “computer” may be used in the description of the examples described herein, but in other examples, multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or processors and controllers, and one or more other hardware components may be implemented by one or more other processors, or another processor and another controller. One or more processors, or processors and controllers, may implement a single hardware component, or two or more hardware components.The hardware components can have any one or more different processing configurations, examples of which include: a single processor, a discrete processor, a parallel processor, a single instruction single data (SISD) multiprocessing, a single instruction multiple data (SIMD) multiprocessing, multiple instruction single data (MISD) multiprocessing, and multiple instruction multiple data (MIMD) multiprocessing.

[0157] Figures 1 to 12 The methods for performing the operations described in this application, as shown, are executed by computing hardware (e.g., one or more processors or a computer), which is implemented as the execution instructions or software described above to perform the operations performed by the methods described in this application. For example, a single operation or two or more operations may be executed by a single processor, or two or more processors, or a processor and a controller. One or more operations may be executed by one or more processors, or a processor and a controller, and one or more other operations may be executed by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, may execute a single operation, or two or more operations.

[0158] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above are written as computer programs, code segments, instructions, or any combination thereof to individually or collectively instruct or configure one or more processors or computers to operate as machines or special-purpose computers to perform operations performed by the hardware components and methods described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) that is directly executed by one or more processors or computers. In another example, the instructions or software include high-level code that is executed by one or more processors or computers using an interpreter. The instructions or software can be written using any programming language based on the block diagrams and flowcharts shown in the accompanying drawings and the corresponding descriptions in the specification, which disclose algorithms for performing operations performed by the hardware components and methods described above.

[0159] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above, along with any associated data, data files, and data structures, may be recorded, stored, or fixed in, or on, one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), random access memory (RAM), flash memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store instructions or software and any associated data, data files, and data structures in a non-transitory manner and to provide the instructions or software and any associated data, data files, and data structures to one or more processors or computers, enabling one or more processors or computers to execute the instructions. In one example, instructions or software, along with any associated data, data files, and data structures, are distributed across a networked computer system, such that the instructions or software, along with any associated data, data files, and data structures, are stored, accessed, and executed in a distributed manner by one or more processors or computers.

[0160] While this disclosure includes specific examples, it will be clear upon understanding this disclosure that various changes in form and detail may be made to these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein are to be considered descriptive only and not for limiting purposes. The description of features or aspects in each example is to be considered applicable to similar features or aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order, and / or if components in the described system, architecture, apparatus, or circuit are combined in a different manner and / or replaced or supplemented by other components or their equivalents. Therefore, the scope of this disclosure is not limited by the specific embodiments but by the claims and their equivalents, and all variations within the scope of the claims and their equivalents shall be construed as included in this disclosure.

Claims

1. A method for depth image generation, comprising: receiving an input image; generating a first low-resolution image having a resolution lower than that of the input image; obtaining a first depth residual image corresponding to the input image by using a first generative model based on a first neural network; generating a first low-resolution depth image corresponding to the first low-resolution image by using a second generative model based on a second neural network; and generating a target depth image corresponding to the input image based on the first depth residual image and the first low-resolution depth image, wherein the first low-resolution depth image includes depth information of low-frequency components compared to the first depth residual image. The step of generating the target depth image includes:

2. The method of claim 1, wherein, upsampling the first low-resolution depth image to the resolution of the input image; and generating the target depth image by combining depth information of the upsampled first low-resolution depth image and depth information of the first depth residual image. The step of generating the first low-resolution depth image includes:

3. The method of claim 1, wherein, obtaining a second depth residual image corresponding to the first low-resolution image using the second generative model; generating a second low-resolution image having a resolution lower than that of the first low-resolution image; obtaining a second low-resolution depth image corresponding to the second low-resolution image using a third generative model based on a third neural network; and generating the first low-resolution depth image based on the second depth residual image and the second low-resolution depth image. The step of generating the second low-resolution image includes down-sampling the first low-resolution image to generate the second low-resolution image.

4. The method of claim 3, wherein, The step of generating the first low-resolution depth image includes:

5. The method of claim 3, wherein, upsampling the second low-resolution depth image to the resolution of the second depth residual image; and generating the first low-resolution depth image by combining depth information of the upsampled second low-resolution depth image and depth information of the second depth residual image. The resolution of the second low-resolution depth image is lower than that of the first low-resolution depth image.

6. The method of claim 3, wherein, The second depth residual image includes depth information of high-frequency components compared to the second low-resolution depth image.

7. The method of claim 3, wherein, The step of generating the first low-resolution image includes down-sampling the input image to generate the first low-resolution image.

8. The method of claim 1, wherein, The input image includes: a color image or an infrared image.

9. The method of claim 1, wherein, The input image includes: a color image and an input depth image, and 10. The method of claim 1, wherein, wherein, in the obtaining of the first depth residual image, the first generative model uses pixel values of the color image and pixel values of the input depth image as input, and outputs pixel values of the first depth residual image. The input image includes: an infrared image and an input depth image, and 11. The method of claim 1, wherein, wherein, in the obtaining of the first depth residual image, the first generative model uses pixel values of the infrared image and pixel values of the input depth image as input, and outputs pixel values of the first depth residual image. 12.A method for depth image generation, the method comprising: receiving an input image; obtaining a first depth residual image and a first low-resolution depth image by using a generative model based on a neural network using the input image as input; and ​ ​ generating a target depth image corresponding to the input image based on the first depth residual image and the first low-resolution depth image, wherein the first low-resolution depth image includes depth information of low frequency components compared to the first depth residual image.

13. The method of claim 12, wherein, The step of obtaining the first depth residual image and the first low-resolution depth image includes: obtaining a second depth residual image and a second low-resolution depth image using a generative model; and generating the first low-resolution depth image based on the second depth residual image and the second low-resolution depth image.

14. The method of claim 13, wherein, The generative model uses the input image as input and outputs the first depth residual image, the second depth residual image, and the second low-resolution depth image.

15. The method of claim 12, wherein, The generative model includes a single neural network model.

16. A method for depth image generation, the method comprising: receiving an input image; obtaining an intermediate depth image having the same size using a generative model based on a neural network using the input image as input; and generating a target depth image by combining the obtained intermediate depth image, wherein the intermediate depth image includes depth information of different frequency components.

17. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 16.

18. An apparatus for depth image generation, comprising: a processor configured to: receive an input image; generate a first low-resolution image having a resolution lower than a resolution of the input image; obtain a first depth residual image corresponding to the input image by using a first generative model based on a first neural network; generate a first low-resolution depth image corresponding to the first low-resolution image by using a second generative model based on a second neural network; and generate a target depth image corresponding to the input image based on the first depth residual image and the first low-resolution depth image, wherein the first low-resolution depth image includes depth information of low frequency components compared to the first depth residual image. The processor is further configured to: upsample the first low-resolution depth image to the resolution of the input image; and 19. The apparatus of claim 18, wherein, generate the target depth image by combining depth information of the upscaled first low-resolution depth image and depth information of the first depth residual image. The step of combining the depth information of the upscaled first low-resolution depth image and the depth information of the first depth residual image includes calculating a weighted sum or a total sum of depth values of pixel positions corresponding to each other in the first depth residual image and the upscaled first low-resolution depth image. The processor is further configured to: obtain a second depth residual image corresponding to the first low-resolution image using the second generative model; 20. The apparatus of claim 19, wherein, generate a second low-resolution image having a resolution lower than a resolution of the first low-resolution image; 21. The apparatus of claim 18, wherein, obtain a second low-resolution depth image corresponding to the second low-resolution image using a third generative model based on a third neural network; and generate the first low-resolution depth image based on the second depth residual image and the second low-resolution depth image. The processor is further configured to: ​ ​ ​ 22. The apparatus of claim 21, wherein, ​ up-sampling the second low-resolution depth image to a resolution of the second depth residual image; and generating the first low-resolution depth image by combining depth information of the up-sampled second low-resolution depth image and depth information of the second depth residual image.

23. The apparatus of claim 22, wherein, The step of combining the depth information of the up-sampled second low-resolution depth image and the depth information of the second depth residual image includes calculating a weighted sum or a total of depth values of pixel positions corresponding to each other in the second depth residual image and the up-sampled second low-resolution depth image.

24. The apparatus of claim 21, wherein, The resolution of the first low-resolution depth image is higher than the resolution of the second low-resolution depth image, and wherein the second depth residual image includes depth information of high-frequency components compared to the second low-resolution depth image.

25. The apparatus of claim 18, wherein, The processor is further configured to down-sample the input image to generate the first low-resolution image.

26. The apparatus of claim 18, wherein, The input image includes a color image and an input depth image, and wherein, in the acquisition of the first depth residual image, the first generative model uses pixel values of the color image and pixel values of the input depth image as input, and outputs pixel values of the first depth residual image.

27. The apparatus of claim 18, wherein, The input image includes an infrared image and an input depth image, and wherein, in the acquisition of the first depth residual image, the first generative model uses pixel values of the infrared image and pixel values of the input depth image as input, and outputs pixel values of the first depth residual image.

28. The device of claim 18, further comprising: a sensor configured to acquire the input image, wherein the input image includes any one or both of a color image and an infrared image.

29. A device for depth image generation, comprising: a processor configured to: receive an input image; acquire a first depth residual image and a first low-resolution depth image by using a generative model based on a neural network using the input image as input; and generate a target depth image corresponding to the input image based on the first depth residual image and the first low-resolution depth image, wherein the first low-resolution depth image includes depth information of low-frequency components compared to the first depth residual image. The processor is further configured to:

30. The apparatus of claim 29, wherein, acquire a second depth residual image and a second low-resolution depth image using the generative model; and generate the first low-resolution depth image based on the second depth residual image and the second low-resolution depth image. The first low-resolution depth image has a resolution lower than that of the input image, and the second low-resolution depth image has a resolution lower than that of the first low-resolution depth image.

31. The apparatus of claim 30, wherein, 32. A device for depth image generation, comprising: a processor configured to: receive an input image: acquire intermediate depth images having the same size by using a generative model based on a neural network using the input image as input; and generate a target depth image by combining the acquired intermediate depth images, wherein the acquired intermediate depth images include depth information of different frequency components. The step of combining the acquired intermediate depth images includes calculating a weighted sum or a total of depth values of pixel positions corresponding to each other in the acquired intermediate depth images.

33. The apparatus of claim 32, wherein, The step of combining the acquired intermediate depth images includes calculating a weighted sum or a total of depth values of pixel positions corresponding to each other in the acquired intermediate depth images.

Images