Method and apparatus with image processing

Abstract

A method and apparatus having image processing are disclosed. The method having image processing includes generating a first surface normal image including surface normal vectors corresponding to pixels of a first depth image, and applying the first depth image and the first surface normal image to a first neural network and acquiring a second depth image by changing the first depth image using the first neural network. The first neural network generates the second depth image having improved quality compared to the first depth image based on an embedding vector including a feature of the first depth image and a feature of the first surface normal image.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2020-0175858, filed on December 15, 2020, with the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference for all purposes. Technical Field

[0002] The following description relates to methods and apparatus for processing images. For example, the description relates to methods and apparatus for processing three-dimensional (3D) depth images. Background Technology

[0003] A depth image is a three-dimensional (3D) image that includes information about the distance between the camera and the subject, and can be acquired by a depth camera (such as a time-of-flight (TOF) camera) that measures the distance between the subject and the camera.

[0004] The Time-of-Flight (TOF) method measures distance by emitting a strong light from a camera in front of it and then detecting the reflected light. TOF cameras use light-emitting diodes (LEDs) or laser diodes (LDs) to project light of a specific wavelength (e.g., near-infrared light at 850 nanometers (nm)) onto a subject. A photodiode or camera measures or captures the light of the same wavelength reflected from the subject, and the phase difference between the incident light signal and the signal of the light reflected from the subject and returning within the integration time of each frame is calculated to generate a depth image.

[0005] If the camera or subject moves within a time interval shorter than the integration time, the reflected light signal can change in phase. In this case, depth measurement errors can occur due to external environmental factors and imaging conditions, including the material, color, distance, and illumination of the object being imaged. To address this issue, techniques for correcting depth images are being developed, such as weighted least squares (WLS) filters for removing noise from depth images acquired via a TOF camera. Summary of the Invention

[0006] The present invention is provided in a brief form to introduce the choice of concepts further described in the following detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to help determine the scope of the claimed subject matter.

[0007] In one general aspect, a method for image processing includes: generating a first surface normal image comprising surface normal vectors corresponding to pixels of a first depth image; and applying the first depth image and the first surface normal image to a first neural network, and obtaining a second depth image by modifying the first depth image using the first neural network. The first neural network generates the second depth image with improved quality compared to the first depth image based on an embedding vector comprising features of the first depth image and features of the first surface normal image.

[0008] The first neural network can remove noise included in the first depth image and improve the clarity of the first depth image.

[0009] The first depth image may include a time-of-flight (TOF) image.

[0010] The method may further include: generating a second surface normal image including surface normal vectors corresponding to pixels of the second depth image; and applying the second depth image and the second surface normal image to a second neural network, and obtaining a third depth image by modifying the second depth image using the second neural network. The second neural network may generate a third depth image of degraded quality compared to the second depth image based on an embedding vector including features of the second depth image and features of the second surface normal image.

[0011] The image processing method may further include generating training data by pairing a second depth image and a third depth image.

[0012] The first and second neural networks can be trained using a recurrent generative adversarial network (GAN) training method.

[0013] In another general aspect, a non-transitory computer-readable storage medium stores instructions that, when executed by a processor, cause the processor to perform the methods described above.

[0014] In another general aspect, a training method includes: acquiring a second depth image of a second type by applying a first depth image of a first type and a first surface normal image corresponding to the first depth image to a first neural network; acquiring a third depth image of the first type by applying the acquired second depth image and a second surface normal image corresponding to the second depth image to a second neural network; acquiring a first generative adversarial network (GAN) loss related to whether the second depth image is an image generated by the first neural network; acquiring a first cycle consistency loss related to the difference between the third depth image and the first depth image; and training the first neural network based on the first GAN loss and the first cycle consistency loss.

[0015] The steps of training the first neural network may include: obtaining a fifth depth image of a first type by applying a fourth depth image of a second type and a fourth surface normal image corresponding to the fourth depth image to the second neural network; obtaining a sixth depth image of a second type by applying the obtained fifth depth image and a fifth surface normal image corresponding to the fifth depth image to the first neural network; obtaining a second GAN loss related to whether the fifth depth image is an image generated by the second neural network; obtaining a second cycle consistency loss related to the difference between the sixth depth image and the fourth depth image; and training the first neural network and the second neural network based on the first GAN loss, the second GAN loss, the first cycle consistency loss and the second cycle consistency loss.

[0016] The training method may further include: obtaining a first identity (ID) loss related to the difference between a first depth image and a depth image obtained by applying the first depth image and a first surface normal image to a second neural network; obtaining a second ID loss related to the difference between a fourth depth image and a depth image obtained by applying the fourth depth image and a fourth surface normal image to a first neural network; and further training the first neural network and the second neural network based on the first ID loss and the second ID loss.

[0017] The training method may further include: training a first neural network and a second neural network based on a first range-preserving loss for limiting the difference between a first depth image and a second depth image and a second range-preserving loss for limiting the difference between a fourth depth image and a fifth depth image.

[0018] The training method may further include: training a second neural network based on a loss used to distinguish the difference between the fourth depth image and the fifth depth image and the difference between the fourth depth image and the first type of real depth image corresponding to the fourth depth image.

[0019] The first neural network may include a pre-trained neural network trained to generate a second type of depth image by modifying a first type of depth image along a first direction. The second neural network may include a pre-trained neural network trained to generate a first type of depth image by modifying a second type of depth image along a second direction opposite to the first direction.

[0020] The first type of depth image may include a noisy depth image. The second type of depth image may include a noise-free depth image. The first neural network may be configured to generate a noise-removed depth image by removing noise from the input depth image. The second neural network may be configured to generate a noise-added depth image by adding noise to another input depth image.

[0021] In another general aspect, a non-transitory computer-readable storage medium stores instructions that, when executed by a processor, cause the processor to perform the methods described above.

[0022] In another general aspect, an image processing apparatus includes: at least one processor configured to: generate a first surface normal image including surface normal vectors corresponding to pixels of a first depth image; apply the first depth image and the first surface normal image to a first neural network; and obtain a second depth image by modifying the first depth image using the first neural network. The first neural network may be configured to: generate a second depth image of improved quality compared to the first depth image based on an embedding vector including features of the first depth image and features of the first surface normal image.

[0023] The first neural network can also be configured to remove noise included in the first depth image and improve the clarity of the first depth image.

[0024] The first depth image may include a time-of-flight (TOF) image.

[0025] The processor can also be configured to: generate a second surface normal image including surface normal vectors corresponding to pixels of the second depth image; apply the second depth image and the second surface normal image to a second neural network; and obtain a third depth image by modifying the second depth image using the second neural network. The second neural network can be configured to: generate a third depth image of degraded quality compared to the second depth image based on an embedding vector including features of the second depth image and features of the second surface normal image.

[0026] The processor can also be configured to generate training data by pairing second and third depth images.

[0027] The first and second neural networks can be trained using a recurrent generative adversarial network (GAN) training method.

[0028] In another general aspect, an image processing apparatus includes: at least one processor configured to: acquire a second depth image with weak noise by applying a first depth image with strong noise and a first surface normal image corresponding to the first depth image to a first neural network; acquire a third depth image with strong noise by applying the acquired second depth image and a second surface normal image corresponding to the second depth image to a second neural network; acquire a first generative adversarial network (GAN) loss related to whether the second depth image is an image generated by the first neural network; acquire a first cycle consistency loss related to the difference between the third depth image and the first depth image; and train the first neural network based on the first GAN loss and the first cycle consistency loss.

[0029] The steps of training the first neural network may include: obtaining a fifth depth image with strong noise by applying a fourth depth image with weak noise and a fourth surface normal image corresponding to the fourth depth image to a second neural network; obtaining a sixth depth image with weak noise by applying the obtained fifth depth image and a fifth surface normal image corresponding to the fifth depth image to the first neural network; obtaining a second GAN loss related to whether the fifth depth image is generated by the second neural network; obtaining a second cycle consistency loss related to the difference between the sixth depth image and the fourth depth image; and training the first neural network and the second neural network based on the first GAN loss, the second GAN loss, the first cycle consistency loss and the second cycle consistency loss.

[0030] The at least one processor may also be configured to: acquire a first identity (ID) loss relating to the difference between a first depth image and a depth image acquired by applying the first depth image and a first surface normal image to a second neural network; acquire a second ID loss relating to the difference between a fourth depth image and a depth image acquired by applying the fourth depth image and a fourth surface normal image to a first neural network; and further train the first neural network and the second neural network based on the first ID loss and the second ID loss.

[0031] The at least one processor may also be configured to: further train a first neural network and a second neural network based on a first range-preserving loss for limiting the difference between a first depth image and a second depth image and a second range-preserving loss for limiting the difference between a fourth depth image and a fifth depth image.

[0032] The at least one processor may also be configured to train a second neural network based on a loss used to distinguish the difference between the fourth depth image and the fifth depth image and the difference between the fourth depth image and the first type of real depth image corresponding to the fourth depth image.

[0033] Other features and aspects will become clear from the following detailed description, drawings, and claims. Attached Figure Description

[0034] Figure 1 An example of an image processing system for acquiring enhanced depth images is shown.

[0035] Figure 2 An example of a training method for a neural network used to acquire enhanced depth images is shown.

[0036] Figure 3 Examples of training methods for enhanced convolutional neural networks (CNNs) and noise-simulated CNNs are shown.

[0037] Figure 4A and Figure 4B Examples of loss functions used for training augmented CNNs and noisy simulated CNNs are shown respectively.

[0038] Figure 5 An example of a loss function used to train a noisy simulated CNN is shown.

[0039] Figure 6A and Figure 6B Examples of methods for using pre-trained models to train augmented CNNs and noisy simulated CNNs are shown.

[0040] Figure 7 An example of an image processing device is shown.

[0041] Throughout the accompanying drawings and detailed embodiments, unless otherwise described or provided, the same reference numerals will be understood to denote the same elements, features, and structures. The drawings may not be to scale, and for clarity, illustration, and convenience, the relative sizes, proportions, and depictions of elements in the drawings may be exaggerated. Detailed Implementation

[0042] The following detailed description is provided to assist the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, upon understanding the disclosure of this application, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, except for those that must occur in a specific order, but may be changed as will become clear upon understanding the disclosure of this application. Furthermore, for clarity and brevity, descriptions of features known in the art may be omitted.

[0043] The features described herein may be implemented in different forms and should not be construed as limited to the examples described herein. Rather, the examples described herein have been provided merely to illustrate some of the many feasible ways of implementing the methods, apparatus, and / or systems described herein that will be clear upon understanding the disclosure of this application.

[0044] The structural or functional descriptions of the examples disclosed below are intended for illustrative purposes only, and the examples may be implemented in various forms. The examples are not intended to be limiting, but rather to include various modifications, equivalents, and substitutions within the scope of the claims.

[0045] Although the terms “first” or “second” are used to describe various components, the components are not limited to the terms. These terms should only be used to distinguish one component from another. For example, within the scope of the claims based on the concept of this disclosure, a “first” component may be referred to as a “second” component, or similarly, a “second” component may be referred to as a “first” component.

[0046] It will be understood that when a component is referred to as being "connected to" another component, the component may be directly connected to or combined with the other component, or there may be an intermediate component.

[0047] As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. It should also be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of the stated features, integrals, steps, operations, elements, components, or combinations thereof, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof.

[0048] Unless otherwise defined, all terms used herein (including technical or scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which the examples pertain. It will also be understood that, unless expressly defined herein, terms (such as those defined in a general dictionary) shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field and shall not be interpreted in an idealized or overly formalized manner.

[0049] The examples will be described in detail below with reference to the accompanying drawings. Regarding the reference numerals assigned to the elements in the drawings, it should be noted that the same elements will be represented by the same reference numerals, and redundant descriptions will be omitted.

[0050] Figure 1 An example of an image processing system for acquiring enhanced depth images is shown.

[0051] Reference Figure 1The image processing system may include, for example, a surface normal image generation module 110 and a neural network 120. The surface normal image generation module 110 can generate a surface normal image 102 corresponding to the input depth image 101. The neural network 120 can receive the input depth image 101 and the surface normal image 102 corresponding to the input depth image 101 as input, and generate an enhanced depth image 103.

[0052] A depth image is an image that includes depth information associated with the distance from the viewpoint to the surface of an object, and is a three-dimensional (3D) image that includes depth information in addition to the two-dimensional (2D) positional information of each pixel that makes up the image. A depth image can be, for example, an image acquired using a time-of-flight (TOF) sensor camera (hereinafter referred to as a TOF image) or an image acquired using a laser scanner (hereinafter referred to as an LS image). A TOF sensor camera provides a pixel-by-pixel depth value of a photographed subject based on the TOF, which is the reflection and return of an optical signal emitted to the subject at the subject. Compared to LS images, TOF images can be generated by measuring depth information much faster. However, when compared to LS images, TOF images have a larger error in depth measurement and include relatively strong noise due to errors in depth measurement, etc. LS images are generated by accurately measuring depth; therefore, compared to TOF images, LS images have no noise or have very little noise. A depth image that is input into an image processing system to obtain an enhanced depth image can correspond to a noisy depth image (such as a TOF image). Hereinafter, a depth image may also be simply referred to as an image.

[0053] The surface normal image generation module 110 calculates the surface normal vector corresponding to each pixel of the input depth image 101 and images the calculated surface normal vectors to generate a surface normal image 102 corresponding to the input depth image 101. The surface normal image corresponding to the depth image is obtained by imaging the surface normal vectors corresponding to the pixels of the depth image. A surface normal vector represents a unit vector perpendicular to the tangent plane of a point on the surface. The surface normal image generation module 110 can obtain the surface normal vector by calculating the tangent plane of each pixel based on the depth information of the input depth image 101, and then images the surface normal vectors of the pixels of the input depth image 101 to generate the surface normal image 102.

[0054] Neural network 120 can correspond to a learning model that receives an input depth image 101 and a surface normal image 102 corresponding to the input depth image 101 as input and generates an enhanced depth image 103. The input depth image 101 and the surface normal image 102 can be concatenated and then input into neural network 120. Neural network 120 can generate the enhanced depth image 103 based on an embedding vector including the features of the input depth image 101 and the features of the surface normal image 102. In other words, neural network 120 can modify the features of the input depth image 101 based on the features extracted by encoding the input depth image 101 and the surface normal image 102, decode the modified features, and generate the enhanced depth image 103.

[0055] The enhanced depth image 103 is an image after the input depth image 101 has been modified, and may correspond to an image in which some features of the input depth image 101 are altered by reflecting features extracted from the surface normal image 102 in the neural network 120. The enhanced depth image 103 may correspond to an image modified to improve some features of the input depth image 101 (i.e., an image with improved quality compared to the input depth image 101). For example, the enhanced depth image 103 may correspond to an image in which noise in the input depth image 101 has been removed or whose quality (such as sharpness) has been improved.

[0056] For example, neural network 120 may correspond to a learning model configured as a convolutional neural network (CNN). Neural network 120 may include multiple layers. The parameters between layers can be determined through learning. In neural network 120, computational operations can be performed on the input depth image 101 and the surface normal image 102 based on the parameters between the layers, thereby generating an enhanced depth image 103.

[0057] Although not in Figure 1 As shown, however, in addition to neural network 120, the image processing system may also include another neural network that modifies the depth image in a manner opposite to that of neural network 120. While neural network 120 is a network that generates an image of improved quality compared to the input image, the other neural network may correspond to a network that generates a depth image of degraded quality compared to the input depth image. In the following text, to distinguish between neural network 120 and the other neural network, neural network 120 that generates the enhanced depth image may be referred to as the first neural network, and the neural network that generates the depth image in a manner opposite to that of the first neural network may be referred to as the second neural network.

[0058] Similar to the first neural network, the second neural network can receive a depth image and a corresponding surface normal image as input. The second neural network can generate a depth image of degraded quality based on an embedding vector that includes features from the input depth image and features from the input surface normal image. In this case, the second neural network can modify some features of the input depth image by reflecting features extracted from the input surface normal image. The features of the input depth image can be modified in the opposite direction to the modification direction in the first neural network. The second neural network can correspond to the concept of an inverse function of the first neural network.

[0059] For example, a first neural network can generate a depth image from which an input depth image is altered along a first direction. In this example, the first direction is the direction in which a first type of depth image is altered into a second type of depth image. Simultaneously, a second neural network can generate a depth image from which an input depth image is altered along a second direction opposite to the first direction. The second direction may correspond to the direction in which a second type of depth image is altered into a first type of depth image.

[0060] For example, the first type could correspond to an image with strong noise, and the second type could correspond to a clean image without noise. In this example, the first neural network could correspond to a network trained to generate a denoised image from a depth image with strong noise. Furthermore, the second neural network could correspond to a network trained to generate an image that simulates noise from a clean depth image.

[0061] For example, the first and second neural networks can be trained based on a recurrent generative adversarial network (GAN) learning method. The first neural network can be trained such that the depth image generated by the first neural network is recovered as the depth image input to the first neural network via the second neural network. The second neural network can be trained such that the depth image generated by the second neural network is recovered as the depth image input to the second neural network via the first neural network. The method for training the first and second neural networks will be described later.

[0062] A depth image can be generated in which two or more features are altered by a first neural network and a second neural network. For example, the first neural network can output a depth image obtained by removing noise and improving sharpness from the input depth image. In this example, the second neural network can output a depth image obtained by adding noise to the input depth image and reducing sharpness. In other words, the first and second neural networks can correspond to networks that generate images by altering two or more features of the input depth image separately along different directions. One or more features of the depth image altered by the first and second neural networks can be correlated with one or more features of the surface normal image input along with the depth image.

[0063] In the following description, the first neural network that outputs an enhanced depth image may also be referred to as an augmentation CNN, and the second neural network that outputs a depth image that changes in the opposite direction compared to the first neural network may also be referred to as a noise-simulating CNN. The first type of depth image can be an image with strong noise, and the second type of depth image can be a clean image with weak noise. The augmentation CNN may correspond to a learning model that receives a TOF image with strong noise and a surface normal image corresponding to the TOF image and outputs a clean, denoised depth image. The noise-simulating CNN may be a learning model that receives a clean LS image with weak noise and a surface normal image corresponding to the LS image and outputs a noisy depth image. In one example, strong noise may represent noise greater than a first threshold, weak noise may represent noise less than a second threshold, and the first threshold is greater than or equal to the second threshold.

[0064] Figure 2 An example of a training method for a neural network used to obtain enhanced depth images is shown.

[0065] Reference Figure 2 A training method for a neural network used to acquire enhanced depth images may include the following operations: acquiring a second-type second-depth image 202 by applying a first-type first depth image 201 and a first surface normal image 211 corresponding to the first depth image 201 to a first neural network (hereinafter referred to as "enhanced CNN") 210; acquiring a first-type third-depth image 203 by applying the acquired second-depth image 202 and a second surface normal image 212 corresponding to the second depth image 202 to a second neural network 220; acquiring a GAN loss 221 related to whether the second depth image 202 was generated by the first neural network 210; acquiring a cycle consistency loss 231 related to the difference between the third depth image 203 and the first depth image 201; and training the first neural network 210 based on the GAN loss 221 and the cycle consistency loss 231. In one example, training data may be generated by pairing the second depth image 202 and the third depth image 203.

[0066] An augmented CNN (e.g., a first neural network 210) can be trained based on the GAN loss 221. The GAN loss 221 is a loss related to whether the image at a predetermined depth is an image generated in the augmented CNN 210, and can be obtained based on the result of verifying whether the image at the predetermined depth is a denoised image generated in the augmented CNN 210 or a clean image without noise.

[0067] The enhanced quality discriminator 230 is a discriminator corresponding to the enhanced CNN 210 as a generator, and can correspond to a network that verifies whether the input depth image is a genuine clean depth image captured by a laser scanner or a fake clean depth image generated by the enhanced CNN 210. A genuine clean depth image can be a clean depth image that does not inherently contain noise, rather than an image obtained through processing (such as noise removal). A fake clean depth image can be a depth image that has been made clean by performing noise removal processing.

[0068] When the input image is a genuine, clean depth image, the enhanced quality discriminator 230 can be trained to determine that the input image is genuine. Furthermore, when the input image is a fake, clean depth image generated by applying a TOF image to the enhanced CNN 210, the enhanced quality discriminator 230 can be trained to determine that the input image is fake. Based on the output of the enhanced quality discriminator 230, the GAN loss function can be configured. Based on the GAN loss, the enhanced CNN 210 as the generator and the discriminator 230 can be trained.

[0069] The generator 210 and the discriminator 230 can be trained during adversarial competition based on the GAN loss 221. Based on the GAN loss 221, the discriminator 230 can be trained to distinguish between real clean images and fake clean images generated in the generator 210, and the generator 210 can be trained to generate depth images that are close to real clean depth images.

[0070] The augmented CNN 210 can be trained based on a cycle consistency loss 231. The cycle consistency loss 231 is the loss used to control the augmented CNN 210 to alter the depth image so that the second depth image 202 generated in the augmented CNN 210 is restored to the first depth image 201 as the original image. Furthermore, the cycle consistency loss 231 can be obtained based on the difference between a third depth image 203, which is the output image of the noisy simulation CNN 220, and the first depth image 201, which is the input image of the augmented CNN 210. The third depth image 203 can be obtained by inputting the second depth image 202, which is the output of the augmented CNN 210, into the noisy simulation CNN 220.

[0071] The noise-simulated CNN 220 can receive a second depth image 202 output from the enhanced CNN and a second surface normal image 212 corresponding to the second depth image 202, and output a third depth image 203 as a noisy image. Although Figure 2Not shown, but the second depth image 202 output from the enhanced CNN 210 can be input to the surface normal image generation module to obtain a second surface normal image 212 based on the depth information of the second depth image 202. In other words, the training method may also include the following operations: calculating the surface normal vector corresponding to each pixel of the second depth image 202 output from the enhanced CNN 210, and imaging the calculated surface normal vectors to obtain the second surface normal image 212.

[0072] Figure 2 The diagram illustrates the process of training an augmentation CNN 210 based on a cycle consistency loss (hereinafter also referred to as the "first cycle loss") 231 obtained by inputting a second depth image 202 into a noisy simulation CNN 220, wherein the second depth image 202 is generated by pre-inputting the GAN loss (hereinafter also referred to as the "first GAN loss") 221 of the augmentation CNN 210 and the first depth image 201 into the augmentation CNN 210. In addition to the augmentation CNN 210, the training method may also include the operation of training the noisy simulation CNN 220. The training method may include the operation of training the noisy simulation CNN 220 based on a cycle consistency loss (hereinafter also referred to as the "second cycle loss") obtained by inputting an output image into the augmentation CNN 210, wherein the output image is generated by pre-inputting the GAN loss (hereinafter also referred to as the "second GAN loss") of the noisy simulation CNN 220 and the depth image into the noisy simulation CNN 220. (Refer to...) Figure 3 Describe relevant examples.

[0073] Figure 3 Examples of training methods for augmented CNNs and noise-simulated CNNs are shown.

[0074] Reference Figure 3 The augmented CNN 310 and the noisy simulated CNN 320 can be trained using the recurrent GAN learning method, based on the first GAN loss 321, the second GAN loss 322, the first recurrent loss 331, and the second recurrent loss 332. Figure 3 In the process, the first depth image 301, the first surface normal image 311, the second depth image 302, the third depth image 303, and the enhancement quality discriminator 330 are respectively compared with... Figure 2 The first depth image 201, the first surface normal image 211, the second depth image 202, the third depth image 203, and the enhanced quality discriminator 230 described in the embodiments correspond to each other.

[0075] To obtain the second GAN loss and the second recurrent loss, the training method may include the following operations: obtaining a first-type fifth-type depth image 305 by applying a second-type fourth-depth image 304 and a fourth surface normal image 314 corresponding to the fourth-depth image 304 to a second neural network 320 (hereinafter referred to as "noise-simulated CNN"); obtaining a second-type sixth-depth image 306 by applying the obtained fifth-depth image 305 and a fifth surface normal image corresponding to the fifth-depth image 305 to a first neural network 310 (hereinafter referred to as "enhanced CNN"); obtaining a second GAN loss 322 related to whether the fifth-depth image 305 is an image generated by the second neural network 320; and obtaining a second recurrent loss 332 related to the difference between the sixth-depth image 306 and the fourth-depth image 304.

[0076] The noisy simulation CNN (e.g., the second neural network 320) can be trained based on the second GAN loss 322. The second GAN loss 322 is a loss related to whether the image at a predetermined depth is an image generated in the noisy simulation CNN 320, and can be obtained based on the result of verifying whether the image at the predetermined depth is a noisy image generated in the noisy simulation CNN 320 or an image that inherently contains noise.

[0077] The noise quality discriminator 340 is a discriminator corresponding to the noise-simulating CNN 320, which acts as a generator. The discriminator 340 can correspond to a network that verifies whether the input depth image is a real noisy image captured by a TOF camera or a fake noisy image generated by the noise-simulating CNN 320. A real noisy depth image can be a depth image that inherently contains noise, rather than an image obtained through processing (such as noise simulation). A fake noisy image can be a depth image that has been made into a strongly noisy image by performing noise simulation processing.

[0078] When the input image is a real noisy image, the noise quality discriminator 340 can be trained to determine that the input image is real. Furthermore, when the input image is a fake noisy image generated by applying an LS image to a noisy simulation CNN 320, the noise quality discriminator 340 can be trained to determine that the input image is fake. Based on the output of the noise quality discriminator 340, a second GAN loss 322 can be configured. Based on the second GAN loss 322, the noisy simulation CNN 320 as a generator and the noise quality discriminator 340 can be trained.

[0079] The generator 320 and the discriminator 340 can be trained during adversarial competition based on the second GAN loss 322. Based on the second GAN loss 322, the discriminator 340 can be trained to distinguish between real noisy images and fake noisy images generated in the generator 320, and the generator 320 can be trained to generate depth images that are close to real noisy depth images.

[0080] The noisy simulation CNN 320 can be trained based on a second recurrent loss 332. The second recurrent loss 332 is a loss used to control the changes in the depth image within the noisy simulation CNN 320, such that the fifth depth image 305 generated in the noisy simulation CNN 320 is restored to the fourth depth image 304 as the original image. Furthermore, the second recurrent loss 332 can be obtained based on the difference between the sixth depth image 306, which is the output image of the enhancement CNN 310, and the fourth depth image 304, which is the input image of the noisy simulation CNN 320. The sixth depth image 306 can be obtained by inputting the fifth depth image 305, output from the noisy simulation CNN 320, into the enhancement CNN 310.

[0081] although Figure 3 Not shown, but the enhanced CNN 310 can receive a fifth depth image 305 output from the noisy simulation CNN 320 and a surface normal image corresponding to the fifth depth image 305 as input, and output a sixth depth image 306 with noise removed. The fifth depth image 305 output from the noisy simulation CNN 320 can be input to the surface normal image generation module to obtain a surface normal image based on the depth information of the fifth depth image 305. In other words, the training method may also include the following operations: calculating the surface normal vector corresponding to each pixel of the fifth depth image 305 output from the noisy simulation CNN 320, and imaging the surface normal vectors to obtain the surface normal image.

[0082] Figure 4A and Figure 4B Examples of loss functions used for training augmented CNNs and noisy simulated CNNs are shown.

[0083] Augmented CNNs and noise-simulated CNNs can be trained based on identity (ID) loss. ID loss is used to provide stable guidance in image generation. Furthermore, ID loss is the loss used to train the neural network to output data identical to the input data when data corresponding to a predetermined type is input to the neural network (the neural network output is the result of transforming the input data into data of the predetermined type).

[0084] Reference Figure 4AThe first ID loss 441 is a loss used to train the augmented CNN 410 to generate an output image 402 identical to the clean image 401 in response to the clean image 401 being input to the augmented CNN 410 outputting a clean depth image. The first ID loss 441 can be obtained based on the difference between the clean image 401 input to the augmented CNN 410 and the output image 402. Based on the first ID loss 441, the augmented CNN 410 can be trained to reduce the difference between the input clean image 401 and the generated output image 402.

[0085] Reference Figure 4B The second ID loss 442 is a loss used to train the noisy simulation CNN 420 to output an output image 404 identical to the noisy image 403 in response to the noisy image 403 being input to the noisy simulation CNN 420 outputting a noisy depth image. The second ID loss 442 can be obtained based on the difference between the noisy image 403 input to the noisy simulation CNN 420 and the output image 404. Based on the second ID loss 442, the noisy simulation CNN 420 can be trained to reduce the difference between the input noisy image 403 and the generated output image 404.

[0086] Figure 5 An example of a loss function used to train a noisy simulated CNN is shown.

[0087] Reference Figure 5 The noise-simulated CNN 510 can be trained based on the residual loss 551. The residual loss 551 can be obtained by the residual discriminator 520, which determines the difference between the input image 501 and the output image 502 of the noise-simulated CNN 510.

[0088] The residual discriminator 520 corresponds to a network that discriminates the difference 503 between the input image 501 and the output image 502 of the noisy simulated CNN, as well as the difference 513 between depth image pairs included in the training data. The depth image pair can be a pair of LS images 511 and TOF images 512 simultaneously captured and aligned by a laser scanner and a TOF camera. The difference 513 between the depth image pairs can be the difference between the LS image 511 and the TOF image 512 that form the depth image pair.

[0089] The residual discriminator 520 can determine whether the input data is the difference between the input and output images of the noisy simulated CNN 510 or the difference between depth image pairs included in the training data. Based on the output of the residual discriminator 520, a residual loss function can be constructed. Based on the residual loss, the residual discriminator 520 can be trained to distinguish between the difference between the input and output images of the noisy simulated CNN and the difference between depth image pairs, and the noisy simulated CNN 510 can be trained to generate an output image from the input image that approximates the difference between depth image pairs. In one example, the residual discriminator 520 can receive a surface normal image 504 of the input image 501 and a difference 503 between the input and output images of the noisy simulated CNN 510.

[0090] although Figure 5 The example shown is a noisy simulated CNN trained on residual loss, but even augmented CNNs can be trained on residual loss. The residual loss used to train the augmented CNN can be obtained through a residual discriminator that distinguishes the difference between the input and output images of the augmented CNN. The residual loss used to train the augmented CNN can correspond to a network that distinguishes the difference between the input and output images of the augmented CNN and the difference between depth image pairs included in the training data.

[0091] Augmented CNNs and noisy simulated CNNs can be trained based on range preserving loss. Range preserving loss is a loss mechanism used to limit the difference between the input and output images, and it can be obtained based on this difference. For example, an augmented CNN can be trained based on the range preserving loss obtained from the mean squared error between the input and output images of the augmented CNN. Similarly, a noisy simulated CNN can be trained based on the range preserving loss obtained from the mean squared error between the input and output images of the noisy simulated CNN.

[0092] Table 1 shows an example design of the loss function used in the training method.

[0093] [Table 1]

[0094] GAN loss <![CDATA[D1(G2(B),A)+D2(G1(A),B)]]> Cyclic loss <![CDATA[L1loss(G2(G1(A)),A)+L1loss(G1(G2(B)),B)]]> Identity loss <![CDATA[L1loss(G2(A),A)+L1loss(G1(B),B)]]> Residual loss <![CDATA[D R (B-G2(B),C)]]> Range of loss <![CDATA[MSE(G1(A),A)+MSE(G2(B),B)]]>

[0095] In Table 1, A represents a first type of depth image (e.g., a TOF image), B represents a second type of depth image (e.g., an LS image), C represents the difference between a pair of first and second type depth images (e.g., the difference between a pair formed by a TOF image and an LS image), G1 represents a first neural network (e.g., an augmented CNN), G2 represents a second neural network (e.g., a noise-simulated CNN), D1 represents a first discriminator (e.g., an augmented quality discriminator) corresponding to the first neural network as a generator, D2 represents a second discriminator (e.g., a noise quality discriminator) corresponding to the second neural network as a generator, and D... R denoted as the residual discriminator, L1loss as the function for calculating L1 loss, and MSE as the function for calculating mean squared error.

[0096] Figure 6A and Figure 6B Examples of methods for training augmented CNNs and noisy simulated CNNs using pre-trained models are shown.

[0097] Since ground truth (GT) images are lacking for use as training ground truth (TOF) images, semi-supervised learning methods can be used to train augmented CNNs and noise-simulated CNNs. For example, the augmented CNN and noise-simulated CNN can be trained using the recurrent GAN learning method described above. In this example, the trained CNN model can be pre-trained using TOF depth images and depth images denoised by weighted least squares (WLS) filters as training data.

[0098] Reference Figure 6A and Figure 6B The augmented CNN 610 can use CNN 620, trained to generate a depth image 602 obtained by removing noise from the TOF image 601 through a WLS filter, as a pre-trained model. The noise-simulating CNN 630 can use CNN 640, trained to generate an un-noise-removed TOF image 601 from the denoised image 602 through a WLS filter, as a pre-trained model. In other words, the augmented CNN 610 and the noise-simulating CNN 630, which have not yet been trained according to the recurrent GAN method, can correspond to CNN models 620 and 640 trained using the TOF image 601 and the denoised depth image 602 through a WLS filter as training data. In one example, CNN models 620 and 640 may include convolutional layers (Conv), rectified linear unit layers (ReLU), batch normalization layers (BN), etc.

[0099] The enhanced CNN 610 trained according to the above training method can receive a noisy Time-of-Flight (TOF) image and a surface normal image corresponding to the TOF image, and generate an enhanced depth image. The enhanced CNN 610 can generate depth images with excellent noise removal and improved sharpness. Furthermore, the enhanced CNN 610 can generate depth images with reduced multipath distortion. For example, the enhanced CNN 610 can generate images by reducing multipath distortion in regions with high curvature in the input image.

[0100] The noise-simulating CNN 630 trained according to the above training method can be used to generate training data for neural networks related to noise removal by generating noisy depth images corresponding to clean depth images.

[0101] Figure 7 An example of an image processing device is shown.

[0102] Reference Figure 7 The image processing device 700 may include, for example, a processor 701, a memory 703, and an input and output (I / O) device 705.

[0103] The image processing apparatus 700 may be an apparatus that performs the above-described image processing method or training method, and may include a server and a user device (e.g., a mobile phone, a computer, etc.). The processor 701 may execute the reference... Figures 1 to 6B At least one of the methods described above. Memory 703 may store information associated with the image processing method or training method, or store programs implementing the image processing method or training method described above. Memory 703 may be volatile memory or non-volatile memory.

[0104] The processor 701 can execute programs and control the image processing device 700. The code of the program executed by the processor 701 can be stored in the memory 703. The image processing device 700 can be connected to an external device (e.g., a personal computer or a network) via I / O device 705 and perform data exchange.

[0105] The devices, units, modules, and other components described herein are implemented by hardware components. Examples of hardware components that can be used to perform the operations described herein include, where appropriate, controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described herein. In other examples, one or more of the hardware components performing the operations described herein are implemented by computing hardware (e.g., by one or more processors or computers). The processor or computer may be implemented by one or more processing elements, such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field-programmable gate arrays, programmable logic arrays, microprocessors, or any other means or combination of means configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, the processor or computer includes or is connected to one or more memories storing instructions or software executed by the processor or computer. The hardware components implemented by the processor or computer can execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) for performing the operations described herein. Hardware components can also access, manipulate, process, create, and store data in response to the execution of instructions or software. For simplicity, the singular terms "processor" or "computer" are used in the description of the examples described in this application; however, in other examples, multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component, or two or more hardware components, may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, may implement a single hardware component, or two or more hardware components. Hardware components may have any one or more different processing configurations, examples of which include: a single processor, a standalone processor, a parallel processor, single-instruction single-data (SISD) multiple processing, single-instruction multiple-data (SIMD) multiple processing, multiple-instruction single-data (MISD) multiple processing, and multiple-instruction multiple-data (MIMD) multiple processing.

[0106] The methods for performing the operations described in this application are executed by computing hardware (e.g., by one or more processors or a computer), which is implemented to execute instructions or software as described above to perform the operations performed by the methods described in this application. For example, a single operation, or two or more operations, may be performed by a single processor, or two or more processors, or a processor and a controller. One or more operations may be performed by one or more processors, or a processor and a controller, and one or more other operations may be performed by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, may perform a single operation, or two or more operations.

[0107] Instructions or software for controlling a processor or computer to implement hardware components and perform the methods described above can be written as computer programs, code segments, instructions, or any combination thereof to individually or collectively instruct or configure the processor or computer to operate as a machine or special-purpose computer to perform operations performed by the hardware components and methods described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) that is directly executed by the processor or computer. In another example, the instructions or software include high-level code that is executed by the processor or computer using an interpreter. Those skilled in the art can readily write instructions or software based on the block diagrams and flowcharts shown in the accompanying drawings and the corresponding descriptions in the specification, which disclose algorithms for performing operations performed by the hardware components and methods described above.

[0108] Instructions or software used to control a processor or computer to implement hardware components and perform the methods described above, along with any associated data, data files, and data structures, are recorded, stored, or fixed in, or on, one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage devices, hard disk drive (HDD), solid-state drive (SSD), flash memory, card memory (such as multimedia cards or microcards (e.g., Secure Digital (SD) or Extreme Digital (XD))), magnetic tape, floppy disk, magneto-optical data storage devices, optical data storage devices, hard disks, solid-state drives, and any other device configured to store instructions or software and any associated data, data files, and data structures in a non-transitory manner and to provide instructions or software and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the instructions.

[0109] While this disclosure includes specific examples, it will be apparent to those skilled in the art that various changes in form and detail may be made to these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein should be considered descriptive only and not for limiting purposes. The description of features or aspects in each example should be considered applicable to similar features or aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order, and / or if components in the described system, architecture, apparatus, or circuit are combined in different ways, and / or replaced or supplemented by other components or their equivalents. Therefore, the scope of the disclosure is not limited by the specific embodiments but by the claims and their equivalents, and all variations within the scope of the claims and their equivalents should be construed as included in the disclosure.

Claims

1. A method for image processing, comprising: Calculate the surface normal vectors corresponding to the pixels of the first depth image, and generate a first surface normal image including the surface normal vectors corresponding to the pixels of the first depth image; as well as A first depth image and a first surface normal image are applied to a first neural network, and a second depth image is obtained by modifying the first depth image using the first neural network. The first neural network generates a second depth image with improved quality compared to the first depth image based on an embedding vector that includes features of the first depth image and features of the first surface normal image.

2. The method according to claim 1, wherein, The first neural network removes noise from the first depth image and improves the clarity of the first depth image.

3. The method according to claim 1, wherein, The first depth image includes time-of-flight images.

4. The method according to any one of claims 1 to 3, further comprising: Generate a second surface normal image that includes surface normal vectors corresponding to the pixels of the second depth image; as well as The second depth image and the second surface normal image are applied to the second neural network, and the third depth image is obtained by modifying the second depth image using the second neural network. The second neural network generates a third depth image of degraded quality compared to the second depth image based on an embedding vector that includes features of the second depth image and features of the second surface normal image.

5. The method according to claim 4, further comprising: Training data is generated by pairing second-depth images with third-depth images.

6. The method according to claim 4, wherein, The first and second neural networks were trained using a recurrent generative adversarial network (GAN) training method.

7. A training method, comprising: A second depth image of a second type is obtained by applying a first depth image of a first type and a first surface normal image corresponding to the first depth image to a first neural network, wherein the first surface normal image corresponding to the first depth image includes a calculated surface normal vector corresponding to a pixel of the first depth image. A third depth image of the first type is obtained by applying the acquired second depth image and the second surface normal image corresponding to the second depth image to a second neural network; Obtain the first generative adversarial network (GAN) loss related to whether the second depth image is an image generated by the first neural network; Obtain the first cycle consistency loss associated with the difference between the third depth image and the first depth image; and The first neural network is trained based on the first GAN loss and the first cycle consistency loss.

8. The training method according to claim 7, wherein, The steps for training the first neural network include: A fifth depth image of the first type is obtained by applying a fourth depth image of the second type and a fourth surface normal image corresponding to the fourth depth image to a second neural network; A second type of sixth depth image is obtained by applying the acquired fifth depth image and the fifth surface normal image corresponding to the fifth depth image to a first neural network; Obtain the second GAN loss related to whether the fifth depth image is an image generated by the second neural network; Obtain the second cyclic consistency loss associated with the difference between the sixth depth image and the fourth depth image; and The first and second neural networks are trained based on the first GAN loss, the second GAN loss, the first cycle consistency loss, and the second cycle consistency loss.

9. The training method according to claim 8, further comprising: Obtain a first identity ID loss related to the difference between a first depth image and a depth image obtained by applying the first depth image and a first surface normal image to a second neural network; Obtain a second ID loss related to the difference between the fourth depth image and the depth image obtained by applying the fourth depth image and the fourth surface normal image to the first neural network; and The first neural network and the second neural network are further trained based on the first ID loss and the second ID loss.

10. The training method according to claim 8, further comprising: The first neural network and the second neural network are further trained based on a first range-preserving loss for limiting the difference between the first depth image and the second depth image and a second range-preserving loss for limiting the difference between the fourth depth image and the fifth depth image.

11. The training method according to claim 8, further comprising: The second neural network is trained based on the loss used to distinguish the difference between the fourth depth image and the fifth depth image, as well as the difference between the fourth depth image and the first type of real depth image corresponding to the fourth depth image.

12. The training method according to any one of claims 7 to 11, wherein, The first neural network includes a first pre-trained neural network, which is trained to generate a second type of depth image by modifying a first type of depth image along a first direction. The second neural network includes a second pre-trained neural network, which is trained to generate a first type of depth image by changing a second type of depth image along a second direction opposite to the first direction.

13. The training method according to any one of claims 7 to 11, wherein, The first type of depth image includes depth images with noise. The second type of depth image includes depth images that do not contain noise. The first neural network is configured to generate a noise-removed depth image by removing noise from the input depth image, and The second neural network is configured to generate a noisy depth image by adding noise to another input depth image.

14. A non-transitory computer-readable storage medium storing instructions, which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 13.

15. An image processing apparatus, comprising: At least one processor is configured to: calculate surface normal vectors corresponding to pixels of a first depth image, generate a first surface normal image including the surface normal vectors corresponding to pixels of the first depth image, apply the first depth image and the first surface normal image to a first neural network, and obtain a second depth image by modifying the first depth image using the first neural network. The first neural network is configured to generate a second depth image with improved quality compared to the first depth image, based on an embedding vector that includes features of the first depth image and features of the first surface normal image.

16. The device according to claim 15, wherein, The first neural network is also configured to remove noise included in the first depth image and improve the clarity of the first depth image.

17. The device according to claim 15, wherein, The first depth image includes time-of-flight images.

18. The device according to any one of claims 15 to 17, wherein The processor is also configured to: generate a second surface normal image including surface normal vectors corresponding to pixels of the second depth image; apply the second depth image and the second surface normal image to a second neural network; and obtain a third depth image by modifying the second depth image using the second neural network. in, The second neural network is configured to generate a third depth image of degraded quality compared to the second depth image, based on an embedding vector that includes features of the second depth image and features of the second surface normal image.

19. The device according to claim 18, wherein, The processor is also configured to generate training data by pairing second and third depth images.

20. The device according to claim 18, wherein, The first and second neural networks were trained using a recurrent generative adversarial network (GAN) training method.

21. An image processing apparatus, comprising: At least one processor is configured as follows: A second depth image with weak noise is obtained by applying a first depth image with strong noise and a first surface normal image corresponding to the first depth image to a first neural network, wherein the first surface normal image corresponding to the first depth image includes a calculated surface normal vector corresponding to a pixel of the first depth image. A third depth image with strong noise is obtained by applying the acquired second depth image and the second surface normal image corresponding to the second depth image to a second neural network; Obtain the first generative adversarial network (GAN) loss related to whether the second depth image is an image generated by the first neural network; Obtain the first cycle consistency loss associated with the difference between the third depth image and the first depth image; and The first neural network is trained based on the first GAN loss and the first cycle consistency loss. Wherein, strong noise represents noise greater than the first threshold, weak noise represents noise less than the second threshold, and the first threshold is greater than or equal to the second threshold.

22. The device according to claim 21, wherein, The steps for training the first neural network include: A fifth depth image with strong noise is obtained by applying a fourth depth image with weak noise and a fourth surface normal image corresponding to the fourth depth image to a second neural network. A sixth depth image with weak noise is obtained by applying the acquired fifth depth image and the fifth surface normal image corresponding to the fifth depth image to the first neural network. Obtain the second GAN loss related to whether the fifth depth image is an image generated by the second neural network; Obtain the second cyclic consistency loss associated with the difference between the sixth depth image and the fourth depth image; and The first and second neural networks are trained based on the first GAN loss, the second GAN loss, the first cycle consistency loss, and the second cycle consistency loss.

23. The device according to claim 22, wherein, The at least one processor is further configured to: Obtain a first identity ID loss related to the difference between a first depth image and a depth image obtained by applying the first depth image and a first surface normal image to a second neural network; Obtain a second ID loss related to the difference between the fourth depth image and the depth image obtained by applying the fourth depth image and the fourth surface normal image to the first neural network; and The first neural network and the second neural network are further trained based on the first ID loss and the second ID loss.

24. The device according to claim 22, wherein, The at least one processor is further configured to train a first neural network and a second neural network based on a first range-preserving loss for limiting the difference between a first depth image and a second depth image and a second range-preserving loss for limiting the difference between a fourth depth image and a fifth depth image.

25. The device according to claim 22, wherein, The at least one processor is further configured to train a second neural network based on a loss used to distinguish the difference between the fourth depth image and the fifth depth image and the difference between the fourth depth image and the corresponding real depth image with strong noise.

Images