Image processing method and apparatus for producing a reconstructed image

Abstract

To disclose an image processing method and device for creating a reconstructed image.SOLUTION: An image processing method includes the steps of: determining albedo data in a standard space and depth data in the standard space from input image data including an object by using neural network-based extraction models; applying a target shape deformation value respectively to the albedo data and the depth data to generate deformed albedo data and deformed depth data; performing shading based on the deformed depth data and a target illumination value to generate shading result data; generating intermediate image data based on the shading result data and the deformed albedo data; and generating reconstructed image data from the intermediate image data and the deformed depth data based on a target pose value.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The following embodiments relate to image processing techniques that generate reconstructed images. [Background technology]

[0002] Recently, electronic devices capable of virtual reality (VR), augmented reality (AR), or mixed reality (MR) have been developed, drawing growing interest. Among these, augmented reality is a display technology that combines virtual objects and information with a real-world environment. Augmented reality has the advantage of being applicable to various real-world environments and is attracting attention as a next-generation display technology suitable for ubiquitous environments and Internet of Things (IoT) environments. Augmented reality can provide users with new experiences by combining a panoramic view with virtual images and is a means of conveying information more efficiently and immersively. To realize augmented reality content in a immersive manner, it is necessary to adjust the characteristics of virtual objects according to the user's pose, lighting conditions, etc. Summary of the Invention [Problem to be solved by the invention]

[0003] It is an object of the present invention to provide an image processing technique that produces a reconstructed image. [Means for solving the problem]

[0004] According to one embodiment, an image processing method for generating a reconstructed image includes determining albedo data in a standard space and depth data in the standard space from input image data showing an object using a neural network-based extraction model; generating transformed albedo data and transformed depth data by applying a target shape transformation value to each of the albedo data and the depth data; generating shading result data by performing shading based on the transformed depth data and a target illumination value; generating intermediate image data based on the shading result data and the transformed albedo data; and generating reconstructed image data from the intermediate image data and the transformed depth data based on a target pose value.

[0005] The step of determining the albedo data and the depth data may include a step of determining albedo data in the standard space from the input image data using a neural network-based albedo extraction model, and a step of determining depth data in the standard space from the input image data using a neural network-based depth extraction model.

[0006] The albedo data in the standard space may correspond to the albedo data when the object is transformed into a reference standard form, and the depth data in the standard space may correspond to the depth data when the object is transformed into the standard form.

[0007] The step of generating the transformed albedo data and the transformed depth data may include performing a backward warping operation on the albedo data and the depth data, respectively, based on the target shape transformation value.

[0008] The step of generating the shading result data may include the steps of extracting a surface normal component of the object from the transformed depth data, and performing shading based on the extracted surface normal component and the target illumination value to generate the shading result data.

[0009] The step of generating the reconstructed image data may include a step of transforming the pose of the object shown in each of the intermediate image data and the transformed depth data based on the target pose value, and generating the reconstructed image data by combining the intermediate image data in which the pose of the object has been transformed and the depth data in which the pose of the object has been transformed.

[0010] The target shape deformation value, the target illumination value, and the target pose value may be values ​​extracted from other input image data using a neural network-based second extraction model other than the neural network-based first extraction model.

[0011] According to an embodiment, an image processing device for generating a reconstructed image includes at least one processor, and the at least one processor controls the image processing device to perform the following operations: determining albedo data in a standard space and depth data in the standard space from input image data in which an object is shown, using a neural network-based first extraction model; generating transformed albedo data and transformed depth data by applying a target shape transformation value to each of the albedo data and the depth data; performing shading based on the transformed depth data and a target illumination value to generate shading result data; generating intermediate image data based on the shading result data and the transformed albedo data; and generating reconstructed image data from the intermediate image data and the transformed depth data based on a target pose value.

[0012] According to one embodiment, a training method includes determining albedo data in a standard space and depth data in the standard space from first training image data using a neural network-based first extraction model; extracting a shape deformation value, an illumination value, and a pose value from the second training image data; generating transformed albedo data and transformed depth data by applying the shape deformation value to the albedo data and the depth data, respectively; generating shading result data by performing shading based on the transformed depth data and the illumination value; generating intermediate image data based on the shading result data and the transformed albedo data; generating reconstructed image data from the intermediate image data and the transformed depth data based on the pose value; and updating parameters of the first extraction model based on the reconstructed image data and the second training image data.

[0013] According to an embodiment, an electronic device includes a processor that generates reconstructed image data based on input image data, and a display that visualizes the reconstructed image data, wherein the processor performs the following operations: determining albedo data in a standard space and depth data in the standard space from input image data showing an object using a neural network-based extraction model; generating transformed albedo data and transformed depth data by applying a target shape transformation value to each of the albedo data and the depth data; performing shading based on the transformed depth data and a target illumination value to generate shading result data; and generating intermediate image data based on the shading result data and the transformed albedo data. and generating reconstructed image data from the intermediate image data and the deformed depth data based on the target pose value. [Effects of the Invention]

[0014] According to one embodiment, objects in image data can be efficiently rendered in various viewpoints, poses, or lighting environments.

[0015] According to one embodiment, rendering can be performed efficiently not only for fixed objects but also for various types of moving objects. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 1 is a diagram illustrating an overview of reconstructed image data generation according to an embodiment. [Figure 2] FIG. 1 is a diagram illustrating an image reconstruction framework according to an embodiment. [Figure 3] 1 is a flowchart illustrating an image processing method for generating reconstructed image data according to an embodiment. [Figure 4] 10A and 10B are diagrams illustrating a process of generating reconstructed image data according to an embodiment in more detail. [Figure 5] 10A and 10B are diagrams illustrating a process of generating reconstructed image data according to an embodiment in more detail. [Figure 6] FIG. 10 is a diagram illustrating a learning process for an extraction model according to an embodiment. [Figure 7] 1 is a diagram illustrating a configuration of an image processing device according to an embodiment. [Figure 8] 1 is a block diagram showing a configuration of an electronic device according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0017] The specific structural or functional descriptions disclosed in this specification are merely examples for the purpose of describing the embodiments, and the embodiments may be implemented in various different forms. The present invention is not limited to the embodiments described in this specification, and the scope of the present invention includes modifications, equivalents, or alternatives that fall within the technical idea described in the embodiments.

[0018] Although terms such as "first" or "second" may be used to describe multiple components, such terms should be construed only to distinguish one component from the other components. For example, a first component may be designated as a second component, and similarly, a second component may be designated as a first component.

[0019] When a component is referred to as being "coupled" or "connected" to another component, it should be understood that although it is directly coupled or connected to the other component, there may be other components in between.

[0020] The singular expression includes the plural expression unless the context clearly dictates otherwise. In this specification, the words "comprise" or "have" and the like indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, and should be understood as not precluding the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0021] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art to which the present invention pertains. Commonly used predefined terms should be interpreted as having a meaning consistent with the meaning they have in the context of the relevant art, and should not be interpreted as having an ideal or overly formal meaning unless expressly defined herein.

[0022] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, the same reference numerals will be used to designate the same elements regardless of the reference numerals, and redundant description thereof will be omitted.

[0023] FIG. 1 is a diagram for explaining an overview of generation of reconstructed image data according to one embodiment.

[0024] Referring to FIG. 1, an image processing device 100 (e.g., image processing device 700 shown in FIG. 7) is a device that generates reconstructed image data from one or more input image data, and can be implemented, for example, by a software module, a hardware module, or a combination thereof. The image processing device 100 may include a central processing unit (CPU) and / or a graphics processing unit (GPU), and can generate image data by reconstructing the shape of an object shown in the input image data, the pose of the object, the lighting environment, etc., using the CPU and / or GPU. The object may be, for example, a human face, an object, etc., but the scope of the embodiment is not limited thereto, and the type of object is not limited thereto.

[0025] The generation of reconstructed image data performed by the image processing device 100 is used in applications such as augmented reality, image rendering, video conferencing, face reenactment, landmark detection, image relighting, 3D user interfaces, and image retouching. Augmented reality, for example, to combine real-world images with virtual objects in augmented reality, requires 3D spatial modeling of the real-world image. Based on the 3D spatial modeling, a virtual object with an appropriate lighting environment and pose is generated through image rendering. Image rendering is a process or method of creating an image from a 3D scene containing an object using a computer program. Rendering a 3D scene includes applying visual effects, such as lighting and shading effects, to the 3D scene based on light emitted from one or more virtual light sources, which are virtual light sources in 3D computer graphics. Applying lighting effects determines the color of a 3D model included in the 3D scene, and applying shading effects determines shadows, reflections, transparency effects, and other effects due to occlusion. Such effects are based on the characteristics (e.g., color and direction, etc.) of light emitted from a virtual light source, the characteristics (e.g., color, surface normal, texture, etc.) of the 3D model, the positional relationship between the virtual light source and the 3D model, etc. The image processing device 100 decomposes image components in input image data, transforms the corresponding image components based on desired target shape deformation values, target illumination values, and target pose values, and combines the transformed image components to generate reconstructed image data (e.g., a reconstructed virtual object) having an appropriate lighting environment and pose.

[0026] To generate reconstructed image data, the image processing device 100 may use a neural network-based image reconstruction framework trained using unsupervised learning. The image reconstruction framework may include one or more neural networks, which may be trained using unlabeled data without the need for a ground truth dataset. Thus, the image reconstruction framework may be trained based on a large amount of diverse training data. The image reconstruction framework will be described in detail below with reference to FIGS. 2, 4, and 5, and the training of the image reconstruction framework will be described in detail below with reference to FIG. 6.

[0027] FIG. 2 is a diagram illustrating an image reconstruction framework according to an embodiment.

[0028] 2, there is shown one structure of an image reconstruction framework for generating reconstructed image data 270 from input image data 210. The input image data 210 showing an object is transmitted to an image processing device (e.g., the image processing device 100 shown in FIG. 1 or the image processing device 700 shown in FIG. 7). When the object is a face, the input image data 210 corresponds to a face image including a facial region.

[0029] The image processing device acquires material component data 230 and geometric component data 240 for an object from input image data 210 using a neural network-based first extraction model 220. The first extraction model 220 estimates the material component data 230 and geometric component data 240 based on the input image data 210. The material component data 230 includes albedo data for the object, and the geometric component data 240 includes depth data for the object. The albedo data and depth data acquired from the first extraction model 220 correspond to albedo data in a canonical space warped to canonical space and depth data in the canonical space, respectively. The canonical space corresponds to a normalized pose space. Although the object pose of the input image data 210 may vary, the image processing device can acquire albedo data and depth data in the canonical space corresponding to a reference pose through the first extraction model 220. In this specification, acquiring albedo data and depth data may be replaced with determining albedo data and depth data.

[0030] The first extraction model 220 includes one or more neural networks. For example, the first extraction model 220 includes an albedo extraction model 222, which is a neural network for extracting material component data 230 of an object from the input image data 210, and a depth extraction model 224, which is a neural network for extracting geometric component data 240 of an object. As another example, the first extraction model 220 may be realized as a model in which both the material component data 230 and the geometric component data 240 are output from a single neural network.

[0031] A neural network outputs a value calculated by its internal parameters in response to input data. At least a portion of a neural network may be implemented in software, hardware including a neural processor, or a combination of software and hardware. A neural network is a deep neural network (DNN), which includes a fully connected network, a deep convolutional network, and a recurrent neural network. A DNN includes multiple layers, each of which includes an input layer, one or more hidden layers, and an output layer. A neural network may be trained to perform a given operation by mapping input data and output data that have a nonlinear relationship to each other using deep learning. Deep learning is a machine learning method for solving problems given by big data sets. Deep learning is a neural network optimization process that trains a neural network using prepared training data and aims to minimize energy.

[0032] The image processing device generates reconstructed image data 270 from material component data 230, geometric component data 240, and target construction data 260 via an image reconstruction processor 250. The target construction data 260 includes target shape deformation values, target illumination values, and target pose values ​​as construction data including one or more setting values ​​for reconstructing the input image data 210. The target construction data 260 may be extracted from other input image data depicting the same object as the object in the input image data 210, or from other input image data depicting a different object from the object in the input image data 210. The target shape deformation values, target illumination values, and target pose values ​​of the target construction data 260 may be extracted from the other input image data using a second extraction model based on a neural network.

[0033] The image reconstruction processor 250 can generate shading result data by transforming the shapes represented by the material component data 230 and the geometric component data 240 based on the target shape transformation values ​​and performing shading based on the target lighting values. The image reconstruction processor 250 performs rendering based on the shading result data and the target pose values ​​and generates reconstructed image data 270. The reconstructed image data 270 corresponds to image data in which the input image data 210 has been transformed in terms of the object's shape, pose, and lighting using the target construction data 260. If the target construction data 260 is extracted from the input image data 210 and other input image data, the reconstructed image data 270 represents a combined form of the input image data 210 and other input image data.

[0034] Through the above process, the image processing device may render the object of the input image data 210 at any viewpoint or pose. The image processing in the image reconstruction framework described above is effective when it is necessary to render an object in an augmented reality environment with various viewpoints, poses, and lighting environments. Furthermore, the image reconstruction framework does not require the object in the image to be fixed, and is applicable to various types of moving objects.

[0035] 3 is a flow chart illustrating an image processing method for generating reconstructed image data according to one embodiment, the image processing method being performed by an image processing device described herein.

[0036] 3, in step S310, the image processing device acquires albedo data and depth data from input image data in which an object is shown. The image processing device acquires albedo data in a standard space and depth data in a standard space using a trained neural network-based extraction model. The albedo data in the standard space indicates the albedo data when the object is transformed into a standard form as a reference, and the depth data in the standard space indicates the depth data when the object is transformed into the standard form.

[0037] In one embodiment, the image processing device can obtain albedo data in standard space from input image data using a neural network-based albedo extraction model, and obtain depth data in standard space from input image data using a neural network-based depth extraction model.

[0038] In step S320, the image processing device generates deformed albedo data and deformed depth data based on the target shape transformation value. The image processing device applies the target shape transformation value to each of the albedo data and depth data acquired in step S310 to generate albedo data and deformed depth data in which the shape of the object has been deformed. The process of generating the deformed albedo data and deformed depth data includes a process of performing a backward warping operation on each of the albedo data and depth data based on the transformation value of the target shape.

[0039] In step S330, the image processing device performs shading based on the transformed depth data and the target illumination value to generate shading result data. The image processing device can extract a surface normal component of the object from the transformed depth data and perform shading based on the extracted surface normal component and the target illumination value to generate shading result data. The surface normal component of the object indicates a normal component of the surface of the object. The image processing device may perform shading using second-order spherical harmonics. The shading includes determining a shading value determined by a vector dot product between the illumination direction of the light source indicated by the target illumination value and the surface normal component of the object.

[0040] In step S340, the image processing device generates intermediate image data based on the shading result data and the transformed albedo data. The intermediate image data corresponds to color image data in which the target shape transformation value and the target illumination value are applied to the input image data. The color seen at a certain point on an object is affected by the object's shape and material, the light source, and the viewpoint. The color of an object is classified into an albedo component (or reflection component) and a shading component. The albedo component is an object-specific color component or material characteristic determined by the object's shape and material and is unrelated to the light source and viewpoint. The shading component indicates the illumination characteristic value that is displayed when light emitted from a light source interacts with the surface normal of the object. The albedo component and shading component of the input image data are transformed to suit the desired shape transformation and illumination transformation, and then combined to generate intermediate image data.

[0041] In step S350, the image processing device generates reconstructed image data from the intermediate image data and the deformed depth data based on the target pose value. The image processing device may change the pose of the object shown in each of the intermediate image data and the deformed depth data based on the target pose value. The image processing device may deform the pose of the object shown in each of the intermediate image data and the deformed depth data to the pose indicated by the target pose value via a projection matrix determined by the target pose value. The image processing device may generate reconstructed image data by combining the intermediate image data in which the pose of the object has been deformed and the depth data in which the pose of the object has been deformed.

[0042] The target shape deformation value, target illumination value, and target pose value may be values ​​extracted from input image data different from the input image data. Here, the object shown in the other input image data may be the same as or different from the object shown in the input image data. In one embodiment, the target shape deformation value, target illumination value, and target pose value may be values ​​extracted from the other input image data using a second extraction model based on a trained neural network. In this case, the other input image data is input to the second extraction model, and the shape deformation value, illumination value, and pose value indicating the object characteristics of the other input image data may be output from the second extraction model as the target shape deformation value, target illumination value, and target pose value, respectively. When reconstructed image data is generated based on input image data different from the input image data, the material and overall shape characteristics of the object in the reconstructed image data indicate the characteristics of the object shown in the input image data, and local shape deformation, illumination, and pose characteristics such as facial expression indicate the characteristics of the object shown in the other input image data.

[0043] 4 and 5 are diagrams for more specifically explaining a process of generating reconstructed image data according to an embodiment. The process of generating reconstructed image data described below may be performed by the image processing device described in this specification.

[0044] Referring to FIG. 4, a trained neural network-based extraction model 415 (e.g., the first extraction model 220 shown in FIG. 2) extracts physical elements of an object from input image data 410. Standard space albedo data 420 and standard space depth data 425 are extracted through the extraction model 415. The albedo data 420 represents the inherent color of a material, independent of viewpoint or lighting, as a material component of the object. The depth data 425 represents the shape of the object. The depth data 425 is expressed as depth values ​​in x and y coordinates. The albedo data 420 and the depth data 425 are expressed as image data having the same resolution as the input image data 410. The albedo data 420 and the depth data 425 may be attributes that define a dense point cloud of the object displayed on a two-dimensional regular grid.

[0045] The albedo data in standard space 420 and the depth data in standard space 425 represent the albedo data and the depth data, respectively, aligned with the reference pose indicated by the standard space. The standard space is a space for aligning a deformable object within a single pose space, and corresponds to a deformation-free normalized pose space. Variables in the standard space are considered to remain unchanged under all environmental conditions and / or deformations. The albedo data in standard space 420 and the depth data in standard space 425 are each independent of deformation. If the object is a face, the deformation corresponds to facial expression.

[0046] The target shape transformation value 430 is applied to the standard space albedo data 420 and the standard space depth data 425 to obtain transformed albedo data 460 and transformed depth data 465. The function f w 455 corresponds to a backward warping operation that corresponds to moving the point cloud in 3D space. w455 defines the degree of shape deformation indicated by the target shape deformation value 430 as a 3D offset amount in each of the x, y and z axes for moving the point cloud, and by applying the corresponding 3D offset amount to the albedo data in standard space 420 and the depth data in standard space 425, respectively, it is possible to generate deformed albedo data 460 and deformed depth data 465.

[0047] The target shape deformation value 430 is a value that reflects the facial expression, etc., when the object is a face, and reflects the local geometric change of the object. The target shape deformation value 430 corresponds to a local difference value for the shape at the pose in the standard space. For example, if the pose in the standard space is a neutral expression and the target expression is a smiling expression, the target shape deformation value 430 corresponds to the shape difference between the neutral expression and the smiling expression.

[0048] Surface normal data 470 may be generated from the transformed depth data 465. The surface normal data 470 indicates the geometric shape components of the object. The surface normal value of the surface normal data 470 is calculated from the transformed depth data 465 through per-pixel regression for local neighboring pixels. Once a plane that minimizes the sum of squared distances based on the depth value of the current pixel and each depth value of the surrounding neighboring pixels is determined, the normal vector of the determined plane can be determined as the surface normal value of the corresponding current pixel.

[0049] Then, shading 475 is performed based on the surface normal data 470 and the target lighting value 440. The target lighting value 440 represents a desired lighting effect and may be expressed as a multidimensional vector. In the shading 475, lighting components are reflected in the surface normal data 470 by the target lighting value 440. In the Lambertian model, the shading 475 may be expressed as being determined based on the surface normal data 470 and the target lighting value 440. For the shading 475, a spherical harmonic function may be used to represent the shading value for each pixel as a quadratic combination of the surface normal value and the target lighting value 440, which is expressed as a multidimensional vector value. By performing this shading 475 process for each pixel, shading result data can be generated.

[0050] After the shading result data is generated, intermediate image data 485 is generated via a vector dot product operation 480 between the shading result data and the transformed albedo data 460. The intermediate image data 485 may include diffuse color information.

[0051] Reconstructed image data 495 is generated based on the intermediate image data 485, the deformed depth data 465, and the target pose values ​​450. The target pose values ​​450 may be expressed as a multi-dimensional vector that defines a 3D transformation matrix. The 3D transformation matrix includes elements for translation, rotation, etc. The target pose values ​​450 define a mapping from standard space onto a point cloud in input space. Given the target pose values ​​450, a function f r490 sends the transformed depth data 465 and the dense point cloud of intermediate image data 485 to the input space, allowing all points or meshes defining the shape to be projected onto the image plane of the corresponding camera. This process may be performed by a rendering process. The function f r As a result of the processing at 490, reconstructed image data 495 may be generated.

[0052] Through the image reconstruction framework described above, each physical element (such as viewpoint, lighting, and pose) shown in the input image data 410 can be freely manipulated. Meanwhile, the target shape deformation value 430, the target lighting value 440, and the target pose value 450 may be given values, or may be values ​​extracted from the image data as in the embodiment shown in FIG.

[0053] 5, in one embodiment, first and second image data 510, 530 of the same or different objects may be input to an image reconstruction framework, where the first and second image data 510, 530 may comprise image data that constitute a multi-frame image.

[0054] Through a trained neural network-based first extraction model 515 (e.g., extraction model 415 shown in FIG. 4), albedo data 520 in standard space and depth data 525 in standard space are acquired from the first image data 510. Through alignment in standard space, comparison and synthesis between the first and second image data 510 and 530 is possible even when the poses or shapes of the same object shown in the first and second image data 510 and 530 are different from each other, or when the objects shown in the first and second image data 510 and 530 are different from each other.

[0055] Physical elements such as shape deformation values ​​(e.g., expression values) 540, lighting values ​​545, and pose values ​​550 can be extracted from the second image data 530 through the trained neural network-based second extraction model 535. The extracted shape deformation values ​​540, lighting values ​​545, and pose values ​​550 correspond to the target shape deformation values ​​430, target lighting values ​​440, and target pose values ​​450 shown in FIG. 4, respectively. The first extraction model 515 and the second extraction model 535 may be implemented as separate neural networks or a single neural network. An encoder-decoder network may be used as the first extraction model 515, and an encoder network may be used as the second extraction model 535.

[0056] A function f that applies a shape transformation value 540 to the standard space albedo data 520 and the standard space depth data 525, respectively. w 555 obtains transformed albedo data 560 and transformed depth data 565. Surface normal data 570 is generated from the transformed depth data 565, and shading 575 is performed based on the surface normal data 570 and the illumination value 545. Shading result data is generated as a result of the shading 575, and intermediate image data 585 can be generated through a vector dot product operation 580 between the shading result data and the transformed albedo data 560. A function f r Reconstructed image data 595 can be generated by 590 .

[0057] In the image reconstruction framework described above, specific physical elements may be extracted from the second image data 530, and reconstructed image data 595 may be generated by replacing the physical elements of the first image data 510 with the corresponding physical elements of the second image data 530. Meanwhile, in the embodiment shown in FIG. 5, detailed description of parts that overlap with the embodiment shown in FIG. 4 will be omitted, and the description of the embodiment shown in FIG. 4 may also be applied to the embodiment shown in FIG.

[0058] FIG. 6 is a diagram illustrating a learning process for an extraction model according to an embodiment.

[0059] FIG. 6 illustrates a training process for training the image reconstruction framework shown in FIG. 5. The training process involves learning to segment latent physical elements of a deformable object. The training process may be performed by a training device including a processor and a memory. The training device may perform the training process using multiple training image data that differ in one or more of shape deformation, lighting environment, and pose. The first training image data 610 and the second training image data 630 may include the same type of object as images selected from an image sequence constituting a multi-frame image, for example.

[0060] The learning device can use a first extraction model 615 based on a neural network to obtain albedo data 620 in a standard space and depth data 625 in the standard space from first learning image data 610 in which an object is shown.

[0061] The training device can extract shape deformation values ​​640, lighting values ​​645, and pose values ​​650 from the second training image data 630 using a second neural network-based extraction model 635.

[0062] The learning device calculates a function f that applies a shape transformation value 640 to the albedo data 620 in the standard space and the depth data 625 in the standard space. w655 can generate transformed albedo data 660 and transformed depth data 665. The trainer may generate surface normal data 670 from the transformed depth data 665 and perform shading 675 based on the surface normal data 670 and lighting values ​​645.

[0063] As a result of executing shading 675, shading result data is generated, and the learning device generates intermediate image data 685 based on the shading result data and modified albedo data 660. The learning device generates intermediate image data 685 through a vector dot product operation 680 between the shading result data and the modified albedo data 660. The learning device generates image data 695 reconstructed from the intermediate image data 685 and modified depth data 665 based on the pose value 650. The learning device generates a function f r 690 can be used to generate reconstructed image data 695 .

[0064] The learner 600 compares the reconstructed image data 695 with the second training image data 630 and defines a loss function that defines a loss based on the difference between the two. For example, a photometric reconstruction loss may be used to define the loss function. The photometric reconstruction loss aims to minimize the error between the reconstructed image data 695 and the second training image data 630.

[0065] The learning device 600 can train the first extraction model 615 and the second extraction model 635 so that the loss is reduced. The training process includes a process of updating parameters of the first extraction model 615 and the second extraction model 635. The learning device 600 also includes a process of iteratively correcting parameters of the first extraction model 615 and the second extraction model 635 so that the difference between the reconstructed image data 695 and the second training image data 630 is reduced. The training process of the first extraction model 615 and the second extraction model 635 is performed automatically so that the reconstructed image data 695 is fitted to the second training image data 630. The first extraction model 615 after training may be used as the extraction model 415 shown in FIG. 4 and the first extraction model 515 shown in FIG. 5, and the second extraction model 635 after training may be used as the second extraction model 535 shown in FIG. 5. In other embodiments, the target shape deformation values, target illumination values, and target pose values ​​may be extracted from other input image data, and the learner 600 may train the extraction models 615, 635 by updating the parameters of the extraction models 615, 635 based on input image data different from the reconstructed image data 695.

[0066] The above learning process has the advantage that it can be applied to moving objects without requiring a fixed object, and can use a variety of training image data since it does not require a complex writing stage, etc. Also, this learning process has the advantage that it can perform self-supervised learning using unlabeled training image data obtained under different poses and lighting conditions.

[0067] FIG. 7 is a diagram showing the configuration of an image processing device according to an embodiment.

[0068] 7, an image processing device 700 (e.g., image processing device 100 of FIG. 1) is a device that processes image data to generate reconstructed image data. The image processing device 700 includes one or more processors 710 and a memory 720.

[0069] Memory 720 stores computer-readable instructions. When the instructions stored in memory 720 are executed by processor 710, processor 710 performs the operations defined by the instructions. Memory 720 may include, for example, random access memories (RAM), dynamic random access memories (DRAM), static random access memories (SRAM), or other forms of non-volatile memory known in the art.

[0070] The processor 710 controls the overall operation of the image processing device 700. The processor 710 may be a device implemented as hardware with circuits having a physical structure for performing desired operations. The desired operations may include code or instructions contained in a program. The device implemented as hardware may include a microprocessor, a central processing unit (CPU), a graphic processing unit (GPU), a processor core, a multi-core processor, a multiprocessor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a neural processing unit (NPU), etc.

[0071] The processor 710 may control the image processing device 700 by executing functions and commands for the image processing device 700. The processor 710 may control the image processing device 700 to perform at least one of the operations and / or functions described above with reference to FIGS.

[0072] Under the control of the processor 710, the image processing device 700 can obtain albedo data in standard space and depth data in standard space from input image data showing an object, using a first extraction model based on a neural network.

[0073] In one embodiment, the image processing device 700 may acquire albedo data in a standard space from input image data using a neural network-based albedo extraction model, and may acquire depth data in a standard space from the input image data using a neural network-based depth extraction model. The image processing device 700 generates transformed albedo data and transformed depth data by applying a target shape transformation value to each of the acquired albedo data and depth data. The image processing device 700 generates shading result data by performing shading based on the transformed depth data and a target illumination value. The image processing device 700 extracts a surface normal component of an object from the transformed depth data, and performs shading based on the extracted surface normal component and the target illumination value to generate shading result data. The image processing device 700 generates intermediate image data based on the shading result data and the transformed albedo data, and generates reconstructed image data from the intermediate image data and the transformed depth data based on a target pose value. The target shape transformation value, the target illumination value, and the target pose value may be values ​​extracted from other input image data using a second neural network-based extraction model. The image processing device 700 can generate reconstructed image data by transforming the pose of the object shown in each of the intermediate image data and the transformed depth data based on the target pose value, and combining the intermediate image data in which the pose of the object has been transformed and the depth data in which the pose of the object has been transformed.

[0074] FIG. 8 is a block diagram showing the configuration of an electronic device according to an embodiment.

[0075] 8, an electronic device 800 can generate reconstructed image data and provide the generated reconstructed image data to a user. The electronic device 800 includes the image processing device described above (image processing device 100 shown in FIG. 1 or image processing device 700 shown in FIG. 7).

[0076] The electronic device 800 may be a computing device, an image acquisition device, or a display device. For example, the electronic device 800 may be a device in various fields such as a personal computer, an advanced driver assistance system (ADAS), a head-up display (HUD), a camera, a 3D digital information display (DID), a navigation device, a neuromorphic device, a 3D mobile device, a smartphone, a smart TV, a smart vehicle, an Internet of Things (IoT) device, a medical device, etc. Here, the 3D mobile device may include a display device for displaying augmented reality, virtual reality, and / or mixed reality, a head-mounted display (HMD), a face-mounted display (FMD), augmented reality glasses (AR glasses), etc.

[0077] The electronic device 800 includes a processor 810, a storage device 820, a sensor 830, an input device 840, an output device 850, and a communication device 860. Each component of the electronic device 800 can communicate via a communication bus 870.

[0078] The processor 810 controls the overall operation of the electronic device 800 and executes functions and commands to be executed within the electronic device 800. The processor 810 may perform one or more of the operations described above with reference to FIGS. 1 to 7. The processor 810 generates reconstructed image data based on input image data. The processor 810 may acquire albedo data in a standard space and depth data in a standard space from input image data showing an object using a neural network-based extraction model, and apply a target shape transformation value to each of the albedo data and the depth data to generate transformed albedo data and transformed depth data. The processor 810 performs shading based on the transformed depth data and a target illumination value to generate shading result data, and generates intermediate image data based on the albedo data transformed on the shading result data. The processor 810 may generate reconstructed image data from the intermediate image data and the transformed depth data based on the target pose value.

[0079] The storage device 820 stores information necessary for the processor 810 to perform operations. For example, the storage device 820 stores instructions to be executed by the processor 810 and stores related information while software or programs are being executed by the electronic device 800. The storage device 820 may include RAM, DRAM, SRAM, or other forms of non-volatile memory known in the art as memory. The storage device 820 may also include a computer-readable storage medium or computer-readable storage device, such as a storage, a magnetic hard disk, an optical disk, a flash memory device, etc.

[0080] The sensor 830 includes sensors such as an image capture device that captures image data, a motion sensor, etc. The image capture device can capture input image data and communicate the captured input image data to the processor 810.

[0081] Input device 840 receives user input from a user. For example, input device 840 may include a keyboard, a mouse, a touchscreen, a microphone, or any other device capable of detecting user input from a user and communicating the detected user input to electronic device 800.

[0082] The output device 850 provides output of the electronic device 800 to a user through visual, auditory, or tactile channels. The output device 850 may include a display, a touch screen, a speaker, a vibration generator, or any other device capable of providing output to a user. The display may present a visualization of the reconstructed image data generated by the processor 810.

[0083] The communication device 860 can communicate with external devices via a wired network and / or a wireless network.

[0084] The above-described embodiments may be implemented using hardware components, software components, or a combination of hardware and software components. For example, the devices and components described herein may be implemented using one or more general-purpose or special-purpose computers, such as a processor, controller, arithmetic logic unit (ALU), digital signal processor, microcomputer, field programmable array (FPA), programmable logic unit (PLU), microprocessor, or other device that executes and responds to instructions. The processing device may execute an operating system (OS) and one or more software applications that run on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of software. For convenience of understanding, a processing device may be described as being a single device, but those skilled in the art will recognize that a processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing device may include multiple processors or one processor and one controller. Other processing configurations, such as parallel processors, are also possible.

[0085] Software includes computer programs, codes, instructions, or a combination of one or more thereof, which can configure a processing device to operate as desired or can independently or in combination instruct the processing device. The software and / or data can be permanently or temporarily embodied in any type of machine, component, physical device, virtual device, computer storage medium or device, or transmitted signal wave to be interpreted by the processing device or to provide instructions or data to the processing device. The software can be distributed across computer systems coupled to a network and stored and executed in a distributed manner. The software and data can be stored on one or more computer-readable recording media.

[0086] The methods of the present invention may be embodied in the form of program instructions that can be executed by various computer means and stored on a computer-readable storage medium. The storage medium may include program instructions, data files, data structures, and the like, alone or in combination. The storage medium and program instructions may be specially designed and constructed for the purposes of the present invention, or they may be well-known and available to those skilled in the art of computer software. Examples of computer-readable storage media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include not only machine code, such as produced by a compiler, but also high-level language code executed by a computer using an interpreter, for example. The hardware devices described above may be configured to operate as one or more software modules to perform the operations described in the present invention, or vice versa.

[0087] Although the embodiments have been described above with reference to limited drawings, those skilled in the art may apply various technical modifications and variations based on the above description. For example, the described techniques may be performed in a different order than described, and / or the components of the described systems, structures, devices, circuits, etc. may be combined or combined in a different manner than described, or may be replaced or substituted with other components or equivalents, and still achieve suitable results.

[0088] Therefore, the scope of the present invention should not be limited to the disclosed embodiments, but should be determined by the appended claims and their equivalents. [Explanation of symbols]

[0089] 100, 700: Image processing device 222: Albedo extraction model 224: Depth extraction model 250: Image reconstruction processor 415: Extraction Model 220, 515, 615: First extracted model 535, 635: Second extraction model 600: Learning machine 710, 810: Processor 720:Memory 800:Electronic equipment 820: Storage device 830: Sensor 840: Input device 850: Output device 860:Communication equipment 870:Communication bus

Claims

1. 1. A method of image processing to produce a reconstructed image, comprising: determining albedo data in a standard space and depth data in the standard space from input image data showing an object using a neural network-based extraction model; applying a target shape transformation value to the albedo data and the depth data, respectively, to generate transformed albedo data and transformed depth data; performing shading based on the transformed depth data and a target illumination value to generate shading result data; generating intermediate image data based on the shading result data and the transformed albedo data; generating reconstructed image data from the intermediate image data and the deformed depth data based on target pose values; An image processing method comprising:

2. The step of determining the albedo data and the depth data comprises: determining albedo data in the standard space from the input image data using a neural network-based albedo extraction model; determining depth data in the standard space from the input image data using a neural network-based depth extraction model; 2. The image processing method of claim 1, comprising:

3. the albedo data in the standard space corresponds to the albedo data when the object is transformed into a reference standard form; 3. The image processing method according to claim 1, wherein the depth data in the standard space corresponds to the depth data when the object is transformed into the standard form.

4. 4. The image processing method of claim 1, wherein generating the transformed albedo data and the transformed depth data comprises performing a backward warping operation on the albedo data and the depth data, respectively, based on the target shape transformation value.

5. The step of generating shading result data includes: extracting a surface normal component of the object from the deformed depth data; performing shading based on the extracted surface normal component and the target illumination value to generate the shading result data; 5. An image processing method according to claim 1, comprising:

6. 5. The image processing method according to claim 1, wherein the step of generating the shading result data includes a step of performing the shading using a second-order spherical harmonic function.

7. 7. The image processing method of claim 1, wherein the step of generating the reconstructed image data includes a step of transforming the pose of the object shown in each of the intermediate image data and the transformed depth data based on the target pose value, and generating the reconstructed image data by combining the intermediate image data in which the pose of the object has been transformed and the depth data in which the pose of the object has been transformed.

8. The target shape deformation values, the target lighting values, and the target pose values ​​are a value extracted from input image data different from the input image data, 8. A method according to any preceding claim, wherein the object shown in the different input image data is the same as the object shown in the input image data.

9. 8. The image processing method according to claim 1, wherein the target shape deformation value, the target illumination value, and the target pose value are values ​​extracted from image data in which an object different from the object is depicted.

10. 8. The image processing method of claim 1, wherein the target shape deformation value, the target illumination value, and the target pose value are values ​​extracted from other input image data using a second neural network-based extraction model other than the neural network-based extraction model.

11. the target shape deformation values, the target illumination values, and the target pose values ​​are extracted from other input image data; 8. The image processing method of claim 1, wherein the extraction model is trained by updating parameters of the extraction model based on the reconstructed image data and the other input image data.

12. 12. A computer-readable recording medium storing one or more computer programs including instructions for performing the image processing method of any one of claims 1 to 11.

13. 1. An image processing device for generating a reconstructed image, comprising: at least one processor; The at least one processor causes the image processing device to perform the image processing method of any one of claims 1 to 11. Image processing device.

14. A learning method comprising: determining albedo data in a standard space and depth data in the standard space from the first training image data using a first extraction model based on a neural network; extracting shape deformation values, illumination values, and pose values ​​from the second training image data; applying the shape transformation value to the albedo data and the depth data, respectively, to generate transformed albedo data and transformed depth data; performing shading based on the transformed depth data and the illumination value to generate shading result data; generating intermediate image data based on the shading result data and the transformed albedo data; generating reconstructed image data from the intermediate image data and the deformed depth data based on the pose values; updating parameters of the first extraction model based on the reconstructed image data and the second training image data; Learning methods including.

15. 15. The training method of claim 14, wherein updating the parameters of the first extraction model comprises iteratively correcting the parameters of the first extraction model so that differences between the reconstructed image data and the second training image data decrease.

16. the extracting the shape deformation values, the illumination values, and the pose values ​​includes extracting the shape deformation values, the illumination values, and the pose values ​​from the second training image data using a second extraction model based on a neural network; 16. The learning method of claim 14 or 15, wherein the step of updating the parameters of the first extraction model includes the step of iteratively correcting the parameters of each of the first extraction model and the second extraction model so that the difference between the reconstructed image data and the second training image data is reduced.

17. 1. An electronic device, comprising: a processor for generating reconstructed image data based on the input image data; a display for visualizing the reconstructed image data; The processor may further comprise a processor configured to: determining albedo data in a standard space and depth data in the standard space from input image data showing an object using a neural network-based extraction model; applying a target shape transformation value to each of the albedo data and the depth data to generate transformed albedo data and transformed depth data; performing shading based on the transformed depth data and a target illumination value to generate shading result data; generating intermediate image data based on the shading result data and the transformed albedo data; generating reconstructed image data from the intermediate image data and the deformed depth data based on target pose values; An electronic device that controls the following:

Images