Method, computer device and computer-readable medium for obtaining a face image with added lighting

Abstract

An electronic apparatus may obtain an input image including an user, identify a face area indicating a face of the user in the input image, obtain a face depth map based on the face area, obtain a 3D mesh representing the face based on the face depth map, obtain a virtual image corresponding to the face based on the 3D mesh, and obtain a result image based on the input image and the virtual image.

Description

DescriptionTitle of Invention :METHOD, COMPUTER DEVICE AND COMPUTER-READABLE MEDIUM FOR OBTAINING A FACE IMAGE WITH ADDED LIGHTINGTechnical Field

[0001] The present invention relates to a technique for enhancing the quality and realism of a face image by adding lighting to a single image without requiring complex processing or specialized computing components.Background Art

[0002] When photographing a face without the use of special lighting equipment (e.g., studio lighting equipment), it is often difficult to obtain a high-quality face image due to the possible presence of unwanted shadows and other effects from natural or artificial light sources, the position of which relative to the face is often impossible to change in a desired manner. At the same time, in most everyday applications (photography, self-photography (selfie), video conferencing), the use of special lighting equipment is difficult or impossible. In addition, even with special lighting equipment, obtaining the desired lighting effect can be difficult due to the need to move light sources in space to change the lighting effect. Therefore, there is a need for the ability to obtain a face image with added lighting that can eliminate unwanted shadows and other undesirable lighting effects of the original face image and thereby improve the quality of the face image.

[0003] The visualizing a "relighted" 3D portrait of a person involves obtaining a point cloud based on a sequence of images formed by a moving camera using a flash, as well as neural networks when forming a portrait image. This visualizing method is complex to implement and requires special actions to prepare the initial data (a sequence of images), and therefore it is not applicable for everyday use (regular photography, selfies, video conferencing) and cannot be used to process existing single photographs of a face. Since the method involves the use of a neural network, additional specialized computing components, such as a Neural Processing Unit (NPU), may be required.

[0004] The relighting photographs using deep neural networks and credible learning method involves the use of a pre-trained convolutional neural network to obtain a photograph with "relighting" based on the original photograph and a lighting model. The result of applying the relighting photographs depends on the initial data used to train the neural network. The physical plausibility of the "relight" images thus obtained is not guaranteed. Since the method involves the use of a neural network, additional specialized computing components, such as a neural processor, may be required.

[0005] The problem to be solved by the present invention is to provide a method for obtaining a face image with added lighting, applicable to individual images, as simple as possible, not requiring specialized computing components, and providing high quality and believability (in other words, realism) of the resulting face image with added lighting. Solution to Problem

[0006] In an embodiment, a method for controlling an electronic apparatus, the method may include obtaining an input image including an user, identifying a face area indicating a face of the user in the input image, obtaining a face depth map based on the face area, obtaining a 3D mesh representing the face based on the face depth map, obtaining a virtual image corresponding to the face based on the 3D mesh, and obtaining a result image based on the input image and the virtual image.

[0007] The obtaining the result image may include obtaining a lighting map by adding at least one virtual light source to the virtual image, and obtaining the result image based on the input image and the lighting map.

[0008] The obtaining the lighting map may include based on user input being received, obtaining the lighting map by adding at least one virtual light source to the virtual image.

[0009] The generating the result image may include obtaining the result image based on a plurality of pixels in the input image and a plurality of pixels of the lighting map.

[0010] The obtaining the lighting map may include changing the virtual image by adding at least one virtual light source to the virtual image, and obtaining the lighting map by rendering the changed virtual image.

[0011] The rendering may include Physical Based Rendering.

[0012] The obtaining the face depth map may include detecting the face area indicating the face in a whole area of the input image, segmenting the face area from the input image, storing the segmented face area in a memory of the electronic apparatus, and obtaining the face depth map based on the segmented face area.

[0013] The electronic apparatus may be one of: a smartphone, a computer device, a video camera, an application specific integrated circuit or a system-on-a-chip.

[0014] The obtaining the input image may include obtaining the input image by means of a camera.

[0015] The obtaining the face depth map may include obtaining the face depth map by depth estimation based on the input image.

[0016] In an embodiment, an electronic apparatus may include at least one processor including processing circuitry, memory storing instructions, wherein the instructions, when executed by the at least one processor individually or collectively, cause the electronic device to: obtain an input image including an user, identify a face area indicating a face of the user in the input image, obtain a face depth map based on the face area, obtain a 3D mesh representing the face based on the face depth map, obtain a virtual image corresponding to the face based on the 3D mesh, and obtain a result image based on the input image and the virtual image.

[0017] The instructions, when executed by the at least one processor individually or collectively, may cause the electronic device to: obtain a lighting map by adding at least one virtual light source to the virtual image, and obtain the result image based on the input image and the lighting map.

[0018] The instructions, when executed by the at least one processor individually or collectively, may cause the electronic device to: based on user input being received, obtain the lighting map by adding at least one virtual light source to the virtual image.

[0019] The instructions, when executed by the at least one processor individually or collectively, may cause the electronic device to: obtain the result image based on a plurality of pixels in the input image and a plurality of pixels of the lighting map.

[0020] The instructions, when executed by the at least one processor individually or collectively, may cause the electronic device to: change the virtual image by adding at least one virtual light source to the virtual image, and obtain the lighting map by rendering the changed virtual image.Brief Description of Drawings

[0021] FIG. 1 shows an exemplary schematic diagram of a structure of a proposed computer device for obtaining a face image with added lighting.

[0022] FIG. 2 shows an exemplary flowchart of a proposed method for obtaining a face image with added lighting.

[0023] FIG. 3 schematically shows the data generated during the implementation of the method for obtaining a face image with added lighting.

[0024] FIG. 4 shows an exemplary diagram of a structure of a virtual scene.

[0025] FIG. 5 shows an exemplary flowchart for controlling method of the electronic apparatus.Description of Embodiments

[0026] The proposed invention is hereinafter described in detail with reference to the accompanying drawings.

[0027] According to the first aspect of the invention, there is provided a method for obtaining a face image with added lighting.

[0028] The proposed method is performed by a computer device 101, the structure of which is schematically shown in FIG. 1. As a non-limiting example, the computer device may be a smartphone, a digital camera (e.g., a digital photo camera or a digital video camera), a personal computer, a server, an application specific integrated circuit, a system-on-a-chip, and the like, or a combination of the above.

[0029] The computer device is assumed to comprise at least one processor 102 and memory 103. In addition, the computer device, depending on the implementation, may further comprise one or more of the following: a camera, an input / output device, a communication device, a data bus, a graphics processing unit (not shown in FIG. 1), etc. The processor 102 may be integrated with the memory (for example, when implemented as a system-on-a-chip), or the processor 102 and the memory 103 may be separate components. Further, in some implementations, a part of the memory 103 may be embedded in the processor 102 (e.g., a central processing unit cache), and another part of the memory 103 may be in the form of one or more separate components (e.g., personal computer random access memory modules). The memory 103 may comprise one or more of the following: Random Access Memory (RAM), flash memory, Erasable Programmable ROM (EPROM), Electrically EPROM (EEPROM), register, hard disk, removable hard disk, compact disk read-only memory (CD-ROM), etc. The memory 103 stores program instructions in the form of, for example, software or firmware, instructing the processor 102 to perform the proposed method for obtaining a face image with added lighting when executed. The input / output device may comprise one or more of the following: a display, a touch screen, a keyboard, a mouse, etc. The communication device may comprise a wired communication device (e.g., an Ethernet adapter) and a wireless communication device (e.g., a Wi-Fi module, a Bluetooth module, etc.). The processor 102 may comprise one or more separate computing elements. As an example, the processor 102 may be a central processing unit (CPU). As another example, the processor 102 may comprise a central processing unit and one or more specialized Application-Specific Integrated Circuits (ASICs) interconnected for data transfer by, for example, a data bus.

[0030] The proposed method for obtaining a face image with added lighting comprises the following steps:

[0031] According to a first aspect of the present invention, there is provided a method of obtaining a face image with added lighting, performed by a computer device(or electronic apparats), comprising: obtaining an original face image (input image); generating a masked face crop(or face area) based on the original face image; generating a face depth map corresponding to the original face image, based on the masked face crop; generating a 3D mesh representing a user's face, based on the face depth map; generating a virtual scene(or virtual image) comprising the said 3D mesh and a virtual camera and corresponding to the original face image; adding at least one virtual light source to the virtual scene; generating an additional lighting map(or lighting map) by rendering the virtual scene; generating a face image(or result image) with added lighting based on the original face image and the additional lighting map.

[0032] According to an embodiment of the first aspect, the generating the face depth map corresponding to the original face image, based on the masked face crop comprises face detection and segmentation.

[0033] According to an embodiment of the first aspect, the generating the additional lighting map by rendering the virtual scene comprises performing Physical Based Rendering.

[0034] According to an embodiment of the first aspect, the computer device is one of: a smartphone, a computer device, a video camera, an application specific integrated circuit, a system-on-a-chip.

[0035] According to an embodiment of the first aspect, the method further comprises generating the original face image by means of a camera, wherein the obtaining the original face image comprises receiving the original face image from the camera.

[0036] According to an embodiment of the first aspect, the generating the face depth map comprises generating a depth map by depth estimation based on the masked face crop.

[0037] According to an embodiment of the first aspect, the adding at least one virtual light source to the virtual scene is performed based on user input.

[0038] According to an embodiment of the first aspect, the generating the face image with added lighting based on the original face image and the additional lighting map comprises per-pixel addition of the pixels of the original face image and the pixels of the additional lighting map.

[0039] According to a second aspect of the present invention, there is provided a computer device for obtaining a face image with added lighting, comprising at least one processor and memory, the memory comprising program instructions, wherein, upon execution of the program instructions by the at least one processor, the method according to the first aspect is performed.

[0040] According to a third aspect of the present invention, there is provided a computer-readable storage medium comprising program instructions, which, when executed by at least one processor of a computer, cause the at least one processor to perform the method according to the first aspect.

[0041] Step S101 - obtaining an original face image (FIG. 3, item 201). As a nonlimiting example, the original face image may be obtained (in other words, generated) by means of a camera if the processing device is equipped with a camera. Further, the original face image may be obtained from an external source (for example, an external camera or a device equipped with a camera) via a communication device, or in any other suitable way, depending on the particular implementation. The term "face image" as used herein does not exclude the presence of other objects of the surrounding space in the image (e.g., background or objects in the background of the image, objects in the foreground of the image, etc.).

[0042] The proposed invention does not exclude the presence of multiple faces in the face image. The invention is hereinafter described in the context of the presence of one face in the image, although it is clear that the proposed invention is also applicable to images containing multiple faces.

[0043] Step SI 02 - obtaining a masked face crop (FIG. 3, item 202). A masked face crop herein refers to a face image obtained from the original face image, in which all pixels that do not belong to the face in the face image (i.e., pixels belonging to, for example, the background or other extraneous objects) are marked as not belonging to the face. Thus, the masked face crop contains information about the area of the original face image (in other words, the set of pixels of the original face image) that belongs to the face.

[0044] The invention is not limited to any particular form of representation of the masked face crop. As an example, the masked face crop can be obtained from the original face image by assigning a predetermined code color value (e.g., black, i.e., RGB (0,0,0) accordingto the RGB color model) to all pixels that do not belong to the face. As another example, the masked face crop can be obtained from the original face image by adding a transparency map (e.g., a binary mask) to it so that all pixels that do not belong to the face are completely transparent, and all pixels that belong to the face (i.e., all remaining pixels of the image) are completely opaque.

[0045] As an example, when performing step SI 02, a face detection and segmentation method may be applied to the original face image to obtain information about the area of the face in the image (in other words, the segment of the image that belongs to the face). Then, based on the original face image and the information about the area of the face in the image, a masked face crop can be obtained, for example, by filling all pixels that do not belong to the face area with a code color, or, as another example, by generating and adding a transparency map generated based on the information about the face area in the image.

[0046] Step S103 - generating a face depth map (FIG. 3, item 203) based on the masked face crop. A face depth map is an image or image channel containing information about the distances from the viewpoint (camera position at the time of shooting) to the face for each pixel of the masked face crop that belongs to the face.

[0047] The proposed invention is not limited to any particular method of generating a face depth map based on the masked face crop. As an example, the face depth map may be generated by applying a suitable image depth estimation method to the masked face crop. As a more specific example, a monocular depth estimation method may be used. As another example, the face depth map may be generated by applying a corresponding pre-trained neural network to the masked face crop.

[0048] The proposed invention is not limited to any particular form of representation of information about the distances from the viewpoint to the face. As an example, the face depth map may be a raster image having a width and height corresponding to the width and height of the masked face crop, in which the color of each pixel uniquely corresponds to a certain distance.

[0049] Due to the fact that a masked face crop is used when generating the depth map, the computational complexity of performing this processing can be reduced.

[0050] Step SI 04 - generating a 3D mesh representing the face (FIG. 3, item 204) based on the face depth map. A 3D mesh representing a face may also be referred to as a face 3D mesh for brevity.

[0051] The face 3D mesh can be generated as follows. For each pixel of the face depth map, a ray in three-dimensional space originating from the viewpoint (camera position) can be "restored" based on the pixel coordinates. Information about the "depth" of the corresponding pixel, contained in the face depth map, allows us to restore the corresponding point of the face lying on said ray in three-dimensional space in the coordinate system associated with the camera. Thus, for each pixel of the face depth map, a corresponding point of the face in three- dimensional space in the coordinate system associated with the camera can be restored. And based on the set of pixels and, accordingly, the points of the face, a 3D mesh representing the face can be obtained. Any suitable 3D surface reconstruction method can be used to obtain a 3D mesh from said set of points. The proposed invention is not limited to any particular method.

[0052] It is assumed that information about the parameters of the camera used to capture the original face image may be used when generating the face 3D mesh, although this is not required. Using information about the camera parameters, reflecting the parameters of the camera as an optical system, can improve the accuracy of ray reconstruction in three- dimensional space and, accordingly, the accuracy of the 3D mesh representing the face.

[0053] The generation and use of a 3D mesh, hereinafter in the process of obtaining a face image with added lighting, representing the face makes it possible to simulate facelighting more reliably and improve the quality and realism of the face image with added lighting.

[0054] Step S105 - generating a virtual scene 301, comprising the 3D mesh 302 and a virtual camera 303, corresponding to the original face image. A virtual scene 301 herein refers to a mathematical model of space, represented as a data structure, and including a description of said 3D mesh 301, virtual camera 303, and other objects, such as materials and virtual light sources 304, required to generate an image by performing a rendering process. Generating a virtual scene refers to generating said data structure in memory 103 of computer device 101. FIG. 4 shows a schematic representation of the virtual scene 301. A virtual scene corresponding to the original face image herein means that the position of the virtual camera relative to the mesh corresponds to the position of the real camera (which was used when obtaining the original face image) relative to the face during shooting, and the parameters of the virtual camera (in particular, the horizontal and vertical viewing angles) correspond to the parameters of the real camera used during shooting. Due to said correspondence of the virtual scene to the original face image, the image of the 3D mesh obtained when rendering the virtual scene will exactly match the original face image. That is, when said images are aligned, the image of the mesh representing the face will align with the image of the face in the original face image.

[0055] The scene generated in step SI 05 is assumed to contain information about the following: the position of the face mesh, the position of the virtual camera, the direction of the optical axis of the camera (i.e., the direction of the camera's view), the orientation of the virtual camera (i.e., for example, the direction of "up" of the virtual camera), the horizontal and vertical viewing angles of the virtual camera. Information about the material subsequently applied to the face mesh in step SI 07 of generating the additional lighting map may also be generated and included in the scene in step SI 05.

[0056] The invention is not limited to any particular form of representation of the data structure of the virtual scene. For example, the data structure of the virtual scene may correspond to the particular rendering method used in step SI 07 of generating the additional lighting map, i.e., be compatible with the particular rendering method used.

[0057] The position of the virtual camera and the position of the face mesh refer to their position in the scene coordinate system. The invention is not limited to any particular type of scene coordinate system in which the face 3D mesh, the camera, and one or more light sources that are added to the scene in step SI 06 are "placed".

[0058] Step SI 06 - adding at least one virtual light source 304 to the virtual scene 301. Adding a virtual light source to the virtual scene herein refers to including in the data structure of the virtual scene a model describing the virtual light source (i.e., describing, for example, the position in the scene, type, geometric parameters, light intensity, and light color for the virtual light source). The proposed invention is not limited to any particular method of adding a virtual light source to the virtual scene.

[0059] As a non-limiting example, a virtual light source can be added to the scene according to a predetermined algorithm based on analysis of the original face image. For example, said predetermined algorithm may provide an assessment of the presence of unwanted face shadows in the original face image and place the virtual light source in such a way as to eliminate unwanted shadows, reduce them, or provide another additional lighting effect.

[0060] As another non-limiting example, the adding of at least one virtual light source to the virtual scene may be performed based on user input. The computer device may have a touch screen. The computer device may display a user interface showing the result of applying an additional virtual light source to the original face image and controls allowing the user to change the position and / or parameters of the virtual light source. In an exemplary implementation, the user can interactively change the positions and / or parameters of the virtuallight source, see the results of the changes, and provide a desired position of the virtual light source to obtain a face image with added lighting in the best possible way. It is assumed that in such an exemplary implementation, after each change in the position and / or parameters of the virtual light source in response to user input, steps SI 07 and SI 08 or steps similar thereto are performed. As an example, steps similar to steps SI 07 and SI 08 may be performed, but differing in less resource-intensive processing algorithms or processing algorithm parameters designed to generate a preview image of lower quality, the use of which can provide a higher degree of interactivity. As a result, after each change in the position of the virtual light source, the user can see a face image with added lighting, or, for example, a preview face image with added lighting. By evaluating the face image with added lighting, the user can interactively change the position and / or parameters of the virtual light source until the desired result (in other words, effect) of applying the added lighting is obtained.

[0061] The proposed invention is not limited to any particular type of virtual light source. For example, the virtual light source may be a point light source, a directional light source, an area light source, and the like. It is clear that the method of adding, placing the virtual light source in the scene, and setting its parameters may depend on the specific type of virtual light source.

[0062] Step SI 07 - generating an additional lighting map (FIG. 3, item 205) by rendering the virtual scene. An additional lighting map herein refers to a two-dimensional image containing information about the additional lighting of the face by means of said at least one virtual light source, obtained as a result of rendering the virtual scene. The proposed invention is not limited to any particular method of rendering a virtual scene to generate an additional lighting map.

[0063] As a non-limiting example, physically based rendering (PBR), which provides modeling of the flow of light in the real world, may be used for rendering. Using physically based rendering to generate the additional lighting map allows you to obtain an additional lighting map that is as close as possible to reality (i.e., such an additional lighting map, applied to the original image, will make it possible to obtain a lighting effect that is as close as possible to the actual placement of the light source in a given location relative to the face), which makes it possible to improve the quality and realism of the face image with added lighting.

[0064] It is assumed that, when applying a physically based rendering method or another rendering method to the 3D mesh, a so-called “material” imitating the real optical properties of the skin, such as, for example, absorption, scattering, albedo, subsurface scattering, etc., can be applied (i.e., can be assigned to the 3D mesh for rendering). A material herein refers to a data structure containing said optical and other properties associated with the 3D mesh, used by the Tenderer in the process of rendering the 3D mesh. The material parameters can be set in advance. The material parameters can be determined or refined based on the original face image according to a predetermined algorithm. As an example, the properties of the material can be refined based on the result of evaluating the skin tone of the face, performed in relation to the original face image, which allows improving the quality and realism of the face image with added lighting.

[0065] Since the virtual scene does not contain meshes related to extraneous objects other than the face, the computational complexity of rendering the virtual scene can be reduced.

[0066] It is clear that the particular method and / or parameters of rendering the virtual scene and / or the particular form of representation of the additional lighting map may depend on the particular method of implementing step SI 08 of generating the face image with added lighting, for example, on the method of combining the original face image and the additional lighting map at step SI 08.

[0067] Step SI 08 - generating a face image with added lighting (FIG. 3, item 206) based on the original face image 201 and the additional lighting map 205. This step involves processing the original face image and the additional lighting map according to a predetermined algorithm to obtain a face image with added lighting. The proposed invention is not limited to any particular way of implementing this step. As a non-limiting example, the face image with added lighting may be generated as a result of blending the original face image and the additional lighting map. As a more specific example, said blending may be image addition, in which each color component of each pixel of the face image with added lighting is determined as the sum of the corresponding color components of the corresponding pixels of the original face image and the additional lighting map.

[0068] As a result of performing steps S101-S108, a face image with added lighting is obtained.

[0069] According to another aspect of the present invention, there is provided a computer device 101, adapted to perform the above-described method for obtaining a face image with added lighting, comprising at least one processor 102 and memory 103. Since the computer device 101 has already been described in the context of the proposed method for obtaining a face image with added lighting, details are not repeated here.

[0070] FIG. 5 shows an exemplary flowchart for controlling method of the electronic apparatus.

[0071] In an embodiment, a method for controlling an electronic apparatus include obtaining an input image including an user (S510), identifying a face area indicating a face of the user in the input image (S520), obtaining a face depth map based on the face area (S530), obtaining a 3D mesh representing the face based on the face depth map (S540), obtaining a virtual image corresponding to the face based on the 3D mesh (S550), and obtaining a result image based on the input image and the virtual image (S560).

[0072] The electronic apparats may be described as a computer device or electronic device. The electronic apparats may perform a plurality of operations related with the image enhancement. The image enhancement may include face image processing.

[0073] The target object may include a user object. The user object may not necessarily refer to a specific person. The user object may represent an object that indicates a person.

[0074] The electronic apparats may obtain the input image for the image enhancement. The electronic apparats may obtain the input image including an object corresponding to the user. The object may be described as user object. The electronic apparats may identify a target object in the input image.

[0075] The input image may be described as original face image or input data.

[0076] The electronic apparats may identify the face area in the input image. The electronic apparats may identify the face area among the all areas of the input image. The face area may be described as masked face crop, masked area, cropped area, masking area or target area.

[0077] The electronic apparats may obtain the face depth map corresponding to the face area. The electronic apparats may analyze the face area for obtaining depth data of the face area. The electronic apparats may generate the face depth map based on the depth data of face area. For example, the electronic apparats may obtain depth data for each pixel included in the face area. The electronic apparatus may generate the face depth map based on depth data for each of all pixels included in the face area.

[0078] The electronic apparats may obtain the 3D mesh corresponding to the face area based on the face depth map. The 3D mesh may refer to data representing the shape of a user's facial region, generated through three-dimensional modeling. The 3D mesh may bedescribed as 3D data, 3D shape, 3D form or 3D model. The electronic apparats may generate the 3D mesh including a data indicating a depth of the face of the user.

[0079] The electronic apparats may obtain the virtual image corresponding to the face area based on the 3D mesh. The 3D mesh may simply refer to a set of data representing the face area, whereas the virtual image may represent image data. The virtual image may be described as virtual scene or artificial image. The electronic apparats may generate the virtual image corresponding to the face of the user. The virtual image may include information related to light to be added to the user's face.

[0080] The electronic apparats may obtain the result image by using at least one of the input image or the virtual image. In an embodiment, the electronic apparats may obtain the result image by combining the input image and the virtual image. The result image may be described as face image, output image, rendering image or rendered image. The electronic apparats may generate the result image as the image enhancement.

[0081] The obtaining the result image may include obtaining a lighting map by adding at least one virtual light source to the virtual image, and obtaining the result image based on the input image and the lighting map.

[0082] The electronic apparats may obtain the lighting map corresponding to the face area based on the light. The lighting map may be described as additional lighting map, illumination map, light distribution map or shading map. The lighting map may be described as lighting information, lighting data or light pattern.

[0083] The obtaining the lighting map may include based on user input being received, obtaining the lighting map by adding at least one virtual light source to the virtual image. The electronic apparats may display guide UI(User Interface) for obtaining the lighting map. The electronic apparats may receive the user input for obtaining the lighting map through the guide UI.

[0084] The generating the result image may include obtaining the result image based on a plurality of pixels in the input image and a plurality of pixels of the lighting map.

[0085] The obtaining the lighting map may include changing the virtual image by adding at least one virtual light source to the virtual image, and obtaining the lighting map by rendering the changed virtual image.

[0086] The rendering may include a Physical Based Rendering.

[0087] The obtaining the face depth map may include detecting the face area indicating the face in a whole area of the input image, segmenting the face area from the input image, storing the segmented face area in a memory of the electronic apparatus, and obtaining the face depth map based on the segmented face area.

[0088] The electronic apparatus may be one of: a smartphone, a computer device, a video camera, an application specific integrated circuit or a system-on-a-chip.

[0089] The obtaining the input image may include obtaining the input image by means of a camera.

[0090] The obtaining the face depth map may include obtaining the face depth map by depth estimation based on the input image.

[0091] In an embodiment, an electronic apparatus may include at least one processor including processing circuitry, memory storing instructions. The instructions, when executed by the at least one processor individually or collectively, cause the electronic device to perform a plurality of operations.

[0092] The at least one processor may obtain an input image including an user, identify a face area indicating a face of the user in the input image, obtain a face depth map based on the face area, obtain a 3D mesh representing the face based on the face depth map,obtain a virtual image corresponding to the face based on the 3D mesh, and obtain a result image based on the input image and the virtual image.

[0093] The at least one processor may obtain a lighting map by adding at least one virtual light source to the virtual image, and obtain the result image based on the input image and the lighting map.

[0094] The at least one processor may, based on user input being received, obtain the lighting map by adding at least one virtual light source to the virtual image.

[0095] The at least one processor may obtain the result image based on a plurality of pixels in the input image and a plurality of pixels of the lighting map.

[0096] The at least one processor may change the virtual image by adding at least one virtual light source to the virtual image, and obtain the lighting map by rendering the changed virtual image,

[0097] The rendering include a Physical Based Rendering.

[0098] The at least one processor may detect the face area indicating the face in a whole area of the input image, segment the face area from the input image, store the segmented face area in a memory of the electronic apparatus, and obtain the face depth map based on the segmented face area.

[0099] The electronic apparatus may be one of: a smartphone, a computer device, a video camera, an application specific integrated circuit or a system-on-a-chip.

[0100] The at least one processor may obtain the input image by means of a camera. The electronic apparatus may include the camera. The at least one processor may obtain the input image through the camera included in the electronic apparatus.

[0101] In another embodiment, the at least one processor may obtain the input image from an external apparatus including a camera. The external apparatus may obtain the input image through the camera. The external apparatus may transmit the input image to the electronic apparatus.

[0102] The at least one processor may obtain the face depth map by depth estimation based on the input image.

[0103] According to another aspect of the present invention, there is provided a computer-readable storage medium comprising program instructions which, when executed by at least one processor of a computer, cause the at least one processor to perform the method for obtaining a face image with added lighting. The computer-readable storage medium may be one of the following: Random Access Memory (RAM), flash memory, Erasable Programmable ROM (EPROM), Electrically EPROM (EEPROM), register, hard disk, removable hard disk, compact disk read-only memory (CD-ROM), and the like. Since the method for obtaining a face image with added lighting has been described above, details are not repeated here.

[0104] Thus, the present invention provides an efficient method for obtaining a realistic face image with added lighting, not requiring the use of specialized computing components, such as neural processors, applicable to single face images, and not requiring additional data (such as a set of additional face images, etc.), and not requiring special shooting conditions for the original images.

Claims

Claims

1. A method for controlling an electronic apparatus, the method comprising: obtaining an input image including an user; identifying a face area indicating a face of the user in the input image; obtaining a face depth map based on the face area; obtaining a 3D mesh representing the face based on the face depth map; obtaining a virtual image corresponding to the face based on the 3D mesh; and obtaining a result image based on the input image and the virtual image.

2. The method of claim 1, wherein the obtaining the result image comprises: obtaining a lighting map by adding at least one virtual light source to the virtual image; and obtaining the result image based on the input image and the lighting map.

3. The method of claim 2, wherein the obtaining the lighting map comprises: based on user input being received, obtaining the lighting map by adding at least one virtual light source to the virtual image.

4. The method of claim 2, wherein the generating the result image comprises: obtaining the result image based on a plurality of pixels in the input image and a plurality of pixels of the lighting map.

5. The method of claim 2, wherein the obtaining the lighting map comprises: changing the virtual image by adding at least one virtual light source to the virtual image; and obtaining the lighting map by rendering the changed virtual image.

6. The method of claim 5, wherein the rendering includes Physical Based Rendering.

7. The method of claim 1, wherein the obtaining the face depth map comprises: detecting the face area indicating the face in a whole area of the input image, segmenting the face area from the input image, storing the segmented face area in a memory of the electronic apparatus; and obtaining the face depth map based on the segmented face area.

8. The method of claim 1, wherein the electronic apparatus is one of: a smartphone, a computer device, a video camera, an application specific integrated circuit or a system-on-a- chip.

9. The method of claim 1, wherein the obtaining the input image comprises: obtaining the input image by means of a camera.

10. The method of claim 1, wherein the obtaining the face depth map comprises: obtaining the face depth map by depth estimation based on the input image.

11. An electronic apparatus, comprising: at least one processor including processing circuitry; memory storing instructions; wherein the instructions, when executed by the at least one processor individually or collectively, cause the electronic device to: obtain an input image including an user; identify a face area indicating a face of the user in the input image; obtain a face depth map based on the face area; obtain a 3D mesh representing the face based on the face depth map; obtain a virtual image corresponding to the face based on the 3D mesh; and obtain a result image based on the input image and the virtual image.

12. The electronic apparatus of claim 11, wherein the instructions, when executed by the at least one processor individually or collectively, cause the electronic device to: obtain a lighting map by adding at least one virtual light source to the virtual image; and obtain the result image based on the input image and the lighting map.

13. The electronic apparatus of claim 12, wherein the instructions, when executed by the at least one processor individually or collectively, cause the electronic device to: based on user input being received, obtain the lighting map by adding at least one virtual light source to the virtual image.

14. The electronic apparatus of claim 12, wherein the instructions, when executed by the at least one processor individually or collectively, cause the electronic device to: obtain the result image based on a plurality of pixels in the input image and a plurality of pixels of the lighting map.

15. The electronic apparatus of claim 12, wherein the instructions, when executed by the at least one processor individually or collectively, cause the electronic device to: change the virtual image by adding at least one virtual light source to the virtual image; and obtain the lighting map by rendering the changed virtual image.

Images