Generation of reflectance maps for reilluminable 3D models

Abstract

An electronic device and method for generating a reflectogram for a reilluminable 3D model are disclosed. An electronic device acquires multi-view image data including a set of images of an object, and generates a 3D mesh of the object based on the multi-view image data. The electronic device obtains a set of motion-corrected images based on minimization of rigid motion associated with the object between images of the set of images and generates a texture map in the UV space based on the set of motion-corrected images and the 3D mesh. The electronic device obtains a specular reflection map and a diffuse reflection map based on separating the specular reflection component and the diffuse reflection component from the texture map, and obtains a reilluminable 3D model of the object based on the specular reflection map and the diffuse reflection map.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS / INCORPORATED BY REFERENCE

[0002] This application claims the benefit of priority to U.S. Patent Application No. 17 / 965,425, filed in the U.S. Patent Office on October 13, 2022. Each of the above-referenced applications is incorporated herein by reference in its entirety. Technical Field

[0003] Various embodiments of the present disclosure relate to 3D modeling. More specifically, various embodiments of the present disclosure relate to electronic devices and methods for generating reflectance maps for relightable 3D models. Background Art

[0004] Advances in the field of three-dimensional (3D) computer graphics have provided the ability to create 3D models and visualize real objects in a 3D computer graphics environment. A 3D model is a static 3D mesh that resembles the shape of a specific object. Typically, such 3D models are manually designed by computer graphics artists (often referred to as modelers) by using modeling software applications. Such 3D models may not be used in the same way for animation or various virtual reality systems or applications. Texture mapping is an important method for defining the texture details to be applied to a 3D model in order to texturize the 3D model. Creating realistic 3D models and high-fidelity textures / reflections Figure 1 This has always been a difficult problem in the field of computer graphics and computer vision. With the increase in applications in virtual reality, 3D avatars, games, and virtual simulations, it is becoming increasingly important to generate accurate and high-fidelity texture or reflection maps and give 3D models photo-realism.

[0005] By comparing the described system with certain aspects of the present disclosure as set forth in the remainder of this application and with reference to the accompanying figures, the limitations and disadvantages of conventional and traditional approaches will become clear to those skilled in the art. Summary of the invention

[0006] As more fully set forth in the claims, there is provided an electronic device and method for generating a reflectance map for a relightable 3D model, substantially as shown and / or described in connection with at least one of the figures.

[0007] These and other features and advantages of the present disclosure may be understood by reviewing the following detailed description of the disclosure and the accompanying drawings, wherein like reference numerals refer to like parts throughout. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a block diagram illustrating an exemplary network environment for generating a reflectance map for a relightable 3D model according to an embodiment of the present disclosure.

[0009] Figure 2 is a diagram illustrating an embodiment according to the present disclosure Figure 1 A block diagram of an exemplary electronic device.

[0010] Figure 3A and Figure 3B are diagrams that collectively illustrate an exemplary processing pipeline for generating a reflectance map according to an embodiment of the present disclosure.

[0011] Figure 4 is a diagram illustrating an exemplary processing pipeline for obtaining a set of color-corrected images according to an embodiment of the present disclosure.

[0012] Figure 5 is a diagram illustrating an exemplary processing pipeline for generating a refined normal map according to an embodiment of the present disclosure.

[0013] Figure 6 is a diagram illustrating an exemplary processing pipeline for generating a reflectance map for a relightable 3D model according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0014] The embodiments described below may be found in electronic devices and methods for generating a reflection map for a relightable 3D model. Exemplary aspects of the present disclosure may provide an electronic device (e.g., a server, desktop, laptop, or personal computer) that can perform operations to generate a reflection map for a relightable 3D model. The electronic device may acquire multi-view image data including a set of images of an object. The object may be exposed to a set of illumination patterns (e.g., omnidirectional illumination patterns, polarized and / or gradient illumination patterns) during the capture duration of the multi-view image data. The electronic device may generate a 3D mesh of the object based on the multi-view image data, and may obtain a set of motion-corrected images based on minimization of rigid motion associated with the object between images in the set of images. The electronic device may generate a texture map in UV space based on the set of motion-corrected images and the 3D mesh. The electronic device may obtain a specular reflection map and a diffuse reflection map based on separation of a specular reflection component and a diffuse reflection component from the texture map. The electronic device may obtain a relightable 3D model of the object based on the specular reflection map and the diffuse reflection map.

[0015] Typically, a 3D model may be manually designed by a computer graphics artist (often referred to as a modeler) using a modeling software application. Such a 3D model may not be used in the same way for animation or various virtual reality systems or applications, and texture mapping may be used to define texture details to be applied to the 3D model to texture the 3D model. Creating realistic models and textures Figure 1It has always been a difficult problem in computer graphics and computer vision. Moreover, estimating full-head skin reflectance is key to generating relightable 3D head models for photo-realistic game and movie creation.

[0016] To address these requirements, the present disclosure introduces a method for generating high-quality and high-resolution skin reflectance maps, including diffuse, specular, color, normal, and height maps for objects such as 3D head scans using polarized spherical gradient illumination patterns. The present disclosure also introduces a robust diffuse and specular separation method that allows the camera to be positioned farther from the equator of the light cage. The present disclosure also introduces an operation for generating a color map to match the unpolarized scan results and a pipeline for generating and refining a normal map. The present disclosure also introduces a fast color correction method and an operation for performing fast inter-frame motion correction.

[0017] Figure 1 is a block diagram illustrating an exemplary network environment for generating a reflectance map for a relightable 3D model according to an embodiment of the present disclosure. Figure 1 , a network environment 100 is shown. The network environment 100 may include an electronic device 102, a server 104, a database 106, an imaging setup 108, and a communication network 110. The database 106 may include multi-view image data 112. The imaging setup 108 may include a first structure 114A, a second structure 114B, and an Nth structure 114N.

[0018] refer to Figure 1 , also shows a plurality of image capture devices 116 that can be mounted on a 3D cage structure including a first structure 114A, a second structure 114B, and an Nth structure 114N. The plurality of image capture devices 116 may include, for example, a first image capture device 116A, a second image capture device 116B, and an Nth image capture device 116N. The electronic device 102 and the server 104 may be communicatively coupled to each other via a communication network 110. Figure 1 , a user 120 (eg, a 3D artist or developer) and an object (eg, actor 118) that may be associated with the electronic device 102 are also shown.

[0019] The electronic device 102 may include suitable logic, circuitry, interfaces, and / or code that may be configured to acquire multi-view image data 112 including a set of images of an object, such as an actor 118. Based on the image data, the electronic device 102 may perform a set of operations to generate a set of reflectance maps that may be required to obtain a relightable 3D model of the object. The object may be exposed to a set of lighting patterns for a capture duration of the multi-view image data. Examples of the electronic device 102 may include, but are not limited to, a computing device, a smart phone, a cellular phone, a mobile phone, a gaming device, a mainframe, a server, a computer workstation, and / or a consumer electronics (CE) device.

[0020] The server 104 may include suitable logic, circuit systems and interfaces and / or code that may be configured to perform operations such as data / file storage, 3D rendering, or 3D reconstruction operations (such as photogrammetric reconstruction operations). In one or more embodiments, the server 104 may store multi-view image data and may perform at least one operation associated with the electronic device 102. The server 104 may be implemented as a cloud server and may perform operations through web applications, cloud applications, HTTP requests, repository operations, file transfers, etc. Other example implementations of the server 104 may include, but are not limited to, a database server, a file server, a web server, a media server, an application server, a mainframe server, or a cloud computing server.

[0021] In at least one embodiment, the server 104 can be implemented as a plurality of distributed cloud-based resources using several techniques well known to those of ordinary skill in the art. It will be appreciated by those of ordinary skill in the art that the scope of the present disclosure may not be limited to implementing the server 104 and the electronic device 102 as two separate entities. In some embodiments, the functionality of the server 104 may be fully or at least partially incorporated into the electronic device 102 without departing from the scope of the present disclosure. In some embodiments, the server 104 may host the database 106. Alternatively, the server 104 may be separate from the database 106 and may be communicatively coupled to the database 106.

[0022] The database 106 may include suitable logic, interfaces, and / or code that may be configured to store the multi-view image data 112 or metadata associated with the multi-view image data 112. For example, the metadata may include an identifier of an image capture device that captured the image, an illumination pattern used at the time of capture, or an identifier of a viewpoint from which the image was captured, or an index value indicating the position of the image within a set of images (included in the multi-view image data 112). The database 106 may be stored or cached on a device such as a server (e.g., server 104) or the electronic device 102. The device storing the database 106 may be configured to receive queries for the multi-view image data 112 or metadata. In response, the device storing the database 106 may retrieve and provide the multi-view image data 112 or metadata to the electronic device 102.

[0023] In some embodiments, database 106 can be hosted on multiple servers stored in the same or different locations. The operations of database 106 can be performed using hardware, where the hardware includes a processor, a microprocessor (e.g., to perform one or more operations or control the execution of one or more operations), a field programmable gate array (FPGA), or an application specific integrated circuit (ASIC). In some other cases, database 106 can be implemented using software.

[0024] The imaging setup 108 may correspond to a 3D cage structure, and a plurality of image capture devices 116 may be deployed on the cage structure and oriented to scan objects within the 3D cage structure from multiple viewpoints. The imaging setup 108 may include a plurality of structures 114, each of which may be connected at certain locations to form a cage-like structure (e.g., a 3D dome structure as shown). The present disclosure may not be limited to any particular shape of the 3D cage structure. In some embodiments, the shape of the cage-like structure may be cylindrical, cuboid, or any arbitrary shape, depending on the requirements of the volume studio / capture. In some embodiments, each of the plurality of structures 114 may have the same or different dimensions, depending on the requirements of the volume studio / capture. In addition to the plurality of image capture devices 116, a plurality of audio capture devices (not shown) and / or a plurality of light sources (not shown) may also be deployed at certain locations on the plurality of structures 114 to form the imaging setup 108.

[0025] By way of example, and not limitation, each structure may include a mount to hold at least one image capture device (in Figure 1 and at least one processing device. Figure 1As shown in , each structure (e.g., truss) may include a frame of a particular material (e.g., metal, plastic, or fiber) to hold at least one of an image capture device, a processing device, an audio capture device, and a light source (e.g., a flash). Different 3D structures of the same or different shapes may be connected to form an imaging setup 108. In an embodiment, the processing device may be the electronic device 102.

[0026] In some embodiments, a mobile imaging setup may be created. In such embodiments, each of the plurality of structures 114 of the mobile imaging setup may correspond to an unmanned aerial vehicle (UAV), and the plurality of image capture devices 116, the plurality of light sources, and / or other devices may be mounted on the plurality of unmanned aerial vehicles (UAVs).

[0027] The communication network 110 may include a communication medium through which the electronic device 102 and the server 104 can communicate with each other. The communication network 110 may be one of a wired connection or a wireless connection. Examples of the communication network 110 may include, but are not limited to, the Internet, a cloud network, a cellular or wireless mobile network (such as long-term evolution and fifth-generation (5G) new radio (NR)), a wireless fidelity (Wi-Fi) network, a personal area network (PAN), a local area network (LAN), or a metropolitan area network (MAN). Various devices in the network environment 100 may be configured to connect to the communication network 110 according to various wired and wireless communication protocols. Examples of such wired and wireless communication protocols may include, but are not limited to, Transmission Control Protocol and Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), Zig Bee, EDGE, IEEE 802.11, Light Fidelity (Li-Fi), 802.16, IEEE 802.11s, IEEE 802.11g, multi-hop communication, wireless access point (AP), device-to-device communication, cellular communication protocol, and at least one of Bluetooth (BT) communication protocol.

[0028] In operation, the electronic device 102 may be configured to acquire multi-view image data 112 including a set of images of an object, such as an actor 118. The object may be exposed to a set of illumination patterns during a capture duration of the multi-view image data 112. By way of example, and not limitation, the set of illumination patterns may include one or more of a cross-polarized omnidirectional illumination pattern, a gradient illumination pattern, and a polarized illumination pattern, including a cross-polarized illumination pattern and a parallel polarized illumination pattern.

[0029] In an exemplary embodiment, the object may be a human head (with a face), and multi-view image data 112 may be acquired from an imaging setup 108 that may operate as a polarization-based light cage. The object may be scanned from multiple viewpoints via one or more cameras of the imaging setup 108 to obtain the multi-view image data 112. In order to obtain high-fidelity reflections and normal / height maps of the object, the object must be exposed to different illumination patterns while capturing images of the object from different viewpoints. For example, in Figure 3A Further details regarding the multi-view image data 112 are provided in FIG.

[0030] The electronic device 102 may be configured to generate a 3D mesh of an object based on the multi-view image data 112. By way of example, and not limitation, a 3D mesh may be generated from a set of images using methods based on photogrammetry, such as structure from motion (SfM), methods requiring stereo images, or methods requiring monocular cues, such as shape from shading (SfS), photometric stereo, or shape from texture (SfT). For the sake of brevity, the present disclosure omits the details of such methods. The 3D mesh may be a textureless mesh that resembles the 3D shape of the object. The 3D mesh may use polygons to define the shape or geometry of the object. For example, in Figure 3A An example 3D mesh of a human head is provided in .

[0031] When scanning an object to capture a set of images, the object may need to remain still throughout the scanning phase. However, the object may have some unavoidable movement (e.g., head movement). The actual inter-frame motion may be assumed to be small. Rigid motion may be estimated and removed by performing tile matching between images or frames to obtain a set of motion-corrected images. Specifically, the electronic device 102 may obtain a set of motion-corrected images based on minimizing rigid motion associated with the object between images in the set of images. For example, in Figure 3A Further details regarding a set of motion corrected images are provided at 306 .

[0032] In some cases, it may be necessary to perform color correction on the motion corrected images to obtain accurate albedo values ​​for both diffuse and specular components. Therefore, color correction may be applied to a subset of images in a set of motion corrected images to obtain a set of color corrected images.

[0033] In order to obtain a relightable 3D model, it may be necessary to separate the specular component and the diffuse component from the image to obtain the albedo for the 3D mesh. The accuracy of the separation generally decreases as the camera view increases. Therefore, a texture map can be generated in UV space, and then the diffuse and specular separation can be performed in UV space. The electronic device 102 can generate a texture map in UV space based on a set of motion-corrected images (or color-corrected images) and the 3D mesh. The texture map can help include high-frequency details of the object onto the 3D model of the object, and such a map can be obtained based on mapping a set of motion-corrected images to UV space. Examples of texture maps may include cross-polarized UV texture maps and parallel-polarized UV texture maps. For example, in Figure 3B Further details related to the texture map are provided at 308.

[0034] The electronic device 102 may obtain the specular reflection map and the diffuse reflection map based on separating the specular reflection component and the diffuse reflection component from the texture map. The specular reflection map may depict the glossiness of the surface of the object, while the diffuse reflection map may depict the reflection from the object without any atmospheric reflection. The specular reflection component and the diffuse reflection component may be separated from each texture map (i.e., the cross-polarized UV texture map and the parallel-polarized UV texture map). For example, in Figure 3B Details related to specular and diffuse maps are provided in.

[0035] The electronic device 102 may also obtain a relightable 3D model of the object based on the specular reflection map and the diffuse reflection map. The relightable 3D model may be a static 3D mesh similar to the shape of the object (e.g., the head of the actor 118). Figure 3B Details related to the relightable 3D model are provided at 314 .

[0036] Figure 2 is a diagram illustrating an embodiment according to the present disclosure Figure 1 A block diagram of an exemplary electronic device of Figure 1 To explain the elements in Figure 2 . refer to Figure 2 , an electronic device 102 is shown. The electronic device 102 may include a circuit system 202, a memory 204, an input / output (I / O) device 206, and a network interface 208. The input / output (I / O) device 206 may include a display device 210.

[0037] The circuit system 202 may include suitable logic, circuit systems and / or interfaces that may be configured to execute program instructions associated with different operations to be performed by the electronic device 102. The circuit system 202 may include one or more processing units that may be implemented as separate processors. In an embodiment, one or more processing units may be implemented as an integrated processor or a cluster of processors that together perform the functions of one or more dedicated processing units. The circuit system 202 may be implemented based on a variety of processor technologies known in the art. Examples of implementations of the circuit system 202 may be based on an X86-based processor, a graphics processing unit (GPU), a reduced instruction set computing (RISC) processor, an application specific integrated circuit (ASIC) processor, a complex instruction set computing (CISC) processor, a microcontroller, a central processing unit (CPU), and / or other control circuits.

[0038] The memory 204 may include suitable logic, circuitry, interfaces, and / or code that may be configured to store one or more instructions to be executed by the circuitry 202. The memory 204 may be configured to store multi-view image data. Examples of implementations of the memory 204 may include, but are not limited to, a random access memory (RAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a hard disk drive (HDD), a solid state drive (SSD), a CPU cache, and / or a secure digital (SD) card.

[0039] The I / O device 206 may include suitable logic, circuitry, interfaces, and / or code that may be configured to receive input and provide output based on the received input. For example, the I / O device 206 may receive a first user input indicating a selection of the multi-view image data 112. The I / O device 206 may also be configured to display a set of images included in the multi-view image data 112. The I / O device 206 may include a display device 210. Examples of the I / O device 206 may include, but are not limited to, a touch screen, a keyboard, a mouse, a joystick, a microphone, or a speaker.

[0040] The network interface 208 may include suitable logic, circuit systems, interfaces, and / or code that may be configured to facilitate communication between the electronic device 102 and the server 104 via the communication network 110. The network interface 208 may be implemented using various known technologies to support wired or wireless communication between the electronic device 102 and the communication network. The network interface 208 may include, but is not limited to, an antenna, a radio frequency (RF) transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a coder-decoder (CODEC) chipset, a subscriber identity module (SIM) card, or a local buffer circuit system.

[0041] The network interface 208 may be configured to communicate with a network such as the Internet, an intranet, a wireless network, a cellular telephone network, a wireless local area network (LAN), or a metropolitan area network (MAN) via wireless communication. The wireless communication may be configured to use one or more of a variety of communication standards, protocols, and technologies, such as Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (W-CDMA), Long Term Evolution (LTE), Fifth Generation (5G) New Radio (NR), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (such as IEEE 802.11a, IEEE 802.11b, IEEE802.11g, or IEEE 802.11n), Voice over Internet Protocol (VoIP), Light Fidelity (Li-Fi), Worldwide Interoperability for Microwave Access (Wi-MAX), email protocols, instant messaging, and short message service (SMS).

[0042] The display device 210 may include suitable logic, circuit systems, and interfaces that may be configured to display a 3D mesh and / or a set of images contained in the multi-view image data 112. The display device 210 may be a touch screen that enables a user (e.g., user 120) to provide user input via the display device 210. The touch screen may be at least one of a resistive touch screen, a capacitive touch screen, or a thermal touch screen. The display device 210 may be implemented using several known technologies, such as, but not limited to, at least one of a liquid crystal display (LCD) display, a light emitting diode (LED) display, a plasma display, or an organic LED (OLED) display technology, or other display devices. Depending on the embodiment, the display device 210 may refer to a display screen of a head mounted device (HMD), a smart glasses device, a see-through display, a projection-based display, an electrochromic display, or a transparent display. For example, in Figure 3A and Figure 3B Various operations of the circuit system 202 for generating a reflectance map for a relightable 3D model are further described in .

[0043] Figure 3A and Figure 3B are diagrams that collectively illustrate an exemplary processing pipeline for generating a reflectance map according to an embodiment of the present disclosure. Figure 3A and Figure 3B Combined from Figure 1 and Figure 2 The elements of Figure 3A and Figure 3B , shows an exemplary processing pipeline 300 illustrating exemplary operations from 302 to 314. The exemplary operations 302 to 314 may be performed by any computing system, for example, by Figure 1 Electronic device 102 or Figure 2 The exemplary processing pipeline 300 also illustrates multi-view image data 302A, a 3D mesh 304A, a set of motion-corrected images 306A, a specular map 312A, a diffuse map 312B, and a relightable 3D model 314A. The multi-view image data 302A may include N images, such as image 316A, image 316B, and image 316N. Image 316N includes a color checkerboard image 318 and a face or head of an object. Figure 3A The number of images shown in is presented merely as an example, and such an example should not be construed as limiting the present disclosure.

[0044] At 302, multi-view image data acquisition may be performed. Circuit system 202 may acquire multi-view image data 302A including a set of images of an object. The object may be exposed to a set of illumination patterns for a capture duration of multi-view image data 302A. The object may be any animate or inanimate object. Figure 3A An example of an object as a human head is shown in FIG. When the object is exposed to a set of illumination patterns for a capture duration, one or more cameras can scan the object from each viewpoint to capture one or more images. According to each camera view, multiple images of the object can be captured. The electronic device 102 can receive a set of images (i.e., multi-view high-resolution images) from one or more cameras. The set of images may include (one or more) images with different illumination patterns and viewpoints.

[0045] In an embodiment, the object may be a human head, and multi-view image data 302A may be acquired from an imaging setup (e.g., imaging setup 108) operating as a polarization-based light cage. The light cage may be, for example, a dome-shaped cage structure that may include one or more image capture devices placed at different locations on the cage structure and a plurality of movable or static lighting devices. The lighting devices may emit different lighting patterns based on one or more control signals from the electronic device 102 or from an independent controller device. In the case of a polarization-based light cage, the lighting devices may emit polarized light (cross- or parallel polarizations). The object (i.e., an actor) may sit in the center of the light cage, and each image capture device may capture an image of the human head when the human head is exposed to a set of lighting patterns. For example, the object may be an actor, and one or more image capture devices may capture a set of images of the actor's head under different lighting patterns.

[0046] In an embodiment, the set of illumination patterns may include one or more of a cross-polarized omnidirectional illumination pattern, a gradient illumination pattern, and a polarized illumination pattern, the polarized illumination pattern including a cross-polarized illumination pattern and a parallel polarized illumination pattern. In an embodiment, the set of illumination patterns may include a minimum of six polarized gradient illumination patterns, including a cross-polarized illumination pattern and a parallel polarized illumination pattern in three axes (i.e., an "X" axis, a "Y" axis, and a "Z" axis). In some cases, it may be preferred that using nine gradient illumination patterns can provide better quality normal generation than using a minimum of six or a maximum of twelve gradient illumination patterns without performing motion correction.

[0047] At 304, a 3D mesh may be generated. The circuit system 202 may be configured to generate a 3D mesh 304 of an object based on the multi-view image data 302A. The 3D mesh 304 may be a texture-free base mesh that may be used in operations associated with generating a texture, reflection, or normal / height map for the object. As discussed, a 3D mesh may be generated from a set of images using photogrammetry-based methods such as structure from motion (SfM), methods requiring stereo images, or methods requiring monocular cues such as shape from shading (SfS), photometric stereo, or shape from texture (SfT). For brevity, the present disclosure omits the details of such methods.

[0048] At 306, a set of motion-corrected images may be generated. In many cases, an object does not remain stationary for the entire duration of capturing a set of images. For example, an object may be an actor's head, whose image may be captured. Head movement may be unavoidable during the duration of capture. Typically, rigid motion may be estimated and removed based on an estimate and alignment of the 3D position of a marker or encoding target placed on a hat (worn by the actor). However, many studios prefer to capture images of actors with hair (i.e., without a hat). In this case, an encoding target or marker may not be appropriate. If it is assumed that the object remains stationary for at least one second, the actual inter-frame motion may be assumed to be tiny. Rigid motion may be removed by performing tile matching between images to obtain a set of motion-corrected images 306A. The circuit system 202 may be configured to obtain a set of motion-corrected images 306A based on minimization of rigid motion between images in a set of images.

[0049] At 308, color correction may be performed. Circuit system 202 may be configured to acquire a color checkerboard image that may be exposed to a cross-polarized omnidirectional illumination pattern. Circuit system 202 may estimate a color matrix based on a comparison of color values ​​of color checkerboard image 308A with reference color values. Circuit system 202 may apply the estimated color matrix to a subset of images in a set of motion-corrected images 306A to obtain a set of color-corrected images. The subset of images may be associated with one of an omnidirectional cross-polarized illumination pattern or a parallel polarized illumination pattern.

[0050] Typically, for reflectance map generation, one can directly obtain the input image (e.g., from the cross-polarization Image and parallel polarization The specular reflection component and the diffuse reflection component are separated from each other in the image (i.e., the image) to generate a view-dependent specular reflection component and a diffuse reflection component for each camera view. Each of the specular reflection component and the diffuse reflection component can be mapped to UV space to generate a texture map. The processing order mentioned above can be heavily dependent on the accuracy of polarization separation. Unfortunately, the accuracy of polarization separation decreases as the camera view increases, which may be far away from the equator of the polarization-based light cage, resulting in a view-dependent specular reflection map. As a result, the generated specular reflection Figure 1 Generally, it will be noisy and sometimes have significant artifacts. Instead of the above mentioned method, the present disclosure may first perform UV mapping on the multi-view cross-polarization and parallel polarization images, and then may perform diffuse and specular reflection separation in UV space, for example, as described in 310 and 312. This may generate a higher quality reflection map.

[0051] At 310, a texture map may be generated. The circuit system 202 may generate a texture map in a UV space based on a set of motion-corrected images and a 3D mesh. Alternatively, a texture map may be generated based on a set of color-corrected images and a 3D mesh. By way of example, and not limitation, the 3D mesh may be expanded into a two-dimensional (2D) UV space to obtain a UV map. In a UV mapping operation, a color value may be applied to the UV map based on an affine transformation between a plurality of triangles of the 3D mesh in the UV map and a plurality of corresponding triangles in the set of motion-corrected (or color-corrected) images. Since a set of images includes one or more images captured under a cross-polarized illumination pattern and one or more images captured under a parallel polarized illumination pattern, UV mapping may be performed on multi-view cross-polarized and parallel polarized images. The view-related specular component may be considered relatively small compared to the view-related color information. One or more images captured under a cross-polarized illumination pattern may be merged in a UV space to obtain a cross-polarized UV texture map. In addition, one or more images captured under a parallel polarized illumination pattern may be merged in a UV space to obtain a parallel polarized UV texture map. The texture map may then be used to perform diffuse and specular separation in UV space, as described in 312 .

[0052] At 312, a reflectance map may be obtained. The circuit system 202 may be configured to obtain specular and diffuse reflectance maps 312A and 312B based on separating the specular and diffuse reflectance components from the texture map. More specifically, the specular and diffuse reflectance components may be separated from the cross-polarized UV texture map and the parallel-polarized UV texture map. The specular reflectance map 312A may depict the glossiness of a surface of an object (such as human skin), while the diffuse reflectance map 312B may depict the reflection from the object without any atmospheric reflection. Separating the specular and diffuse reflectance components from the texture map may enable the generation of a higher quality reflectance map. In addition, the above-mentioned method may allow the camera to be placed at a position 45 to 60 degrees from the equator within a polarization-based light cage without generating polarization-related artifacts on the albedo. Moreover, in order to separate the specular and diffuse reflectance components from the texture map, the camera may not need to be centered around the equator.

[0053] The separated diffuse and specular components may contain ambient occlusion (AO) which may need to be removed to generate the albedo. Ambient occlusion may be a shadow present in a set of images due to occlusion between a point on the surface of an object and a light source. Certain points on an object may be occluded from the light source, and the shadows of such points may make certain parts of the relightable 3D model darker. Therefore, the specular and diffuse maps 312A and 312B may appear inaccurate. According to an embodiment, ambient occlusion may be roughly estimated from a 3D shape by a path tracing operation. Specifically, a hemisphere of light may be considered to originate from a given point, and the path of each light ray may be traced to check for intersections of the light ray with obstacles. Once ambient occlusion is estimated, ambient occlusion may be removed from the specular and diffuse maps 312A and 312B to obtain refined specular and diffuse maps. In an embodiment, circuitry 202 may be configured to estimate AO based on the 3D shape of 3D mesh 304A and refine specular and diffuse maps 312A and 312B based on removing ambient occlusion from specular and diffuse maps 312A and 312B, respectively.

[0054] At 314, a relightable 3D model 314A may be obtained. The circuit system 202 may be configured to obtain a relightable 3D model 314A of the object based on the specular and diffuse reflectance maps 312A and 312B. The relightable 3D model 314A may be a model of the object in 3D that may be illuminated with a different lighting pattern than the set of lighting patterns under which the set of images were captured.

[0055] In an embodiment, the circuit system 202 may also be configured to generate a color map based on a linear combination of the specular and diffuse maps 312A and 312B. In some cases, game and movie studios may prefer to maintain consistency in color or contrast between the multi-view image data 302A (acquired from the imaging setup 108 (i.e., a polarization-based light cage)) and the texture map generated by a traditional 3D / 4D scanning system using non-polarized illumination. However, due to the polarized illumination, the diffuse map 312B generated from the multi-view image data 302A may have lower contrast and more saturated colors. Due to the difference between the color checkerboard image (as shown in image 316N) and the actor's skin, color matching using the color checkerboard image may not work in this case. Therefore, the color map I may be generated based on a linear combination of the specular map 312A and the diffuse map 312B. i Color , as given by equation (1):

[0056]

[0057] Among them, I iColor It can be a color map, I i specular can be the specular component of a 3D surface point and I i diffuse λ may be the diffuse reflection component of the 3D surface point. λ may be a constant. When λ is 1, the linear combination of the specular reflection map 312A and the diffuse reflection map 312B may match a texture map generated by a conventional 3D / 4D scanning system using non-polarized illumination.

[0058] Figure 4 is a diagram illustrating an exemplary processing pipeline for obtaining a set of color-corrected images according to an embodiment of the present disclosure. Figure 4 is a combination of Figure 1 , Figure 2 , Figure 3A and Figure 3B Reference Figure 4 , shows an exemplary processing pipeline 400, which illustrates exemplary operations 402 to 406 for obtaining a set of color-corrected images. The exemplary operations 402 to 406 may be performed by any computing system, such as by Figure 1 The electronic device 102 or Figure 2 The circuit system 202 is executed. The exemplary processing pipeline 400 also illustrates a color checkerboard image 402A and a set of color corrected images 406A.

[0059] At 402, color checkerboard image acquisition can be performed. Circuit system 202 can be configured to acquire color checkerboard image 402A. Color checkerboard image 402A can be an x-rite color checkerboard image, which can be captured under omnidirectional cross-polarized illumination. Color checkerboard image 402A can include various color gradients and can be used as a reference for color correction.

[0060] At 404, color matrix estimation can be performed. Conventionally, color checkerboard image 402A can be used to perform color correction on each image in a group of images. An image of an object can be captured by color checkerboard image 402A. The color of each image can be matched with color checkerboard image 402A to balance the color in each image in a group of images. However, due to repeated color matrix estimation processing, repeated file reading operations and unnecessary color correction to certain frames, this processing is generally slow for large-scale image data sets. In contrast, circuit system 202 can estimate color matrix based on the comparison of color values ​​of color checkerboard image 402A with reference color values. The value of each color in color checkerboard image 402A can be compared with the reference color value, and the color matrix can be estimated using the least square method. As an example, the color matrix can include three rows and three columns.

[0061] At 406, the estimated color matrix may be applied to a subset of images. Circuitry 202 may apply the estimated color matrix to only a subset of images in a set of motion corrected images to obtain a set of color corrected images. The subset of images may be associated with one of an omnidirectional cross-polarized illumination pattern or a parallel polarized illumination pattern. Since the normal and height map generation process is largely irrelevant to image color accuracy, color correction may not be performed on other motion corrected images associated with polarized illumination patterns. By correcting a subset of images, less time may be spent performing color correction compared to conventional methods for color correction.

[0062] In an embodiment, the set of motion corrected images includes fourteen bit raw files that may be converted to sixteen bit demosaiced images to preserve dynamic range.A linear color space may be selected because the generation of normal and height maps may require linear intensity measurements.

[0063] Figure 5 is a diagram illustrating an exemplary processing pipeline for generating a refined normal map according to an embodiment of the present disclosure. Figure 5 is a combination of Figure 1 , Figure 2 , Figure 3A , Figure 3B and Figure 4 Reference Figure 5 , shows an exemplary processing pipeline 500, which illustrates exemplary operations 502 to 518 for generating a reflectance map for a relightable 3D model. The exemplary operations 502 to 514 may be performed by any computing system, for example, by Figure 1 The electronic device 102 or Figure 2 The exemplary processing pipeline 500 also illustrates a set of specularly separated gradient images 502A, surface normals 504A, geometric normals 506A, rotation matrices 508A, normal maps 514A, height maps 516A, and refined normal maps 518A.

[0064] At 502, a specularly separated gradient image may be obtained. The circuit system 202 may be configured to generate a specularly separated gradient image based on the image obtained from the specularly separated gradient image. Figure 1 and Figure 3A-3B The diffuse reflection component may be removed from each image in a set of motion-corrected images (obtained in the example of FIG. 5 ) to obtain a set of specular reflection separated gradient images 502A. While the diffuse reflection component may be removed from each image associated with the gradient illumination pattern and included in the set of motion-corrected images, the specular reflection component may be retained in each image associated with the gradient illumination pattern.

[0065] At 504, surface normal calculations may be performed. The circuit system 202 may also be configured to calculate a surface normal 504A corresponding to each pixel of an image in the set of specular reflection separated gradient images 502A. The surface normal 504A may be a vector perpendicular to a given pixel of an image in the set of specular reflection separated gradient images 502A. The calculated surface normal 504A may contain finer microgeometric details. However, the calculated surface normal 504A may be located in a world space, which may not correspond to the grid space of a 3D grid. In addition, due to the nature of photometric-based normal estimation, the calculated surface normal 504A may have a non-uniform low-frequency deviation. Therefore, the calculated surface normal 504A may need to be converted from world space to object space. In addition, the converted surface normal may be corrected based on removing the low-frequency deviation from the converted surface normal.

[0066] At 506, a geometric normal calculation may be performed. The circuit system 202 may be configured to calculate a geometric normal 506A based on a linear combination of a set of vertex normals corresponding to the 3D mesh. The combination may be performed based on the barycentric coordinates of each pixel of the texture map generated in UV space. The barycentric coordinates may define the position of each pixel of the texture map generated in UV space relative to a reference triangle or tetrahedron. The geometric normal 506A may have the same level of geometric detail as the 3D mesh.

[0067] At 508, a surface normal conversion may be performed. Circuit system 202 may be configured to convert surface normal 504A from world space to object space based on application of rotation matrix 508A to surface normal 504A. Surface normal 504A may be located in world space (i.e., the "X", "Y", and "Z" axes of a set of illumination patterns) and may include finer micro-geometric details of an object. Since surface normal 504A is located in world space (which does not correspond to grid space (i.e., 3D grid space)), it may be necessary to convert surface normal 504A from world space to object space. Therefore, rotation matrix 508A may be required to convert surface normal 504A from world space to object space.

[0068] In an embodiment, the circuit system 202 can be configured to estimate the rotation matrix 508A based on applying the least square difference operation to the surface normal and geometric normal 506A after Gaussian filtering. In order to obtain the surface normal after Gaussian filtering, the surface normal 504A can be first passed through a Gaussian filter. The Gaussian filter can remove the finer micro-geometric details of the surface normal 504A. The surface normal after Gaussian filtering can be smooth and may not include low-frequency deviations. Since the geometric normal 506A can contain fewer geometric details compared to the surface normal 504A, the geometric normal 506A may not contain low-frequency deviations. Therefore, the geometric normal 506A may be smooth and may not need to pass through a Gaussian filter. The surface normal after Gaussian filtering can be compared with the geometric normal 506A using the least square difference operation to obtain the rotation matrix 508A. It should be noted that surface normal 504A may not be directly compared to geometric normal 506A because there are multiple outliers in surface normal 504A that may not make surface normal 504A similar to geometric normal 506A. Rotation matrix 508A may transform surface normal 504A from world space to object space based on the rotation of surface normal 504A. Geometric normal 506A may then be discarded because it may only be needed for the transformation of surface normal 504A from world space to object space.

[0069] At 510, surface normal correction may be performed. The circuit system 202 may also be configured to correct the converted surface normal based on removing low-frequency deviations from the converted surface normal. As discussed, the converted surface normal may include finer microgeometric details and low-frequency deviations that may need to be removed. To remove the low-frequency deviations, the converted surface normal may be passed through a low-frequency Gaussian filter, which may filter out the low-frequency deviations from the converted surface normal to obtain a corrected surface normal. The corrected surface normal may be smooth because it may not contain low-frequency deviations.

[0070] At 512, the corrected surface normal can be converted from the object space to the tangent space. The circuit system 202 can be configured to convert the corrected surface normal from the object space to the tangent space based on the 3D mesh. The corrected surface normal can be in the object space, and therefore, the orientation of the corrected surface normal can be relative to the orientation of the object. Similarly, the orientation of the converted surface normal vector can be relative to the surface of the object and can be independent of the geometry of the object.

[0071] At 514, normal map generation may be performed. Circuitry 202 may be configured to generate a normal map 514A based on the transformed surface normal vectors. Generated normal map 514A may be associated with a surface and may be independent of the geometry of the object.

[0072] At 516, an optimization operation may be performed. The circuit system 202 may be configured to perform an optimization operation to generate a height map 516A in UV space. The optimization operation may be performed so that the tangent vector of the height map 516A in UV space remains perpendicular to the normal vector corresponding to the generated normal map 514A. It should be noted that the height map 516A may be a raster image for representing elevations (such as bumps in a 3D mesh). The generated height map 516A may appear to be a grayscale image, and may directly reflect high-frequency bumps and cavities on the geometry, and may be easily used for geometry cleaning and detail processing. The generated height map 516A and the generated normal map 514A may contain micro-geometry details that may be missing on the 3D mesh (i.e., the base mesh generated at 304 of FIG. 3). The generated height map 516A and the generated normal map 514A may be used to generate realistic renderings without subdividing the 3D mesh. However, in general, the generated normal map 514A may be preferred.

[0073] It should be noted that 3D meshes prepared by game and / or movie studios may not have an exact shape correspondence with the shape of the scanned object, which makes it difficult to directly use the normal map 514A generated from the polarization-based light cage. Instead, a generated height map 516A may be preferred.

[0074] Typically, a height map can be generated based on the refinement of a high-resolution 3D mesh and a separately measured normal map. After refinement, the height map can be calculated based on the high-resolution 3D mesh. However, the height map generated using the above operation may be too smooth and may lack details compared to the generated normal map 514A. In contrast, the circuit system 202 can generate a height map 516A in UV space based on an optimization operation. In the optimization operation, it can be assumed that the 3D mesh has a well-organized topology and the UV pixel neighborhood can be roughly equivalent to its corresponding 3D neighborhood. Moreover, the height map 516A can be optimized so that the tangent vector of the height map 516A in the UV space remains perpendicular to the normal vector. The number and direction of the tangent vectors can be optional, wherein at least two can provide the maximum feasibility space for optimizing the height map 516A. Finally, a regularization term can be used to reduce the height variation from the surface of the 3D mesh.

[0075] At 518, a gradient operation may be performed. Circuitry 202 may be configured to generate a refined normal map 518A based on applying the gradient operation to height map 516A. After generating height map 516A, refined normal map 518A may be generated. Refined normal map 518A may include details and features present in normal map 514A. Moreover, refined normal map 518A may have finer details and less noise overall than normal map 514A.

[0076] Figure 6 is a flow chart illustrating the operation of an exemplary method for generating a reflectance map for a relightable 3D model in accordance with an embodiment of the present disclosure. Figure 6 is a combination of Figure 1 , Figure 2 , Figure 3A , Figure 3B , Figure 4 and Figure 5 Reference Figure 6 , a flowchart 600 is shown. The flowchart 600 may include operations from 602 to 614 and may be Figure 1 The electronic device 102 or Figure 2 The circuit system 202 is implemented. The flowchart 600 can start from 602 and proceed to 604.

[0077] At 604, multi-view image data 112 including a set of images of the object may be acquired. The circuit system 202 may be configured to acquire the multi-view image data 112 including a set of images of the object. The object may be exposed to a set of illumination patterns for a capture duration of the multi-view image data 112. For example, Figure 3A Details related to acquiring multi-view image data 112 are provided in .

[0078] At 606, a 3D mesh of the object may be generated based on the multi-view image data 112. The circuit system 202 may be configured to generate a 3D mesh of the object based on the multi-view image data 112. For example, Figure 1 Neutralization Figure 3A Details related to the 3D mesh are provided at 304 .

[0079] At 608, a set of motion-corrected images may be obtained based on minimization of rigid motion associated with the object between images in the set of images. The circuit system 202 may be configured to obtain a set of motion-corrected images (such as Figure 3A A set of motion-corrected images 306A). For example, Figure 3A Details related to correction of a set of images are provided at 306 in FIG.

[0080] At 610, a texture map may be generated in UV space based on the set of motion-corrected images and the 3D mesh. Circuit system 202 may be configured to generate a texture map in UV space based on the set of motion-corrected images and the 3D mesh. For example, Figure 3B Further details related to the generation of the texture map are provided at 310 .

[0081] At 612, a specular reflection map and a diffuse reflection map may be obtained based on separating the specular reflection component and the diffuse reflection component from the texture map. The circuit system 202 may be configured to obtain the specular reflection map and the diffuse reflection map based on separating the specular reflection component and the diffuse reflection component from the texture map. The specular reflection component and the diffuse reflection component may be separated from the cross-polarized UV texture map and the parallel-polarized UV texture map. For example, in Figure 3B Details related to the generation of specular reflection maps and diffuse reflection maps are provided at 312 in FIG.

[0082] At 614, a relightable 3D model of the object may be obtained based on the specular map and the diffuse map. The circuit system 202 may be configured to obtain a relightable 3D model of the object based on the specular map and the diffuse map. Control may pass to end.

[0083] Although flowchart 800 is shown as discrete operations, such as 604, 606, 608, 610, 612, and 614, the present disclosure is not limited thereto. Thus, in certain embodiments, such discrete operations may be further divided into additional operations, combined into fewer operations, or eliminated, depending on the implementation, without compromising the essence of the disclosed embodiments.

[0084] Various embodiments of the present disclosure may provide non-transitory computer-readable media and / or storage media on which are stored information executable by a machine and / or computer to operate an electronic device (e.g., Figure 1 Such instructions may cause the electronic device 102 to perform operations that may include acquiring multi-view image data including a set of images of an object (e.g., Figure 3A The operation may include generating a 3D mesh of the object based on the multi-view image data (e.g., Figure 3A The operations may also include obtaining a set of motion-corrected images based on minimizing rigid motion associated with the object between images in the set of images (e.g., Figure 3A The operation may also include generating a texture map in UV space based on the set of motion-corrected images and the 3D mesh. The operation may also include obtaining a specular reflection map and a diffuse reflection map based on separating the specular reflection component and the diffuse reflection component from the texture map (e.g., Figure 3B The operation may also include obtaining a relightable 3D model of the object based on the specular reflection map and the diffuse reflection map (e.g., Figure 3B The relightable 3D model 314A).

[0085] Exemplary aspects of the present disclosure may provide an electronic device (such as a Figure 1 The circuit system 202 may be configured to acquire multi-view image data including a set of images of an object (eg, Figure 3A In this context, the object may be exposed to a set of illumination patterns during the capture duration of the multi-view image data. The circuit system 202 may be configured to generate a 3D mesh of the object based on the multi-view image data (e.g., Figure 3A The circuit system 202 may be configured to obtain a set of motion-corrected images (e.g., Figure 3A The circuit system 202 may be configured to generate a texture map in UV space based on the set of motion-corrected images and the 3D mesh. The circuit system 202 may be configured to obtain a specular reflection map and a diffuse reflection map based on separating the specular reflection component and the diffuse reflection component from the texture map (e.g., Figure 3B The circuit system 202 may be configured to obtain a relightable 3D model of the object based on the specular reflection map and the diffuse reflection map (eg, Figure 3B A relightable 3D model 314A).

[0086] In an embodiment, a set of illumination patterns may include one or more of a cross-polarized omnidirectional illumination pattern and a gradient illumination pattern and a polarized illumination pattern, the polarized illumination pattern including a cross-polarized illumination pattern and a parallel polarized illumination pattern.

[0087] In an embodiment, the circuit system 202 may be configured to acquire a color checkerboard image exposed to a cross-polarized omnidirectional illumination pattern. The circuit system 202 may be configured to estimate a color matrix based on a comparison of color values ​​of the color checkerboard image with reference color values. The circuit system 202 may be configured to apply the estimated color matrix to a subset of images in the set of motion-corrected images to obtain a set of color-corrected images. In this context, the subset of images may be associated with one of an omnidirectional cross-polarized illumination pattern or a parallel polarized illumination pattern.

[0088] In an embodiment, the set of color-corrected images may also be obtained based on converting each image in the subset into a set of demosaiced images.

[0089] In an embodiment, the circuit system 202 may be configured to estimate ambient occlusion based on the 3D shape of the 3D mesh. The circuit system 202 may be configured to refine the specular map and the diffuse map based on removing ambient occlusion from the specular map and the diffuse map. In this context, a relightable 3D mesh of the object may be obtained based on the refined specular map and the diffuse map.

[0090] In an embodiment, circuitry 202 may be configured to generate a color map based on a linear combination of a specular reflectance map and a diffuse reflectance map.

[0091] In an embodiment, circuitry 202 may be configured to obtain a set of specular separated gradient images based on removing a diffuse component from each image associated with the gradient illumination pattern and included in the set of motion corrected images.

[0092] In an embodiment, the circuit system 202 may be configured to calculate a surface normal corresponding to each pixel of an image in the set of specularly separated gradient images. The circuit system 202 may be configured to calculate a geometric normal by linearly combining a set of vertex normals corresponding to a 3D mesh, wherein the combination is performed based on the barycentric coordinates of each pixel of a texture map generated in UV space.

[0093] In an embodiment, the circuit system 202 may be configured to transform the surface normal from world space to object space by applying a rotation matrix to the surface normal. The circuit system 202 may be configured to correct the transformed surface normal based on removing low-frequency deviations from the transformed surface normal. The circuit system 202 may be configured to transform the corrected surface normal vector from object space to tangent space based on the 3D mesh. The circuit system 202 may be configured to generate a normal map based on the transformed surface normal vector.

[0094] In an embodiment, the circuitry 202 may be configured to estimate the rotation matrix based on applying a least squares difference operation on the Gaussian filtered surface normal and the geometric normal.

[0095] In an embodiment, the circuit system 202 may be configured to perform an optimization operation to generate a height map in UV space. Here, the optimization operation may be performed so that the tangent vector of the height map in UV space is perpendicular to the normal vector corresponding to the generated normal map.

[0096] In an embodiment, the circuitry 202 may be configured to generate a refined normal map based on applying a gradient operation to the height map.

[0097] The present disclosure may also be located in a computer program product comprising all the features enabling the implementation of the methods described herein and capable of carrying out these methods when loaded in a computer system. In this context, a computer program refers to any expression in any language, code or notation of a set of instructions intended to cause a system with information processing capabilities to perform certain functions, either directly or after either or both of the following: a) conversion into another language, code or notation; b) reproduction in a different material form.

[0098] Although the present disclosure is described with reference to certain embodiments, it will be appreciated by those skilled in the art that various changes may be made and equivalents may be substituted without departing from the scope of the present disclosure. In addition, many modifications may be made to adapt specific circumstances or materials to the teachings of the present disclosure without departing from its scope. Therefore, it is intended that the present disclosure is not limited to the disclosed embodiments, but that the present disclosure will include all embodiments falling within the scope of the appended claims.

Claims

1. An electronic device, comprising: A circuit system is configured to: acquiring multi-view image data comprising a set of images of an object, wherein the object is exposed to a set of illumination patterns for a capture duration of the multi-view image data; generating a 3D mesh of the object based on the multi-view image data; obtaining a set of motion-corrected images based on minimization of rigid motion associated with the object between images in the set of images; generating a texture map in UV space based on the set of motion-corrected images and the 3D mesh; A specular reflection map and a diffuse reflection map are obtained based on separating the specular reflection component and the diffuse reflection component from the texture map; as well as Obtain a relightable 3D model of the object based on the specular and diffuse maps.

2. The electronic device of claim 1, wherein the object is a human head and the multi-view image data is acquired from an imaging setup operating as a polarization-based light cage. 3 . The electronic device according to claim 1 , wherein the set of illumination patterns comprises polarized illumination patterns and one or more of a cross-polarized omnidirectional illumination pattern and a gradient illumination pattern, the polarized illumination patterns comprising a cross-polarized illumination pattern and a parallel polarized illumination pattern.

4. The electronic device of claim 1 , wherein the circuit system is further configured to: acquiring a color checkerboard image exposed to a cross-polarized omnidirectional illumination pattern; estimating a color matrix based on a comparison of color values ​​of the color checkerboard image with reference color values; and applying the estimated color matrix to a subset of images in the set of motion-corrected images to obtain a set of color-corrected images, Wherein a subset of the images is associated with one of an omnidirectional cross-polarized illumination pattern or a parallel polarized illumination pattern. 5 . The electronic device of claim 4 , wherein the set of color-corrected images is further obtained based on converting each image in the subset into a set of demosaiced images.

6. The electronic device of claim 1 , wherein the circuit system is further configured to: Estimating ambient occlusion based on the 3D shape of the 3D mesh; and Refine the specular and diffuse maps based on removing ambient occlusion from them, A relightable 3D mesh of the object is obtained based on the refined specular and diffuse maps.

7. The electronic device of claim 1, wherein the circuit system is further configured to generate a color map based on a linear combination of the specular reflectance map and the diffuse reflectance map.

8. The electronic device of claim 1 , wherein the circuit system is further configured to obtain a set of specular-separated gradient images based on removing a diffuse component from each image associated with the gradient illumination pattern and included in the set of motion-corrected images.

9. The electronic device of claim 8, wherein the circuit system is further configured to: calculating a surface normal corresponding to each pixel of an image in the set of specularly separated gradient images; and The geometric normals are calculated by linearly combining a set of vertex normals corresponding to the 3D mesh, where the combination is performed based on the barycentric coordinates of each pixel of the texture map generated in UV space.

10. The electronic device of claim 9, wherein the circuit system is further configured to: Transform the surface normal from world space to object space by applying a rotation matrix to the surface normal; correcting the transformed surface normal based on removing low frequency deviations from the transformed surface normal; Transforming the corrected surface normal vector from object space to tangent space based on the 3D mesh; as well as Generate a normal map based on the transformed surface normal vectors.

11. The electronic device of claim 10, wherein the circuit system is further configured to estimate the rotation matrix based on applying a least squares difference operation to the Gaussian filtered surface normal and the geometric normal.

12. The electronic device of claim 10, wherein the circuit system is further configured to perform an optimization operation to generate a height map in UV space, and The optimization operation is performed so that the tangent vector of the height map in UV space is perpendicular to the normal vector corresponding to the generated normal map.

13. The electronic device of claim 12, wherein the circuit system is further configured to generate the refined normal map based on applying a gradient operation to the height map.

14. A method comprising: In electronic devices: acquiring multi-view image data comprising a set of images of an object, wherein the object is exposed to a set of illumination patterns for a capture duration of the multi-view image data; generating a 3D mesh of the object based on the multi-view image data; obtaining a set of motion-corrected images based on minimization of rigid motion associated with the object between images in the set of images; generating a texture map in UV space based on the set of motion-corrected images and the 3D mesh; A specular reflection map and a diffuse reflection map are obtained based on separating the specular reflection component and the diffuse reflection component from the texture map; as well as Obtain a relightable 3D model of the object based on the specular and diffuse maps.

15. The method of claim 14, wherein the object is a human head and the multi-view image data is acquired from an imaging setup operating as a polarization-based light cage.

16. The method of claim 14, wherein the set of illumination patterns comprises polarized illumination patterns and one or more of a cross-polarized omnidirectional illumination pattern and a gradient illumination pattern, the polarized illumination patterns comprising a cross-polarized illumination pattern and a parallel polarized illumination pattern.

17. The method according to claim 14, further comprising: acquiring a color checkerboard image exposed to a cross-polarized omnidirectional illumination pattern; estimating a color matrix based on a comparison of color values ​​of the color checkerboard image with reference color values; as well as applying the estimated color matrix to a subset of images in the set of motion-corrected images to obtain a set of color-corrected images, Wherein a subset of the images is associated with one of an omnidirectional cross-polarized illumination pattern or a parallel polarized illumination pattern.

18. The method of claim 17, wherein the set of color-corrected images is further obtained based on converting each image in the subset into a set of demosaiced images.

19. The electronic device of claim 14, wherein the circuit system is further configured to: Estimating ambient occlusion based on the 3D shape of the 3D mesh; and Refine the specular and diffuse maps based on removing ambient occlusion from them, A relightable 3D mesh of the object is obtained based on the refined specular and diffuse maps.

20. A non-transitory computer-readable medium having computer-executable instructions stored thereon, the computer-executable instructions, when executed by an electronic device, causing the electronic device to perform operations comprising: acquiring multi-view image data comprising a set of images of an object, wherein the object is exposed to a set of illumination patterns for a capture duration of the multi-view image data; generating a 3D mesh of the object based on the multi-view image data; obtaining a set of motion-corrected images based on minimization of rigid motion associated with the object between images in the set of images; generating a texture map in UV space based on the set of motion-corrected images and the 3D mesh; A specular reflection map and a diffuse reflection map are obtained based on separating the specular reflection component and the diffuse reflection component from the texture map; as well as Obtain a relightable 3D model of the object based on the specular and diffuse maps.

Images