Texture acquisition system, texture acquisition device, and texture acquisition program

Abstract

To acquire detailed information of a three-dimensional model necessary for photorealistic CG rendering capable of expressing the texture of the surface of a subject in real time.SOLUTION: A texture acquisition device 1 of a texture acquisition system 1000 comprises: a subject shape acquisition unit 120 that acquires shape information of a subject from a depth measurement device 20; a reflection component separation unit 141 that separates reflected light from the subject irradiated with polarized light of multiple wavelengths into a diffuse reflection component and a specular reflection component; a diffuse reflection calculator 142 that calculates a diffuse reflection coefficient and normal information using the diffuse reflection component; a specular reflection calculation unit that calculates the roughness and reflectance of the surface of the object using a predetermined specular reflection model, and calculates a refractive index using the Fresnel equation; and a three-dimensional model detailed information output unit 150 that outputs the calculated information as three-dimensional model detailed information.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a texture acquisition system, a texture acquisition device, and a texture acquisition program for acquiring shape, texture, and the like as subject information, which is material data handled in computer graphics (CG). [Background technology]

[0002] With the recent spread of AR (Augmented Reality) / VR (Virtual Reality) technology, there is an increasing need to convert real-world subjects into subject information (hereinafter sometimes referred to as "3D models") that can be handled using CG technology, etc. This is a technology generally called volumetric capture, which captures the surface pattern and shape of the entire subject using cameras and depth measurement devices arranged to surround the subject (see Non-Patent Documents 1 and 2).

[0003] In conventional volumetric capture, especially when it is necessary to change the lighting conditions after shooting, the subject is placed in a flat lighting environment and photographed, so that the subject surface is illuminated uniformly and photographed without shadows. Since the reflectance of the subject cannot be measured directly, the lighting is artificially made uniform in the amount of light incident on the subject surface and photographed under a lighting effect that has no directionality, so that the amount of light incident on each part of the subject is normalized and the brightness on the obtained photographed image can be treated as a diffuse reflection coefficient in a pseudo manner. This allows the appearance of a 3D model to be expressed as expected under the lighting conditions in the virtual space by adding pseudo shading during rendering. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] O. Schreer, et al., “Capture and 3D Video Processing of Volumetric Video,” 2019 IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan, 2019, pp. 4310-4314. [Non-Patent Document 2] O. Schreer, et al., “ADVANCED VOLUMETRIC CAPTURE AND PROCESSING,” [online], IBC2018, [Retrieved July 1, 2021], Internet<URL:https: / / www.ibc.org / download?ac=6559> Summary of the Invention [Problem to be solved by the invention]

[0005] However, in conventional volumetric capture techniques, the diffuse reflection component of reflected light is calculated artificially. Also, the specular reflection component is not handled, or the specular reflection coefficient is set manually either uniformly for the entire subject or for each part of the subject. When manually assigning coefficients (specular reflection coefficients), if the subject is non-rigid and changes over time, assigning coefficients for each part of the subject is extremely laborious and difficult in practice, and it is even more difficult to apply this to live video in particular.

[0006] Furthermore, when observing the surface of a subject microscopically, many subject surfaces have small irregularities, which causes the appearance to vary greatly when illuminated. The irregularities of a subject's surface are an element that greatly affects the appearance, such as gloss and feel, that is, the impression of texture. Therefore, in order to express this texture, it is necessary to describe the behavior of specular reflection for each part of the subject or in units smaller than the pixel unit of the captured pixel, but this cannot be expressed at present.

[0007] In AR applications, when the acquired subject information is used as content, it is rendered using CG technology based on the subject information and superimposed on the real space using AR glasses, etc. In that case, if the lighting conditions in the real space are not matched, the image will be unnatural. Furthermore, while the lighting in the real space has complex spectral characteristics, the surface pattern of a typical CG model is expressed using RGB three-color spectrum, so it is difficult to match them, and the composite image may be unnatural.

[0008] These factors make it difficult to match the environments of multiple subjects when multiple subject information is placed and rendered in one scene simultaneously in AR / VR. As a result, not only does it cause discomfort due to the mismatch between the real and virtual environments as in AR, but it also causes discomfort between subjects in the virtual space, which negatively impacts the sense of immersion and realism.

[0009] The present invention has been made in consideration of the above-mentioned points, and has an objective of providing a texture acquisition system, texture acquisition device, and texture acquisition program that can acquire detailed information of a three-dimensional model in real time, which is required for photorealistic CG rendering that can express the texture of the subject's surface. [Means for solving the problem]

[0010] In order to solve the above problems, the texture acquisition system of the present invention is configured to include a photography booth having a depth measurement device, a polarized lighting device, and a polarized photography device arranged to surround a subject, and a texture acquisition device that is communicatively connected to the depth measurement device, the polarized lighting device, and the polarized photography device, and acquires the texture represented by the uneven state of the subject's surface. The texture acquisition device is also configured to include a memory unit, a subject shape acquisition unit, a measurement control unit, a reflection component separation unit, a diffuse reflection calculation unit, a specular reflection calculation unit, and a 3D model detailed information output unit.

[0011] With this configuration, the texture acquisition device acquires shape information of the subject by receiving depth information measured by the depth measurement device through the subject shape acquisition unit, and the measurement control unit controls the polarized lighting device to irradiate polarized light of multiple wavelengths onto the subject, and the reflected component separation unit acquires the reflected light from the subject irradiated with each of the polarized light of multiple wavelengths as information captured by the polarized imaging device, and separates the reflected light into a diffuse reflection component and a specular reflection component. This allows the texture acquisition device of the texture acquisition system to separate the reflected light of each of the polarized light beams of multiple wavelengths that is irradiated onto the subject into a diffuse reflection component and a specular reflection component.

[0012] In addition, the texture acquisition device can calculate the diffuse reflection coefficient and normal information of the subject by applying the diffuse reflection components separated by the reflection component separation unit and the lighting position and lighting intensity of the polarized lighting devices to Lambert's cosine law for the reflected light captured by the polarized shooting device when the polarized lighting devices arranged in three directions each irradiate the subject with polarized light of multiple wavelengths, using the diffuse reflection calculation unit. As a result, the texture acquisition device of the texture acquisition system can calculate the diffuse reflection coefficient and normal information of the subject using only the diffuse reflection components excluding the specular reflection components, thereby enabling more accurate acquisition of the diffuse reflection coefficient and normal information compared to a case where the reflected light is not separated.

[0013] In addition, the texture acquisition device applies the specular reflection components separated by the reflection component separation unit for each of the multiple wavelengths of polarized light, the illumination position of the polarized lighting device, the camera position of the polarized imaging device, and the calculated normal information to a predetermined specular reflection model, calculates the angle at which the normal distribution function obtained by the specular reflection model is half of the maximum value as a parameter of the roughness of the subject's surface, and applies the illumination position of the polarized lighting device, the camera position of the polarized imaging device, the normal information, and the normal distribution function to the predetermined specular reflection model to calculate the reflectance of the subject for each of the p-waves and s-waves of polarized light of the multiple wavelengths, and applies the calculated reflectance to the Fresnel equation for reflection in media with different refractive indices to calculate the refractive index of the subject. This allows the texture acquisition device of the texture acquisition system to calculate roughness as a parameter indicating the detailed normal distribution condition of the subject's surface, and to calculate the subject's reflectance and refractive index for each wavelength of p-waves and s-waves.

[0014] In addition, the texture acquisition device can output information including any of the calculated diffuse reflection coefficient, normal information, subject surface roughness, reflectance, and refractive index by the 3D model detail information output unit as 3D model detail information in accordance with the time when the polarization imaging device captured the image to calculate the information. This allows the texture acquisition device of the texture acquisition system to output more detailed information of the three-dimensional model to, for example, an external device in real time. Effect of the Invention

[0015] According to the present invention, it is possible to provide a texture acquisition system, a texture acquisition device, and a texture acquisition program that acquire detailed information of a three-dimensional model in real time, which is required for photorealistic CG rendering that can express the texture of the subject's surface. [Brief description of the drawings]

[0016] [Figure 1]1 is a functional block diagram showing an overall configuration of a texture acquisition system including a texture acquisition device according to an embodiment of the present invention. [Diagram 2] 4 is an explanatory diagram relating to a texture acquisition process performed by the texture acquisition device of the present embodiment. FIG. [Diagram 3] FIG. 1 is an explanatory diagram showing an example of a photography booth according to the present embodiment configured as a regular octahedron. [Figure 4] 5A and 5B are diagrams illustrating an example of an illumination pattern in the polarized illumination device according to the present embodiment. [Diagram 5] 2 is an explanatory diagram showing an RGB Bayer arrangement on a camera imaging plate of the polarization imaging device according to the present embodiment. FIG. [Figure 6] 4 is an explanatory diagram relating to separation of a diffuse reflection component and a specular reflection component of reflected light in the present embodiment. FIG. [Figure 7] FIG. 2 is an explanatory diagram of Lambert's cosine law in the present embodiment. [Figure 8] 4A to 4C are explanatory diagrams relating to calculation of a diffuse reflection coefficient and normal information based on Lambert's cosine law in this embodiment. [Figure 9] 4A to 4C are explanatory diagrams relating to calculation of a diffuse reflection coefficient and normal information based on Lambert's cosine law in this embodiment. [Figure 10] FIG. 4 is an explanatory diagram of a reflection model regarding specular reflection in the present embodiment. [Figure 11] 3 is an explanatory diagram regarding reflection and refraction on a surface of a subject in the present embodiment. FIG. [Figure 12] FIG. 1 is a diagram showing an example of a photography booth according to the present embodiment configured with multispectral lighting and a multispectral camera. [Figure 13] FIG. 1 is a diagram showing an example of a photography booth according to the present embodiment configured with multispectral lighting and a multispectral camera. [Figure 14] 13 is an explanatory diagram of a modified example of the three-dimensional model detailed information output unit according to the embodiment. FIG. [Figure 15] 4 is a flowchart showing a processing flow of the texture acquisition device according to the present embodiment. [Figure 16]11A to 11C are diagrams showing the calculation results of the diffuse reflection coefficient and normal information using only the diffuse reflection component in this embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Hereinafter, modes for carrying out the present invention (hereinafter, referred to as "embodiments") will be described with reference to the drawings. First, with reference to FIG. 1, an overview of a texture acquisition system 1000 including a texture acquisition device 1 will be described first, and then each component will be described.

[0018] 1, the texture acquisition system 1000 includes a depth measurement device 20, a polarized lighting device 30, and a polarized imaging device 40, which are arranged in an imaging booth BH for imaging a subject, and a texture acquisition device 1 that is communicatively connected to the depth measurement device 20, the polarized lighting device 30, and the polarized imaging device 40. The texture acquisition device 1 is communicatively connected to one or more external devices 5 via a network or the like.

[0019] The texture acquisition device 1 of this embodiment is useful as a method for measuring subject information (three-dimensional model) used in AR / VR and for capturing subject information in three-dimensional televisions, and realizes the following processes for acquiring the texture of a subject (hereinafter referred to as the "texture acquisition process"): (1) separation process of diffuse reflection components and specular reflection components, (2) calculation process of diffuse reflection coefficient and normal information using the diffuse reflection component, and (3) calculation process of the "roughness" of the subject surface and the specular reflection coefficient (calculated as the "refractive index of the subject" in this embodiment) using the specular reflection component (hereinafter referred to as the "roughness and refractive index calculation process").

[0020] (1) In the process of separating the diffuse reflection component and the specular reflection component, the texture acquisition device 1, as shown in FIG. 2, irradiates the subject S with polarized light (illumination light) from the polarized illumination device 30 (for example, red (R) illumination, green (G) illumination, and blue (B) illumination). Then, the irradiated light reflected from the subject S is photographed by a polarized imaging device 40 that selectively transmits the light in the polarization directions of, for example, 0 degrees, 45 degrees, 90 degrees, and 135 degrees (see FIGS. 5 and 6 described later). The photographed image P contains the specular reflection component I, which appears as a highlight as if the light source is reflected. s The diffuse reflection component I is the incident light that penetrates the subject and absorbs the color components of the object and diffuses. d The texture capture device 1 uses the difference in the reflection behavior of polarized light on the surface of the subject to capture the diffuse reflection component I d and the specular reflection component I s (Details below.)

[0021] (2) In the process of calculating the diffuse reflection coefficient and normal information, the texture acquisition device 1 calculates the separated diffuse reflection component I d Using this, the normal information N (normal vector N) of the object surface for each wavelength and the diffuse reflection coefficient K d In this embodiment, for each wavelength (e.g., red, green, and blue), the reflected light from three directions is observed using a photometric stereo technique, and the surface normal vector and the diffuse reflection coefficient K d Calculate (details below).

[0022] (3) In the roughness and refractive index calculation process, the texture acquisition device 1 calculates the separated specular reflection component I s Using this, the roughness of the object surface γ and the refractive index of the object n j The texture acquisition device 1 calculates the specular reflection coefficient K s It is possible to obtain the specular reflection coefficient K s Since the refractive index n of the subject for each wavelength band is used as information indicating the reflection tendency, the refractive index n of the subject for each wavelength band is used as information indicating the reflection tendency. jIn addition, it is possible to obtain a coefficient that takes into account Fresnel reflection, wavelength dependency, and polarization by calculating it as Ro (reflection coefficient of light incident parallel to the normal) of the Fresnel coefficient approximation formula proposed by Schlick et al., which is shown in Reference 3 described below.

[0023] In this way, the texture capture device 1 can obtain the accurate diffuse reflectance coefficient K d The normal information, the roughness γ, which is a parameter that indicates the unevenness of the object surface, and the refractive index n j The texture acquisition device 1 then outputs detailed information of the 3D model thus acquired ("detailed 3D model information" to be described later) to an external device 5 such as a CG rendering device (see FIG. 1) according to the performance (specifications) of each device. The texture acquisition system 1000 including the texture acquisition device 1 according to this embodiment will be described in detail below.

[0024] <Configuration of the texture acquisition device> The texture acquisition device 1 according to this embodiment is configured by a general computer, and includes a control unit 10, an input / output unit 11, and a storage unit 12, as shown in FIG. The input / output unit 11 is made up of a communication interface for transmitting and receiving information, and an input / output interface for transmitting and receiving information to and from an input device such as a keyboard and an output device such as a monitor.

[0025] The storage unit 12 is configured with a flash memory, a hard disk, a RAM (Random Access Memory), etc. In this storage unit 12, initial setting information 200 (details will be described later) for the texture acquisition process by the texture acquisition device 1 is stored. The storage unit 12 also temporarily stores a program for executing each function of the control unit 10 (a texture acquisition program) and information necessary for the processing of the control unit 10.

[0026] The control unit 10 is responsible for overall control of the texture acquisition device 1, and is composed of an initial setting information acquisition unit 110, a subject shape acquisition unit 120, a measurement control unit 130, a texture calculation unit 140, and a 3D model detailed information output unit 150, as shown in FIG. The control unit 10 is realized, for example, by a CPU (Central Processing Unit) (not shown) expanding a program (texture acquisition program) stored in the storage unit 12 into a RAM and executing the program.

[0027] When the texture acquisition device 1 executes the texture acquisition process, the initial setting information acquisition unit 110 acquires information including the lighting position, light distribution characteristics, and camera posture of each device installed in the photography booth BH in advance as initial setting information 200, and stores the information in the storage unit 12. An example of the initial setting information acquisition unit 110 acquiring the initial setting information 200 will be described with reference to FIG. Fig. 3 shows an example of a photography booth BH configured as a regular octahedron. The depth measurement device 20, polarized illumination device 30, and polarized photography device 40 are arranged to surround the subject. As shown in Fig. 3, each polarized illumination device 30 is arranged as a point light source at the vertices of the regular octahedron. The depth measurement device 20 and polarized photography device 40 are arranged at the center of the surface, with their gaze directed toward the center of the regular octahedron.

[0028] The measurement control unit 130 sets multiple combinations of wavelength bands and wavelength patterns emitted by each light source (polarized lighting device 30) that do not cause crosstalk. There are at least three combinations, but more are possible (see Figs. 12 and 13 described below). Here, by using a multispectral polarized light source and a multispectral polarized camera, it is possible to set three or more axes, but for simplicity of explanation, in this embodiment, three wavelengths of R, G, and B are used in principle, and explanation is given on three XYZ axes with the R, G, and B light sources installed in the plus and minus directions of each of the XYZ axes.

[0029] As shown in FIG. 3, in each light source (polarization illumination device 30), illumination of the same wavelength band is irradiated from above and below (for example, G (green)), front and back (for example, R (red)), and left and right (for example, B (blue)) of the subject. That is, polarized light of a plurality of wavelengths is irradiated from the polarization illumination device 30. In the present embodiment, the shape of the subject is known by the depth measurement device 20, and the illumination position is also known by prior initial setting. Therefore, it is possible to acquire information on the front and back of the subject surface with respect to the installed illumination, and it is possible to irradiate illumination light of the same wavelength band or wavelength pattern simultaneously from the plus and minus directions. This is because it is possible to determine which illumination on the plus side or minus side in the XYZ axis is irradiated to that part. Furthermore, the positional relationship between each light source (polarization illumination device 30) and each adjacent camera is all targeted at the center of the shooting booth installed inside the regular octahedron. This is to make it possible to ignore the consideration of Fresnel reflection when obtaining the "roughness" of the subject surface, and thereby to derive the "roughness" after reducing the unknowns (details will be described later). The regular octahedron is an example and is not limited thereto.

[0030] Although each light source can be configured as a surface light source (such as a flat panel display), by using a point light source as in the present embodiment, even when the installation target range of the shooting booth BH is wide, it is not necessary to install a large-scale flat panel display and the construction becomes easy. Also, a point light source that does not surround the subject with a display is more capable of easily recording the voice of the subject than when the shooting booth BH is surrounded by a surface light source (flat panel display).

[0031] The initial setting information acquisition unit 110 acquires, as initial setting information 200, information on the illumination position (X, Y, Z coordinates) where the polarization illumination device 30, which is each light source, is arranged, the light distribution characteristics obtained by measuring the luminous intensity in each direction in space, etc., the camera position (X, Y, Z coordinates) and camera posture (roll, pitch, yaw) where the depth measurement device 20 and the polarization imaging device 40 are arranged, from a system management device or the like, and stores it in the storage unit 12. Note that the combination of the depth measurement device 20 and the polarization imaging device 40 arranged at one location in the photographing booth BH shall have the same camera position and the same camera posture.

[0032] Returning to FIG. 1, the description of the configuration will be continued. The subject shape acquisition unit 120 transmits instruction information for causing each depth measurement device 20 arranged in the photographing booth BH to execute measurement via the measurement control unit 130. Then, the subject shape acquisition unit 120 acquires the (rough) subject shape by receiving the measurement information from each depth measurement device 20. Note that, for example, the depth measurement device 20 can use a ToF (Time of Flight) camera that emits near-infrared rays from the camera and calculates the depth by measuring the time until the rays are reflected back from the object. Alternatively, without installing the depth measurement device 20 shown in this embodiment, the shape of the entire subject may be acquired by using a stereo matching shape measurement method for a camera image (for example, the image of the polarization imaging device 40) installed so as to surround the subject.

[0033] The measurement control unit 130 controls each device by transmitting instruction information in cooperation with the subject shape acquisition unit 120 and the texture calculation unit 140 to the depth measurement device 20, the polarization illumination device 30, and the polarization imaging device 40 arranged in the photographing booth BH. Specifically, the measurement control unit 130 causes the depth measurement device 20 to execute measurement of the subject shape by transmitting instruction information thereto.

[0034] Furthermore, when the texture calculation unit 140 executes the texture acquisition process, the measurement control unit 130 transmits instruction information to the polarized lighting device 30 to control the device to have a predetermined lighting state pattern, for example, as shown in Fig. 4. Fig. 4 shows an example in which the polarization imaging device 40 that performs measurement uses RGB spectrum, one sequence is composed of six frames (Figs. 4(a) to 4(f)), and the pattern is one in which the wavelength band and polarization direction of the lighting by the polarized lighting device 30 are changed. Here, each lighting is turned on so that the wavelength band and polarization direction of adjacent lighting are not the same, and the colors are changed from R to G to B over time, and the polarization direction is also changed between 0 and 90 degrees. For example, the polarized illumination device 30 located at the top of the regular octahedron changes the wavelength band from G to B to R to G to B to R, and the polarization direction also changes from 90 degrees to 0 degrees to 90 degrees to 0 degrees to 90 degrees to 0 degrees, as shown in Figures 4(a) to 4(f).

[0035] The reason for changing the polarization direction here is that depending on the angle of incidence of the p-wave (p-polarized) illumination light on the surface of the object, it may become the Brewster angle, causing the reflected light to disappear. However, this Brewster angle is a rare case that occurs only when the surface is not rough and is in a mirror state. Therefore, if there is no problem with the application, the change in the polarization direction over time can be omitted, and one sequence can be composed of three frames. In this case, for example, as shown in Figure 4(a) → Figure 4(c) → Figure 4(e), the polarization direction is not changed, and only the wavelength band of the illumination is changed in three patterns of RGB. When taking into consideration the loss of reflected light at Brewster's angle, this may be solved by irradiating illumination light whose polarization direction is successively changed, and then interpolating using information before and after the loss, but this is not limited to this.

[0036] In addition, the measurement control unit 130 transmits instruction information to the polarized photographing device 40 to photograph the reflected light in response to the texture calculation unit 140 executing the texture acquisition process and the polarized lighting device 30 irradiating the subject with illumination light.

[0037] The camera imaging board of the polarization imaging device 40 (hereinafter referred to as the "polarization RGB camera") has an RGB Bayer array, as shown in FIG. 5 for example. Further, each RGB pixel is composed of four divided pixels, each of which is a pixel that selectively transmits components with polarization directions of 0 degrees, 45 degrees, 90 degrees, and 135 degrees. In FIG. 5, the direction of the hatching indicated by the slashes shows the direction of the polarized light component that is transmitted.

[0038] Returning to FIG. 1, the description of the configuration will be continued. As a texture acquisition process, the texture calculation unit 140 executes the above-described (1) separation process of the diffuse reflection component and the specular reflection component, (2) calculation process of the diffuse reflection coefficient and the normal information using the diffuse reflection component, and (3) calculation process of the "roughness" of the subject surface and the refractive index of the subject using the specular reflection component. The texture calculation unit 140 includes a reflection component separation unit 141, a diffuse reflection calculation unit 142, and a specular reflection calculation unit 143.

[0039] The reflection component separation unit 141 separates the diffuse reflection component and the specular reflection component in the reflected light. Here, FIG. 6 shows an example in which the polarization illumination device 30 is used as a point light source and the subject is irradiated with RGB three-color illumination under the control of the measurement control unit 130. At that time, the polarization imaging device 40 is used as the polarization RGB camera 40a, and the subject is photographed by this polarization RGB camera 40a. The camera imaging board of this polarization RGB camera 40a has an RGB Bayer array as indicated by reference numeral 101 in FIG. 6. Further, each RGB pixel is composed of four divided pixels, each of which is a pixel that selectively transmits components with polarization directions of 0 degrees, 45 degrees, 90 degrees, and 135 degrees. Note that the four divided pixels and the polarization directions of 0 degrees, 45 degrees, 90 degrees, and 135 degrees are examples and are not limited thereto.

[0040] The observed luminance I is represented by the following equation (1) based on the dichroic reflection model. The dichroic reflection model is that the spectrum of the reflected light can be represented by the linear sum of the spectra of the diffuse reflection component and the specular reflection component.

[0041] I = I d + a(1 + cos(2Θ - β)) ··· Equation (1) Here, I d is the diffuse reflection component, a(1+cos(2Θ-β)) is the specular reflection component I s In addition, a represents the amplitude, Θ represents the polarization direction of the pixel, and β represents the phase.

[0042] The brightness values ​​obtained at the pixels with polarization directions of 0 degrees, 45 degrees, 90 degrees, and 135 degrees (respectively, I 0 ,I 45 ,I 90 ,I 135 ) is the average value I ave The wave behaves as a sine wave when the angle is double the polarization angle. Therefore, the trigonometric formula sin(θ-β) 2 +cos(θ-β) 2 For example, I 0 2 +I 45 2 = a (a is the amplitude of the trigonometric function). This allows us to calculate the amplitude a. Also, the minimum value I of the observed brightness I min is the diffuse reflection component I d and 2a, which is twice the amplitude a, is the specular reflection component I s It becomes. In this way, the reflected component separation unit 141 separates the diffuse reflected component I d and the specular reflection component I s and can be separated.

[0043] The reflection component separation unit 141 separates the diffuse reflection component and the specular reflection component from, for example, one sequence (six frames) (see FIG. 4). At this time, the images in one sequence are taken at different times. If the subject is a non-rigid body and there is movement in one sequence, motion compensation is performed on the previous and next frames with respect to the state of the frame at the reference time, and processing is performed on the same part of the subject even if the times are different. In addition, the method of separating the diffuse reflection component and the specular reflection component by the reflection component separation unit 141 is not limited to the above-described separation that utilizes the difference in the behavior of the reflection of polarized light on the subject surface, and other methods (for example, calculating and removing the specular reflection component by deep learning) may be used.

[0044] Returning to FIG. 1, the description of the configuration will be continued. The diffuse reflection calculation unit 142 calculates the diffuse reflection component I d Using the diffuse reflection coefficient K d and normal information. Here, the diffuse reflection calculation unit 142 uses Lambert's cosine law, which states that the intensity of reflected light is proportional to the cosine of the angle of incidence, and the photometric stereo technique. This photometric stereo technique obtains images illuminated with the same wavelength band from three directions. Therefore, the reflected light (diffuse reflection component I d ) for each wavelength of R (red), G (green), and B (blue) in photometric stereo to obtain normal information (surface normal vector) and diffuse reflection coefficient K d Calculate.

[0045] Here, the unevenness of the subject surface is taken as the surface normal vector of the subject surface in units of captured pixels, but in this embodiment, the surface normal vector is taken as the average normal in the area of ​​the subject surface at the pixel unit level. However, since the surface of an actual subject has surfaces in finer units, the specular reflection calculation unit 143 calculates the normal distribution of these finer surfaces as the parameter "γ" of the normal distribution function (parameter of the roughness of the subject surface) (details will be described later).

[0046] On the other hand, Lambert's cosine law is as shown in Figure 7. d Then, it is expressed by equation (2). The observed light intensity I here is the specular reflection component I s Diffuse reflection component I with d In addition, the diffuse reflection coefficient K d and the normal vector N is an unknown.

[0047] Then, the diffuse reflection calculation unit 142 calculates the normal vector N and the diffuse reflection coefficient K d The diffuse reflection component I obtained by photographing the light with the polarized light photographing device 40 is d From the image, it is obtained by solving the simultaneous equations based on the derivation method shown below. Here, the normal vector is a unit vector, the calculated vector N is normalized to the surface normal vector, and the Euclidean norm is the diffuse reflection coefficient K d It becomes.

[0048] Here, as shown in Fig. 8(a), three lights (L 1 , L 2 , L 3 ) is photographed by the polarization photographing device 40, and the formula (3) shown in FIG. 8(b) is derived according to Lambert's cosine law shown in the above formula (2). As shown in FIG. 8(c), the normal vector is expressed by equation (4), and the unit normal vector is expressed by equation (5).

[0049] By substituting these formulas (4) and (5) into formula (3), formula (6) in FIG. 9(a) is obtained. Then, the illumination position and illumination intensity indicated in the initial setting information 200, and the observation light intensity obtained by the camera (diffuse reflection component I d ) are expanded as constants (Ca, Cb, Cc) on the right side of equation (7) (see FIG. 9(b)). This results in the simultaneous linear equations shown in equation (8) (see FIG. 9(c)). By solving this simultaneous equation, the norm of the normal vector n is calculated as the diffuse reflection coefficient K d This results in the diffuse reflection coefficient K d and the normal vector (surface normal vector) can be calculated from equation (9) shown in FIG.

[0050] In this way, the diffuse reflection calculation unit 142 calculates the light intensity from the polarized lighting devices (three lighting devices (L 1 , L 2 , L 3 )) is a diffuse reflection component I separated by the reflection component separation unit 141 for the reflected light obtained by irradiating the subject with polarized light of multiple wavelengths (RGB). dand the illumination position and illumination intensity of the polarized illumination device 30 are applied to Lambert's cosine law to obtain the diffuse reflection coefficient K d And normal information (normal vector) can be calculated.

[0051] Returning to FIG. 1, the description of the configuration will be continued. The specular reflection calculation unit 143 calculates the specular reflection component I s Using this, the parameter (γ) indicating the degree of roughness of the object surface (hereinafter referred to as "roughness (γ)") and the refractive index n j Calculate.

[0052] <Roughness (γ) calculation process> First, a description will be given of the calculation process of the roughness (γ) of the subject surface by the specular reflection calculation unit 143. Note that in this embodiment, the description will be made in accordance with the Cook-Torrance specular reflection model shown in the following equation (10) as a predetermined specular reflection model of physically based rendering.

[0053]

number

[0054] In this embodiment, as shown in FIG. 10, the incident angle (θ i ), the angle (θ r ) and the observed light intensity V is the specular reflection component I sHere, the incident light and reflected light used in the processing are treated as p-waves and s-waves independently so as not to take into account the influence of the Fresnel term (F). In addition, the positions of the lighting (polarized lighting device 30), the adjacent camera (polarized photography device 40), and the lighting adjacent to the camera are all symmetrical with respect to the photography booth BH, and if the subject is less than a certain size, the angle between the lighting vector (L) and the line of sight vector (V) can be assumed to be the same. This makes it possible to ignore the influence of Fresnel reflection. Note that the geometric attenuation coefficient (G) has no effect on determining the brightness during rendering, so it is set to a constant of 1 and is not considered in this embodiment.

[0055] In this embodiment, the normal distribution function D is expressed by the following formula (11). This expresses the normal distribution function as a Gaussian distribution function. Details are shown in Reference 1 (Tanaka Norihiro, Tominaga Masaharu, "Analysis and Estimation of 3D Reflection Models," Information Processing Society of Japan Transactions on Computer Vision and Image Media (CVIM), vol. 41, pp. 1-11, Dec. 2000).

[0056]

number

[0057] Here, the roughness (γ) of the subject surface is a parameter that indicates the degree to which the reflection of the illumination light is dispersed, and indicates the degree to which the normal to the surface is dispersed. 10, the angle between the surface normal vector and half the angle (half vector H) between the light source and the line of sight is defined as (ρ), and the angle at which the normal distribution function D(ρ) is half (1 / 2) of the maximum value is defined as (γ) for roughness (γ). In this embodiment, the illumination position, orientation, intensity, camera orientation, etc. are calibrated in advance from the positions of the three light sources (polarized illumination devices 30) and the position of one camera (polarized imaging device 40), so that (ρ) becomes known, and the unknown quantity (γ) can be found by Gaussian fitting (Gaussian curve fitting).

[0058] In this way, the specular reflection calculation unit 143 calculates the specular reflection component I s By applying the lighting position of the polarized lighting device 30, the camera position of the polarized imaging device 40, and the normal information (normal vector) to a predetermined specular reflection model (Cook-Torrance specular reflection model), the angle at which the normal distribution function D(ρ) is half its maximum value can be calculated as the roughness (γ) of the subject surface. The calculated surface roughness (γ) of the subject is susceptible to noise due to the dynamic range of the camera, etc. Therefore, it is possible to stabilize (γ) by averaging the values ​​of (γ) acquired with different lighting and cameras. Furthermore, the expression of the normal distribution function in this embodiment is not limited to the Gaussian distribution function described above, and it is also possible to derive the roughness (γ) by applying it to a GGX distribution function, a Beckman function, or the like.

[0059] <Calculation process of refractive index of object> Next, the specular reflection calculation unit 143 calculates the refractive index n j The calculation process will be described. In the Cook-Torrance specular reflection model shown in equation (10), as described above, the geometric attenuation coefficient (G) has little effect and is therefore considered to be a constant of 1. In addition, the normal distribution function (D), normal vector (N), lighting vector (L), and line of sight vector (V) are known, and π is a constant. Here, since p-waves and s-waves act independently, the total reflectance based on the Cook-Torrance specular reflection model is R s_spec ,R p_spec These are expressed by the following equations (12) and (13). rs is the amount of reflected s-wave light, i ps represents the amount of reflected p-wave light, and i s is the amount of incident light of s wave, i p represents the amount of incident p-wave light.

[0060]

number

[0061] Moreover, the following equations (14) and (15) can be derived from the Cook-Torrance specular reflection model (equation 10).

[0062]

number

[0063] By solving equations (12), (14), (13), and (15), respectively, r s,i,j ,r p,i,j In general, the illumination is unpolarized, so F=(r s,i,j +r p,i,j ) / 2. In this way, the specular reflection calculation unit 143 applies the lighting position of the polarized lighting device 30, the camera position of the polarized imaging device 40, the normal information (normal vector), and the normal distribution function (D) to a predetermined specular reflection model (Cook-Torrance specular reflection model), and can calculate the reflectance of the subject for each of the p-waves and s-waves for the number of observed light sources.

[0064] On the other hand, the behavior of reflection and refraction on the surface of a subject in different media is as shown in FIG. 11, where the refractive index of medium 1 (air in this embodiment) is defined as "n i ” and the refractive index of medium 2 (the object in this embodiment) is “n j " Then, according to the Fresnel equation, the reflectance of s-polarized light (r s,i,j ) is expressed by equation (16), and the reflectance of p-polarized light (r p,i,j ) is expressed by equation (17).

[0065]

number

[0066] From equations (16) and (17), the refractive index of the subject (n j ) the equation is summarized as the following equation (18).

[0067]

number

[0068] Here, since the medium 1 is air, it can be assumed to be known based on the following (Reference 2: Junichiro Nakanishi, Refractive index of air and propagation characteristics of electromagnetic waves, Sagami Institute of Technology Bulletin, 10(1), pp:23-31, 1976 / 3 / 31). At 15°C and 760mmHg, 0.03% carbon dioxide CO 2 The refractive index of dry air containing i is the wavelength λ 0 When (μ)=0.2 to 1.25, it is given by the following equation (19).

[0069]

number

[0070] In this manner, the specular reflection calculation unit 143 calculates the reflectance (r s,i,j ,r p,i,j ) and the refractive index of air n calculated from equation (19) i By substituting this into equation (18), the refractive index of the object (n j ) can be obtained.

[0071] In the above embodiment, as described in Fig. 3 and Fig. 4, the polarization direction of the RGB three-color light source is changed to 0 degrees and 90 degrees to light the light. However, it is also possible to obtain detailed spectral reflectance by multi-spectrum. For example, as shown in Fig. 12, the polarization direction of a light source with six different wavelengths (B+, B-, G+, G-, R+, R-) may be changed to 0 degrees and 90 degrees, and one sequence may be composed of 12 frames. In this way, it is possible to measure detailed spectral reflectance.

[0072] As another example, as shown in FIG. 13, by using 12-color spectral lighting (B+, B'+, B-, B'-, G+, G'+, G-, G'-, R+, R'+, R-, R'-) as more wavelength bands and using a corresponding multispectral polarization camera, it is possible to observe more detailed spectral reflectance without increasing the number of shooting frames required for one sequence. This makes it possible to reduce the blind spots of lighting that occur in subjects with complex shapes. Furthermore, by coding not only wavelength bands but also, for example, one light source wavelength using a comb filter or multispectral lighting with a controllable spectral distribution, it is possible to reduce the number of shooting frames required for one sequence without increasing the number of light source points.

[0073] In an implementation example in which this embodiment is actually applied, one surface of the regular octahedron of the photography booth BH shown in Fig. 3 is rotated and arranged so as to be installed on the bottom surface as shown in Fig. 12 and Fig. 13. This is because, in the arrangement shown in Fig. 3, it is difficult to install lighting from the bottom surface of the photography booth BH, and because it is easier to install the photography stage ST on which the subject is placed in the photography booth when it is set on the bottom surface as shown in Fig. 12 and Fig. 13.

[0074] Returning to FIG. 1, the description of the configuration will be continued. The three-dimensional model detailed information output unit 150 outputs the diffuse reflection coefficient K d , normal information (surface normal vector), roughness of the object surface (γ), refractive index of the object (n j ), and the reflectance of the subject (three-dimensional model detailed information) are output to an external device 5 such as a CG drawing device as predetermined three-dimensional model detailed information in accordance with the time when the polarization photographing device 40 photographed the subject in order to calculate each piece of information. d , normal information (surface normal vector), roughness of the object surface (γ), refractive index of the object (n j), and information on the reflectance of the subject may all be included, but depending on the performance and use of the external device 5, information to be included in the 3D model detail information is set in advance as specified 3D model setting information from among these pieces of information, and output to each external device 5. By acquiring this detailed three-dimensional model information, the CG rendering device (external device 5) is able to express the fine irregularities of the subject's surface as texture in real time.

[0075] In addition, the subject spectral reflectance (s-wave reflectance (r s,i,j ), p-wave reflectivity (r p,i,j ) may be fitted to a low-dimensional linear model and expressed, for example, using 7 to 8 basis functions for output. By approximating the calculated subject spectral reflectance using basis functions and their weights, complex spectral reflectance can be expressed using only weights and basis functions. When the number of basis functions is 7 to 8 and the expression is performed using 7 to 8 weighting coefficients, the data size increases and the load on data storage and transmission increases, but the accuracy increases. On the other hand, when the number of basis functions is reduced to 3, the data size decreases but the accuracy decreases, but this is advantageous when there are limitations on the transmission line or storage capacity.

[0076] In addition, the three-dimensional model detailed information output unit 150 determines the refractive index of the object (n j ) to the external device 5, rendering that takes p-waves and s-waves into consideration becomes possible. However, there are cases where the CG renderer of the external device 5 does not take polarization into consideration, or where it is preferable not to increase the load on the external device 5. In such cases, for example, R o The three-dimensional model detailed information output unit 150 may convert and output the result as R o is the specular reflection coefficient of light incident parallel to the normal, and is described in detail in (Reference 3) below. o By outputting the above to the external device 5, it becomes possible to render the image without significant degradation in quality. (Reference 3) “Schlicks approximation,” Wikipedia,<URL:https: / / en.wikipedia.org / wiki / Schlick's_approximation>

[0077] <Modification of the 3D model detailed information output unit> Next, a modified example of the three-dimensional model detail information output unit 150 (see Fig. 1) will be described with reference to Fig. 14. Instead of the three-dimensional model detail information output unit 150 according to this embodiment outputting predetermined three-dimensional model detail information, the three-dimensional model detail information output unit 150a according to the modified example has a function of acquiring information including the processing capability and presentation conditions related to rendering of each of the external devices 5 shown in Fig. 1 ("optimization condition information 51" to be described later) from each of the external devices 5, and generating and outputting optimal information for each of the external devices 5 (three-dimensional model detail information corresponding to each external device).

[0078] The three-dimensional model detailed information output unit 150a includes a transmission content determination unit 151, a detailed information output unit 152, and a 3DCG rendering unit 153, as shown in FIG. As an example, the external device 5, which is a CG drawing device, will be described as an external device 5A having high 3DCG drawing capability (represented as "High" in FIG. 14) and an external device 5B having low 3DCG drawing capability (represented as "Low" in FIG. 14).

[0079] The transmission content determination unit 151 acquires optimization condition information 51 from each external device 5 (here, external devices 5A and 5B). This optimization condition information 51 includes, for example, information on the type of device (stationary 3D television, HMD, smartphone, PC, etc.) for each external device 5, information on the specifications of hardware resources for 3D rendering (memory capacity, hardware capacity, CPU processing speed, etc.), information on the type of 3D data used such as point clouds, meshes, and NURBS curves, the diffuse reflection coefficient K d , normal information (surface normal vector), roughness of the object surface (γ), refractive index of the object (n j), the reflectance of the subject, and other information required by the external device 5 (presentation conditions), which are necessary for optimizing the 3DCG rendering of each external device 5.

[0080] The transmission content determination unit 151 first determines whether to output the three-dimensional model detail information 52 or output it as a two-dimensional image 53 as output information, using the optimization condition information 51 acquired from each external device 5. Here, the transmission content determination unit 151 determines to output the three-dimensional model detail information 52 in the case of a device with high 3DCG drawing capability (for example, a dedicated stereoscopic television equipped with a high-performance CPU) like the external device 5A. Then, the transmission content determination unit 151 instructs the detail information output unit 152 to generate the three-dimensional model detail information 52 according to the specifications and settings of the external device 5. For example, the transmission content determination unit 151 instructs the external device 5 to perform reduction (thinning out) of point clouds and polygons / control points according to the specifications of the external device 5, and generates the three-dimensional model detail information 52.

[0081] On the other hand, when the 3DCG drawing capability is low, such as in the case of external device 5B (for example, a PC equipped with a CPU with low 3DCG drawing processing capability), the transmission content determination unit 151 determines that texture information should be sent as a 2D image 53 rather than transmitting 3D model detail information 52, and outputs information to that effect to the 3DCG rendering unit 153. In this case, the transmission content determination unit 151 resamples the two-dimensional image for the external device 5B, acquires information such as resolution, and then outputs instruction information to the 3DCG rendering unit 153 including that information.

[0082] Based on an instruction from the transmission content determination unit 151, the detailed information output unit 152 outputs the diffuse reflection coefficient K d , normal information (surface normal vector), roughness of the object surface (γ), refractive index of the object (n j), and the reflectance of the subject are generated as detailed 3D model information 52 appropriate for each external device, and output to that external device 5 (external device 5A in this example).

[0083] The 3DCG rendering unit 153 has a 3DCG rendering function, and performs rendering processing on behalf of an external device 5 that does not have a rendering function or an external device 5 with low processing capabilities for 3D drawing (e.g., external device 5B), and converts and outputs a two-dimensional image 53 appropriate for the external device 5.

[0084] In this way, the three-dimensional model detail information output unit 150a can optimize the information to be output in accordance with the hardware resources and specifications of each external device 5. Note that the three-dimensional model detail information output unit 150a may reduce and transmit shape data or reduce the number of basis functions expressing spectral reflectance in accordance with the level of the 3DCG rendering capability of each external device 5 (three or more levels), rather than dividing the 3DCG rendering capability into two levels, such as high and low, as shown in Fig. 14.

[0085] Furthermore, when the three-dimensional model detail information output unit 150a transmits the three-dimensional model detail information 52 to the external device 5 and causes rendering on the external device 5 side, delays are expected to occur due to data distribution and an increase in processing paths. Therefore, the transmission content determination unit 151 may determine whether the delay time falls within a delay time defined in advance as a latency requirement based on the performance and network status of the external device 5 indicated in the obtained optimization condition information 51, and dynamically control whether to output the data as the three-dimensional model detail information 52 or as the two-dimensional image 53. The three-dimensional model detail information output unit 150a may obtain the latency requirement from each external device 5 by including it in the optimization condition information 51. By doing this, the texture acquisition device 1 can automatically control whether to output the data as three-dimensional model detail information 52 or as a two-dimensional image 53, based on whether the delay time (latency requirement) required for establishing interaction with the external device 5 falls within the range.

[0086] <Operation of the texture acquisition device> Next, the operation performed by the texture acquisition device 1 according to the present embodiment will be described with reference to FIG.

[0087] First, the initial setting information acquisition unit 110 of the texture acquisition device 1 acquires information such as the camera position, camera attitude, lighting position, light distribution characteristics, etc. of each device (depth measurement device 20, polarized lighting device 30 and polarized shooting device 40) installed in the shooting booth BH from a system management device or the like, and stores the information in the memory unit 12 as initial setting information 200 (step S1).

[0088] Next, the subject shape acquisition unit 120 of the texture acquisition device 1 transmits instruction information for causing the depth measurement device 20 arranged in the shooting booth BH to perform measurement via the measurement control unit 130. Then, the subject shape acquisition unit 120 receives the measurement information from the depth measurement device 20 to acquire a (rough) subject shape (step S2).

[0089] Next, the texture calculation unit 140 (reflection component separation unit 141) of the texture acquisition device 1 calculates the diffuse reflection component I d and the specular reflection component I s Then, separation is performed (step S3). Specifically, the texture calculation unit 140 (reflection component separation unit 141) causes the polarized lighting devices 30 to irradiate the subject with three-color illumination of RGB, for example, in a pattern as shown in Fig. 4, via the measurement control unit 130. Then, the reflected light is photographed by the polarized imaging device 40.

[0090] 6, the reflected component separation unit 141 generates a sine wave from the luminance values ​​obtained from the pixels with the polarization directions of 0 degrees, 45 degrees, 90 degrees, and 135 degrees. The reflected component separation unit 141 then calculates the minimum value I min Diffuse reflection component I d Let 2a, which is twice the amplitude a, be the specular reflection component I s Let us assume that.

[0091] Next, the texture calculation unit 140 (diffuse reflection calculation unit 142) calculates the diffuse reflection component I d Using the diffuse reflection coefficient K d And normal information is calculated (step S4). Here, the diffuse reflection calculation unit 142 applies Lambert's cosine law and the photometric stereo technique to derive the equation (3) shown in FIG. 8(b) and calculates the observed illumination intensity (diffuse reflection component I d 9C based on the information on the illumination position and the like indicated by the initial setting information 200. The diffuse reflection calculation unit 142 then derives the equation (8) shown in FIG. 9C by solving the simultaneous linear equations of equation (8). d And calculate normal information (normal vector).

[0092] Next, the texture calculation unit 140 (specular reflection calculation unit 143) calculates the specular reflection component I s The roughness (γ) of the object surface is calculated using (step S5). 10, the specular reflection calculation unit 143 defines the angle between the surface normal vector and half the angle (half vector H) between the light source and the line of sight as (ρ), and defines the roughness parameter (γ) as the angle at which the normal distribution function D(ρ) is half (1 / 2) of its maximum value as (γ).The specular reflection calculation unit 143 then determines the unknowns (γ) of the normal distribution function expressed by the above formula (11) by Gaussian fitting.

[0093] Next, the texture calculation unit 140 (specular reflection calculation unit 143) calculates the reflectance for each of the s-wave and the p-wave, based on the amount of incident reflected light and the amount of reflected light (step S6). Here, the specular reflection calculation unit 143 calculates the reflectance of each of the s-wave and p-wave based on the above-mentioned formulas (12) and (13).

[0094] Next, the specular reflection calculation unit 143 uses the calculated p-wave and s-wave reflectances to calculate the refractive index of the object (n j ) is calculated. Note that the refractive index of air (n i ) is stored in advance in the initial setting information 200 (step S7).

[0095] The three-dimensional model detailed information output unit 150 of the texture acquisition device 1 outputs the diffuse reflection coefficient K d , normal information (surface normal vector), roughness of the object surface (γ), refractive index of the object (n j ), and certain predetermined information among the reflectance of the subject is generated as detailed 3D model information, and in order to calculate each piece of information, the information is output to an external device 5 such as a CG drawing device in accordance with the time when the polarization imaging device 40 took the image (step S8). At this time, as described above, the three-dimensional model detailed information output unit 150 may express the reflectance as a basis function and its weight and include it in the three-dimensional model detailed information, or may express it as R o may be included in the detailed three-dimensional model information.

[0096] Furthermore, if the texture acquisition device 1 is equipped with a three-dimensional model detail information output unit 150a shown in FIG. 14, it may acquire optimization condition information 55 from each external device 5, and generate and output three-dimensional model detail information 52 according to the specifications and settings of the external device 5, or it may perform rendering in the three-dimensional model detail information output unit 150a (3DCG rendering unit 153), convert it into a two-dimensional image 53, and output it to the external device 5.

[0097] <Implementation example> An implementation example will be described. Here, as shown in Fig. 3, the photography booth BH is set up so that the polarized lighting devices 30 are placed at the vertices of a regular octahedron, and the depth measurement device 20 and the polarized photography device 40 are placed at the center of each face to photograph the inside of the regular octahedron. Also, as shown in Fig. 12 and Fig. 13, the arrangement is rotated so that one face of the regular octahedron becomes the base. Here, it is assumed that a linear polarizing filter is arranged in polarized illumination device 30, the polarization direction is not polarized, and shooting is performed in one sequence with three frames. As the polarization imaging device 40, for example, a 240Hz high-speed polarized RGB camera is used, and motion detection and motion cancellation processing is performed using optical flow between sequences. This suppresses the effects of subject movement within one sequence. In this implementation example, a 240Hz high-speed polarized RGB camera is used, so one sequence of three frames can be captured at 240 / 3=80Hz. With this configuration of the photography booth BH, texture acquisition processing is performed using the texture acquisition device 1 of this embodiment, and a file is output that includes detailed 3D model information along with metadata such as the acquisition time and other information (photography conditions, camera configuration information, etc.).

[0098] Here, the diffuse reflection coefficient K d A specific example of the effect of calculating normal information will be described. In the conventional method, the diffuse reflection coefficient K d Regarding the calculation of normal information (normal vector), the diffuse reflection component of reflected light I d and the specular reflection component I s On the other hand, in the texture acquisition device 1 according to the present embodiment, the diffuse reflection component I d and the specular reflection component I s and the diffuse reflection component I d (diffuse reflected light) only, the diffuse reflection coefficient K d And calculate normal information (normal vector).

[0099] As a result, as shown in FIG. 16(a), the diffuse reflection coefficient K d The result is the diffuse reflection coefficient K d This is closer to the true value than the result of Moreover, as shown in FIG. 16(b), the normal information (normal vector) of this embodiment using only diffuse reflected light (polygon A) is closer to the true value than the normal information (normal vector) without separating the reflected light (polygon B). According to the texture acquisition device 1 of the present embodiment, accurate values ​​of normal information (normal vectors) are calculated in this manner, and the roughness (γ) of the object surface and the refractive index (n j ) will also be more accurate. As a result, the texture acquisition device 1 can acquire detailed information of a three-dimensional model required for photorealistic CG rendering that can express the texture of the subject's surface in real time, and output the information as optimal information to each external device 5. [Explanation of symbols]

[0100] 1 Texture acquisition device 5 External device 10 Control section 11 Input / output section 12 Storage section 20 Depth measurement device 30 Polarized Lighting Device 40 Polarized photography device 40a Polarized RGB camera 51 Optimization Condition Information 52 3D model details 53 2D Images 110 Initial setting information acquisition unit 120 Object shape acquisition section 130 Measurement control section 140 Texture calculation section 141 Reflection component separation section 142 Diffuse reflection calculation unit 143 Specular reflection calculation section 150,150a 3D model detailed information output section 151 Transmission Content Judgment Unit 152 Detailed information output section 153 3DCG Rendering Department 200 Initial setting information 1000 Texture Acquisition System BH Photo Booth ST Shooting Stage

Claims

1. a photography booth having a depth measurement device, a polarized lighting device, and a polarized photography device, the photography booth being arranged to surround a subject; a texture acquisition device that is communicatively connected to the depth measurement device, the polarized illumination device, and the polarized imaging device, and that acquires a texture represented by a state of unevenness on the surface of the subject, The texture acquisition device includes: a storage unit for storing initial setting information including an illumination position and illumination intensity of the polarized illumination device, and a camera position and camera orientation of the depth measurement device and the polarized imaging device; a subject shape acquisition unit that acquires shape information of the subject by receiving depth information measured by the depth measurement device; a measurement control unit that controls the polarized illumination device to irradiate the subject with polarized light of multiple wavelengths; a reflection component separation unit that acquires reflected light from the subject irradiated with each of the polarized light beams of the plurality of wavelengths as information captured by the polarization imaging device, and separates the reflected light into a diffuse reflection component and a specular reflection component; a diffuse reflection calculation unit that applies the diffuse reflection component separated by the reflection component separation unit and the illumination position and illumination intensity of the polarized illumination device to Lambert's cosine law for reflected light captured by the polarization imaging device when the polarized illumination devices arranged in three directions irradiate the subject with polarized light of the multiple wavelengths, and calculates a diffuse reflection coefficient and normal information of the subject; the specular reflection components separated by the reflection component separation unit for each of the polarized light of the multiple wavelengths, the illumination position of the polarized illumination device, the camera position of the polarized imaging device, and the calculated normal information are applied to a predetermined specular reflection model, and an angle at which the normal distribution function obtained by the predetermined specular reflection model is half its maximum value is calculated as a parameter of roughness of the surface of the object; a specular reflection calculation unit that applies the illumination position of the polarized lighting device, the camera position of the polarized imaging device, the normal information, and the normal distribution function to the predetermined specular reflection model, calculates the reflectance of the subject for each of the p-wave and s-wave of the polarized light of the multiple wavelengths, and calculates the refractive index of the subject by applying the calculated reflectance to a Fresnel equation related to reflection in a medium with a different refractive index; and a three-dimensional model detailed information output unit that outputs information including any of the calculated diffuse reflection coefficient, the normal information, the roughness of the subject's surface, the reflectance, or the refractive index as three-dimensional model detailed information in accordance with the time when the polarization imaging device captured the image to calculate the information.

2. A texture acquisition device that is communicatively connected to a depth measurement device, a polarized lighting device, and a polarized photography device that are arranged to surround a subject in a photography booth, and that acquires a texture represented by a state of unevenness on the surface of the subject, a storage unit for storing initial setting information including an illumination position and illumination intensity of the polarized illumination device, and a camera position and camera orientation of the depth measurement device and the polarized imaging device; a subject shape acquisition unit that acquires shape information of the subject by receiving depth information measured by the depth measurement device; a measurement control unit that controls the polarized illumination device to irradiate the subject with polarized light of multiple wavelengths; a reflection component separation unit that acquires reflected light from the subject irradiated with each of the polarized light beams of the plurality of wavelengths as information captured by the polarization imaging device, and separates the reflected light into a diffuse reflection component and a specular reflection component; a diffuse reflection calculation unit that applies the diffuse reflection component separated by the reflection component separation unit and the illumination position and illumination intensity of the polarized illumination device to Lambert's cosine law for reflected light captured by the polarization imaging device when the polarized illumination devices arranged in three directions irradiate the subject with polarized light of the multiple wavelengths, and calculates a diffuse reflection coefficient and normal information of the subject; the specular reflection components separated by the reflection component separation unit for each of the polarized light of the multiple wavelengths, the illumination position of the polarized illumination device, the camera position of the polarized imaging device, and the calculated normal information are applied to a predetermined specular reflection model, and an angle at which the normal distribution function obtained by the predetermined specular reflection model is half its maximum value is calculated as a parameter of roughness of the surface of the object; a specular reflection calculation unit that applies the illumination position of the polarized lighting device, the camera position of the polarized imaging device, the normal information, and the normal distribution function to the predetermined specular reflection model, calculates the reflectance of the subject for each of the p-wave and s-wave of the polarized light of the multiple wavelengths, and calculates the refractive index of the subject by applying the calculated reflectance to a Fresnel equation related to reflection in a medium with a different refractive index; and a three-dimensional model detailed information output unit that outputs information including any of the calculated diffuse reflection coefficient, the normal information, the roughness of the subject's surface, the reflectance, or the refractive index as specified three-dimensional model detailed information in accordance with the time when the polarization imaging device captured the image to calculate the information.

3. The texture acquisition device is communicatively connected to one or more external devices; The texture acquisition device described in claim 2, characterized in that instead of outputting the specified three-dimensional model detail information, the three-dimensional model detail information output unit acquires optimization condition information indicating information regarding the 3DCG rendering processing capacity of each of the external devices, selects information to be included in the three-dimensional model detail information from among the diffuse reflection coefficient, the normal information, the roughness of the subject surface, the reflectance and the refractive index depending on the processing capacity of each of the external devices, and generates and outputs three-dimensional model detail information depending on each of the external devices.

4. The texture acquisition device has a 3DCG rendering function, The three-dimensional model detailed information output unit The texture acquisition device described in claim 3, characterized in that if it is determined that the specified latency requirements acquired as the optimization condition information are not satisfied even if three-dimensional model detail information corresponding to each external device is output to the external device, the 3DCG rendering function performs 3DCG rendering on behalf of the external device, converts it into a two-dimensional image corresponding to the external device, and outputs it.

5. A texture acquisition program for causing a computer to function as the texture acquisition device according to any one of claims 2 to 4.

Images