Texture acquisition system, texture acquisition device, and texture acquisition program

Abstract

To acquire a state of fine irregularities of a surface of a subject for expressing a texture of the subject as a three-dimensional model in real time.SOLUTION: A texture acquisition apparatus 1 of a texture acquisition system 1000 includes a subject shape acquisition unit 120 that acquires shape information of a subject from a depth measurement apparatus 20, a measurement control unit 130 that controls irradiation of the subject with a plurality colors of deflection light by a deflection irradiation apparatus 30, an irregularity state calculation unit 141 that calculates roughness of a surface of the subject from distribution of color vectors of the deflection light generated by the irregularities of the surface of the subject, a reflection coefficient calculation unit 142 that separates reflection light into a diffusion reflection component and a specular reflection component and calculates a diffusion reflection coefficient and a specular reflection coefficient, a surface shape acquisition unit (illumination difference stereo shape acquisition unit 143) that acquires subject surface shape information based on a brightness of the diffusion reflection component of the surface of the subject, and a three-dimensional model detail information output unit 150 that outputs each of the calculated information as three-dimensional detail information.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a texture acquisition system, a texture acquisition device, and a texture acquisition program for acquiring shape, texture, and the like as subject information, which is material data handled in computer graphics (CG). [Background technology]

[0002] With the recent spread of AR (Augmented Reality) / VR (Virtual Reality) technology, there is an increasing need to convert real-world subjects into subject information (hereinafter sometimes referred to as "3D models") that can be handled using CG technology, etc. This is a technology generally called volumetric capture, which captures the surface pattern and shape of the entire subject using cameras and depth measurement devices arranged to surround the subject (see Non-Patent Documents 1 and 2).

[0003] When rendering a 3D model under the desired lighting conditions in CG technology, conventional volumetric capture involves placing the subject under a flat lighting environment, lighting the subject's surface evenly, and photographing the subject without shadows. Since the reflectance of the subject cannot be measured directly, the lighting is photographed as if it has no directionality, normalizing the amount of light incident on each part of the subject, and treating the brightness of the captured image obtained as a pseudo diffuse reflection coefficient. This allows shading to be added during rendering to express the appearance of the 3D model as expected under the lighting conditions in the virtual space. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] O. Schreer, et al., “Capture and 3D Video Processing of Volumetric Video,” 2019 IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan, 2019, pp. 4310-4314. [Non-Patent Document 2] O. Schreer, et al., “ADVANCED VOLUMETRIC CAPTURE AND PROCESSING,” [online], IBC2018, [Retrieved July 20, 2020], Internet<URL:https: / / www.ibc.org / download?ac=6559> Summary of the Invention [Problem to be solved by the invention]

[0005] However, in conventional volumetric capture techniques, the diffuse reflection component of reflected light is calculated artificially. In addition, the specular reflection component is not handled, or the coefficient (specular reflection coefficient) is set manually either uniformly across the entire subject or for each area. When the subject is a non-rigid body, it is difficult to manually assign coefficients for each area. For this reason, it is particularly difficult to apply this technique to live video. Furthermore, when observing the surface of a subject microscopically, the behavior of specularly reflected light differs depending on the presence or absence of fine irregularities. The presence or absence of irregularities on the subject's surface is a factor that greatly affects the visual impression of the subject when looking at it, such as its gloss and feel to the touch. Therefore, in order to express this texture, it is necessary to describe the behavior of surface reflection in units finer than area units, but this cannot be expressed at present.

[0006] The present invention has been made in consideration of the above-mentioned points, and has an objective of providing a texture acquisition system, a texture acquisition device, and a texture acquisition program that can acquire the fine unevenness of a subject's surface in real time as a three-dimensional model in order to express the texture of the subject. [Means for solving the problem]

[0007] In order to solve the above problems, the texture acquisition system of the present invention is configured to include a photography booth having a depth measurement device, a polarized lighting device, and a polarized photography device arranged to surround a subject, and a texture acquisition device that is communicatively connected to the depth measurement device, the polarized lighting device, and the polarized photography device, and acquires the texture represented by the uneven state of the subject's surface. The texture acquisition device includes a storage unit, a subject shape acquisition unit, a concave-convex state calculation unit, a reflection coefficient calculation unit, a surface shape acquisition unit, and a three-dimensional model detailed information output unit.

[0008] With this configuration, the texture acquisition device acquires shape information of the subject by receiving depth information measured by the depth measurement device through the subject shape acquisition unit, and the measurement control unit controls the polarized lighting device to irradiate the subject with polarized light of multiple colors.The unevenness state calculation unit acquires information on the reflected light from the subject irradiated with polarized light of multiple colors captured by the polarized photography device, and can calculate the roughness of the subject surface from the variance of the color vectors of the polarized light generated by the unevenness of the subject surface. This enables the texture acquisition device of the texture acquisition system to calculate the roughness of the object surface based on the variance of the color vector of the deflected light caused by the unevenness of the object surface.

[0009] In addition, the texture acquisition device separates reflected light into a diffuse reflection component and a specular reflection component using the reflection coefficient calculation unit, and calculates the diffuse reflection coefficient of the subject surface using the diffuse reflection component, the amount of light incident on the subject, and the shape information of the subject acquired by the subject shape acquisition unit and the lighting position, and also calculates the specular reflection coefficient of the subject surface using the specular reflection component, the amount of light incident on the subject, the roughness of the subject surface calculated by the unevenness calculation unit, and the camera position. This allows the texture acquisition device of the texture acquisition system to separate the reflected light into a diffuse reflection component and a specular reflection component, and then calculate the diffuse reflection coefficient and the specular reflection coefficient of the subject surface.

[0010] In addition, the texture acquisition device can acquire shape information of the subject's surface as subject's surface shape information by irradiating the subject with polarized light of multiple colors from multiple different light source directions via the measurement control unit, and estimating normals based on the brightness of the diffuse reflection components of the subject's surface photographed by the polarization imaging device. As a result, the texture acquisition device of the texture acquisition system can estimate the normal of the subject's surface using only the diffuse reflection components excluding the specular reflection components, thereby making it possible to more accurately acquire detailed shape information of the subject's surface.

[0011] The texture acquisition device also outputs information including the roughness, diffuse reflection coefficient, specular reflection coefficient, and surface shape information of the object by the three-dimensional model detailed information output unit. Information including the roughness of the object surface, the diffuse reflection coefficient, the specular reflection coefficient, and the object surface shape information In order to calculate the time, the polarization imaging device can output detailed 3D model information according to the time the image was taken. This allows the texture acquisition device of the texture acquisition system to output more detailed information of the three-dimensional model to, for example, a CG drawing device in real time.

[0012] In this way, with the texture acquisition system of the present invention, it is possible to acquire in real time accurate diffuse reflection coefficients and specular reflection coefficients of the subject's surface, the uneven state (roughness) of the subject's surface, and detailed shape information of the subject's surface (subject's surface shape information), which are not possible with methods using conventional volumetric capture technology, in order to express the fine unevenness of the subject's surface in order to express the texture of the subject. Effect of the Invention

[0013] According to the present invention, it is possible to provide a texture acquisition system, a texture acquisition device, and a texture acquisition program that acquire the fine unevenness of a subject's surface in real time as a three-dimensional model in order to express the texture of the subject. [Brief description of the drawings]

[0014] [Figure 1]1 is a functional block diagram showing an overall configuration of a texture acquisition system including a texture acquisition device according to an embodiment of the present invention. [Diagram 2] 4 is an explanatory diagram relating to separation of a diffuse reflection component and a specular reflection component of reflected light in the present embodiment. FIG. [Diagram 3] FIG. 1 is an explanatory diagram showing an example of a photography booth in this embodiment that is configured as a regular hexahedron. [Figure 4] 2 is an explanatory diagram showing an RGB Bayer arrangement on a camera imaging plate of the polarization imaging device according to the present embodiment. FIG. [Diagram 5] 1 is an explanatory diagram showing the relationship between roughness, which indicates the uneven state of the subject surface, and the degree of blurring of the specular reflection component; [Figure 6] 1 is an explanatory diagram showing an example in which coded illumination is irradiated onto an object in the polarized illumination device according to the present embodiment. FIG. [Figure 7] 1 is an explanatory diagram showing the relationship between roughness, which indicates the uneven state of the subject surface, and the dispersion of RGB color vectors due to coded illumination; [Figure 8] FIG. 1 is an explanatory diagram showing the distribution of color vectors of specularly reflected light in RGB space. [Figure 9] The roughness σ calculated based on the principal component analysis is Objects with smooth surfaces, Objects with low roughness surfaces, <c>The graph shows the results of calculations for a subject with a rough surface and a subject with a large roughness. < / c> [Figure 10] 4 is an explanatory diagram relating to separation of a diffuse reflection component and a specular reflection component of reflected light in the present embodiment. FIG. [Figure 11] FIG. 2 is an explanatory diagram of the diffuse reflection component of a reflection model based on Lambert's cosine law. [Figure 12] FIG. 2 is an explanatory diagram of specular reflection based on the Phong reflection model. [Figure 13] FIG. 11 is an explanatory diagram showing an example of a dynamic pattern of coded illumination according to the present embodiment. [Figure 14] FIG. 13 is an explanatory diagram showing another example of a dynamic pattern of coded illumination according to the embodiment. [Figure 15] FIG. 1 is an explanatory diagram showing an example of a photography booth in this embodiment configured in a geodesic dome shape. [Figure 16] FIG. 1 is an explanatory diagram showing an example of a photography booth in this embodiment configured as a dodecahedron. [Figure 17] 4 is a flowchart showing a processing flow of the texture acquisition device according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, modes for carrying out the present invention (hereinafter, referred to as "embodiments") will be described with reference to the drawings. <Configuration of the texture acquisition system> First, with reference to FIG. 1, an overview of a texture acquisition system 1000 including a texture acquisition device 1 will be described, and then each component will be described.

[0016] As shown in Figure 1, the texture acquisition system 1000 includes a depth measurement device 20, a polarized lighting device 30 and a polarized shooting device 40 which are arranged in a shooting booth BH where a subject is photographed, and a texture acquisition device 1 which is communicatively connected to the depth measurement device 20, the polarized lighting device 30 and the polarized shooting device 40.

[0017] The texture acquisition device 1 according to this embodiment is useful for measuring subject information (three-dimensional model) for use in AR / VR, and as a method for capturing subject information in three-dimensional televisions. As a process for acquiring the texture of a subject (hereinafter referred to as "texture acquisition process"), it realizes (1) a process for separating diffuse reflection components and specular reflection components, (2) a process for measuring the unevenness of the subject surface, and (3) a process for calculating detailed shape information of the subject surface using photometric stereo.

[0018] (1) In the process of separating the diffuse reflection component and the specular reflection component, the texture acquisition device 1, as shown in FIG. 2, irradiates the subject S with polarized light (illumination light) produced by polarized lighting (for example, red lighting, green lighting, and blue lighting) from the polarized lighting device 30. The irradiated light reflected by the subject S is then photographed by a polarized imaging device 40 that selectively transmits the light in the polarization directions of, for example, 0 degrees, 45 degrees, 90 degrees, and 135 degrees (see FIGS. 4 and 10 described later). The photographed image P is an image that combines the specular reflection component, which appears as a highlight as if the light source is reflected, and the diffuse reflection component, which is the incident light that penetrates the inside of the subject and is diffused by absorbing the color components of the object. The texture acquisition device 1 utilizes the difference in the reflection behavior of the polarized light on the surface of the subject to obtain the diffuse reflection component I d and the specular reflection component I s By separating these, we obtain the diffuse and specular reflection coefficients (details will be given later).

[0019] (2) In the process of measuring the unevenness of the subject's surface, polarized lighting (for example, three colors: RGB (red, green, and blue)) is irradiated onto the subject. In the case of specular reflection, for example, on the surface of a dielectric subject, only images are captured where the angle of incidence = angle of reflection. However, if the subject surface is uneven, the light vector of the reflected light spreads due to the unevenness, and as a result, the unevenness is observed as a mixture of colors. In this embodiment, the roughness σ, which indicates the unevenness of the subject, is estimated using a principal component analysis technique from the variance state of color vectors in the vicinity of the pixel of interest (details will be described later).

[0020] (3) In the process of calculating detailed shape information of the subject's surface using photometric stereo, the lighting in the photography booth BH is dynamically changed according to the size of the subject and the scale of the unevenness of the surface to be measured. Photometric stereo is a method of measuring the shape of the subject by estimating the normal based on the brightness of the subject's surface photographed from multiple different light source directions to determine the inclination of the surface. The texture acquisition device 1 (1) calculates the diffuse reflection component I d By dynamically changing the lighting in the photography booth BH and taking photographs using the diffuse reflection coefficient, it is possible to obtain a more detailed shape of the subject's surface than with conventional methods.

[0021] In this way, the texture acquisition device 1 can acquire accurate diffuse reflection coefficients and specular reflection coefficients on the subject's surface, the uneven state of the subject's surface (roughness σ), and detailed shape information of the subject's surface based on photometric stereo, which cannot be acquired by a method using conventional volumetric capture technology.The texture acquisition device 1 then outputs detailed information of the three-dimensional model acquired in this way ("detailed three-dimensional model information" to be described later) to a CG drawing device (not shown) or the like. The texture acquisition system 1000 including the texture acquisition device 1 according to this embodiment will be described in detail below.

[0022] <Configuration of the texture acquisition device> The texture acquisition device 1 according to this embodiment is configured by a general computer, and includes a control unit 10, an input / output unit 11, and a storage unit 12, as shown in FIG. The input / output unit 11 is made up of a communication interface for transmitting and receiving information, and an input / output interface for transmitting and receiving information to and from an input device such as a keyboard and an output device such as a monitor.

[0023] The storage unit 12 is configured with a flash memory, a hard disk, a RAM (Random Access Memory), etc. In this storage unit 12, initial setting information 200 (details will be described later) for the texture acquisition process by the texture acquisition device 1 is stored. The storage unit 12 also temporarily stores a program for executing each function of the control unit 10 (texture acquisition program) and information necessary for the processing of the control unit 10.

[0024] The control unit 10 is responsible for overall control of the texture acquisition device 1, and is composed of an initial setting information acquisition unit 110, a subject shape acquisition unit 120, a measurement control unit 130, a texture calculation unit 140, and a 3D model detailed information output unit 150, as shown in FIG. The control unit 10 is realized, for example, by a CPU (Central Processing Unit) (not shown) expanding a program (texture acquisition program) stored in the storage unit 12 into a RAM and executing the program.

[0025] When the texture acquisition device 1 executes the texture acquisition process, the initial setting information acquisition unit 110 acquires information including the lighting position, light distribution characteristics, and camera posture of each device installed in the photography booth BH in advance as initial setting information 200, and stores the information in the storage unit 12. An example of the initial setting information acquisition unit 110 acquiring the initial setting information 200 will be described with reference to FIG. FIG. 3 shows an example in which the photography booth BH is configured as a regular hexahedron. The depth measurement device 20, the polarized lighting device 30, and the polarized photography device 40 are arranged to surround the subject. As shown in FIG. 3, for example, the depth measurement device (depth camera) 20 and the polarized photography device 40 (e.g., a polarized RGB camera) are arranged at the vertices and center of the sides of the regular hexahedron to photograph the subject placed inside the photography booth BH. The polarized lighting devices 30 are arranged on each face of the regular hexahedron. Here, it is assumed that the photography booth BH is configured with a flat panel display (surface light source) and a linear polarizing filter is arranged. Note that the regular hexahedron is just one example, and the invention is not limited to this (see Figs. 15 and 16 described below).

[0026] The initial setting information acquisition unit 110 acquires information on the lighting position where the polarized lighting device 30 is placed, the light distribution characteristics measuring the luminous intensity in each direction in space, and the camera positions (X, Y, Z coordinates) and camera attitudes (roll, pitch, yaw) where the depth measurement device 20 and the polarized shooting device 40 are placed from an external device, etc. as initial setting information 200, and stores it in the memory unit 12. It is assumed that the pair of depth measurement device 20 and polarized illumination device 30 arranged at one location in photography booth BH has the same camera position and the same camera attitude.

[0027] 1, the subject shape acquiring unit 120 transmits instruction information for causing each depth measuring device 20 arranged in the photographing booth BH to perform measurement via the measurement control unit 130. Then, the subject shape acquiring unit 120 receives measurement information from each depth measuring device 20 to acquire a (rough) subject shape. The depth measurement device 20 may be, for example, a Time of Flight (ToF) camera that calculates depth by emitting near-infrared light from the camera, measuring the time it takes for the light to reflect off an object and return.

[0028] The measurement control unit 130 cooperates with the subject shape acquisition unit 120 and the texture calculation unit 140 to control each device by sending instruction information to the depth measurement device 20, polarized lighting device 30, and polarized photography device 40, which are arranged in the photography booth BH. Specifically, the measurement control unit 130 transmits instruction information to the depth measurement device 20 to cause it to measure the subject shape.

[0029] In addition, when the texture calculation unit 140 executes the texture acquisition process, the measurement control unit 130 transmits instruction information to the polarized lighting device 30 to control the device to achieve a predetermined pattern of lighting conditions as shown in Figures 13 and 14 described below. The predetermined pattern shown in Fig. 13 dynamically changes the lighting conditions according to the size of the object and the scale of the unevenness of the surface to be measured, and here, RGB (red, green, blue) lighting is divided into sections of a predetermined size and coded for illumination. Note that the hatched lines in each color section indicate the polarization direction of the linear polarizing filter, and here indicate that polarized light with polarization directions of 0 degrees, 45 degrees, 90 degrees, and 135 degrees is irradiated. The measurement control unit 130 transmits instruction information to the polarized illumination device 30 for controlling the dynamic pattern of this coded illumination.

[0030] In addition, the measurement control unit 130 transmits instruction information to the polarized photographing device 40 to photograph the reflected light in response to the texture calculation unit 140 executing the texture acquisition process and the polarized lighting device 30 irradiating the subject with illumination light.

[0031] The camera imaging plate of the polarization imaging device 40 (referred to here as a "polarized RGB camera") has an RGB Bayer array, for example, as shown in Fig. 4. Furthermore, each RGB pixel is divided into four pixels, and each pixel selectively transmits components with polarization directions of 0 degrees, 45 degrees, 90 degrees, and 135 degrees. In Fig. 4, the direction of the hatched area indicated by diagonal lines indicates the direction of the polarized light components that are transmitted.

[0032] The texture calculation unit 140 performs the above-mentioned texture acquisition processing, which includes separating the diffuse reflection component and the specular reflection component, measuring the unevenness of the object surface, and calculating detailed shape information of the object surface by photometric stereo. The texture calculation unit 140 includes a unevenness calculation unit 141, a reflection coefficient calculation unit 142, and a photometric stereo shape acquisition unit (surface shape acquisition unit) 143.

[0033] The unevenness calculation section 141 calculates roughness σ, which indicates the unevenness of the object surface, from the variance of the color vector of the deflected light caused by the unevenness of the object surface. The unevenness of a subject can be expressed by various parameters such as the fineness of the unevenness, but it can also be treated as the degree to which reflected objects such as light sources are blurred on the subject's surface. 5, when RGB lighting indicated by reference numeral 51 is irradiated onto a spherical subject 52 and the specular reflection component is photographed, the rougher the surface of the subject 52 is, the greater the degree of blurring of the RGB color components becomes, as indicated by reference numeral 53. In other words, the greater the variance of the color vectors, the greater the roughness of the subject surface.

[0034] In this embodiment, polarized lighting is used as a surface light source, and the three colors of RGB are used to code the lighting on the wall of the photography booth (coded lighting γ l ) to be irradiated onto the subject (see FIG. 6). In Fig. 6, the three colors of RGB are polarized at different angles (γ l1 ,γ l2 ,γ l3 ) is shown as an example in which the subject is irradiated with the light. This is to avoid cases in which the incident angle α becomes the Brewster angle and measurement becomes impossible in a single direction. Note that the deflected light is not limited to 0 degrees, 45 degrees, and 90 degrees. Furthermore, the present invention is not limited to using polarized illumination in three colors, RGB, but may use polarized light in a plurality of colors, and may also be applicable to five-color or polychromatic light.

[0035] When focusing on a certain part of the subject's surface, if the surface is flat like a mirror, only parts where the angle of incidence = the angle of reflection are captured by the camera (polarized RGB camera 40a) (see reference numeral 71 in FIG. 7). However, if the subject's surface is uneven, the degree to which RGB is observed as a mixture of colors at the pixel position of interest increases depending on the degree of roughness. In other words, the greater the degree of roughness of a part of the subject's surface (microsurface structure), the greater the variance of the color vectors (see reference numerals 72 and 73 in FIG. 7). In this embodiment, the roughness of the subject's surface is estimated from the variance of the color vectors using the relationship between the roughness of the subject's surface and the variance of the color vectors.

[0036] Specifically, unevenness calculation section 141 estimates roughness σ, which indicates the unevenness of the subject, from the variance state of color vectors in the vicinity of a pixel of interest, using a principal component analysis technique. Figure 8 shows the distribution of color vectors of specularly reflected light in RGB space. The specularly reflected light around a part of a subject of interest was captured with a polarized RGB camera 40a (see Figure 6), and the specularly reflected light was plotted in RGB space. Figure 8(a) shows the distribution of color vectors when the subject has a smooth surface. Figure 8(b) shows the distribution of color vectors when the subject has a rough surface.

[0037] Then, the unevenness calculation unit 141 applies principal component analysis to the specular reflection color vector in the RGB space to calculate the first principal component contribution rate and the third principal component contribution rate. The unevenness calculation unit 141 calculates the roughness σ by the formula (1) shown in FIG. 8 using the calculated first principal component contribution rate and third principal component contribution rate.

[0038] According to formula (1), the closer the values ​​of the first principal component contribution rate and the third principal component contribution rate are, the closer the roughness σ is to "1." In other words, the greater the variance of the color vector of the specular reflected light, the greater the variation in the color vector, the smaller the difference between the values ​​of the first principal component contribution rate and the third principal component contribution rate, and the closer the roughness σ is to "1." In this way, the state of roughness σ can be quantified using the first principal component contribution rate and the third principal component contribution rate obtained by principal component analysis.

[0039] Figure 9 shows the roughness σ calculated based on the principal component analysis. Objects with smooth surfaces, Objects with low roughness surfaces, <c>The graph shows the results of calculations for a subject with a rough surface and a subject with a large roughness. In this graph, the average roughness σ is< / c> "0.55" for subjects with smooth surfaces, "0.82" for subjects with low roughness surfaces, <c>The result was "0.91" for a subject with a rough surface. Thus, the smaller the roughness σ value, the smoother the surface with fewer irregularities, and the larger the value, the rougher the surface with more irregularities. Note that here, principal component analysis is performed on a size of ±3 pixels around the pixel of interest.

[0040] In this way, the unevenness calculating section 141 can calculate the roughness σ of the target portion of the subject's surface by using the contribution rates (the first principal component contribution rate and the third principal component contribution rate) calculated by the principal component analysis.

[0041] Returning to FIG. 1, the reflection coefficient calculation unit 142 calculates the diffuse reflection component I d and the specular reflection component I s Then, the reflection coefficient calculation unit 142 separates the acquired diffuse reflection component I d The diffuse reflection coefficient K d In addition, the reflection coefficient calculation unit 142 calculates the acquired specular reflection component I s Based on the amount of light incident on the object, the roughness σ, and the camera position, the specular reflection coefficient K s The following is a detailed explanation.

[0042] <Separation of diffuse reflection and specular reflection components> First, with reference to FIG. 10, separation of the diffuse reflection component and the specular reflection component in the reflected light will be described. Here, the polarized lighting device 30 is used as a surface light source, and the light is coded on the wall of the photography booth using three colors, RGB, under the control of the measurement control unit 130, and irradiated onto the subject as lighting. At this time, the subject is photographed by the polarized photography device 40 (polarized RGB camera 40a). The camera imaging plate of this polarized RGB camera 40a has an RGB Bayer array as shown by reference numeral 101 in Fig. 10, and each RGB pixel is divided into four pixels, each of which selectively transmits components with polarization directions of 0 degrees, 45 degrees, 90 degrees, and 135 degrees. Note that the four-division pixel and the polarization directions of 0 degrees, 45 degrees, 90 degrees, and 135 degrees are merely examples, and the present invention is not limited to these.

[0043] The observed luminance I is expressed by the following equation (2) based on the dichromatic reflection model. The dichromatic reflection model states that the spectrum of reflected light can be expressed as a linear sum of the spectra of the diffuse reflection component and the specular reflection component.

[0044] I=I d +a(1+cos(2Θ-β))...Equation (2) Here, I d is the diffuse reflection component, a(1+cos(2Θ-β)) is the specular reflection component I s In addition, a represents the amplitude, Θ represents the polarization direction of the pixel, and β represents the phase.

[0045] The brightness values ​​obtained at the pixels with polarization directions of 0 degrees, 45 degrees, 90 degrees, and 135 degrees (I0, I 45 ,I 90 ,I 135 ) is the average value I ave The wave behaves as a sine wave when the angle is double the polarization angle. Therefore, the trigonometric formula sin(θ-β) 2 +cos(θ-β) 2 = 1 (β considers the phase), for example, I0 2 +I 45 2 = a (a is the amplitude of the trigonometric function). This allows us to calculate the amplitude a. Also, the minimum value I of the observed brightness I min is the diffuse reflection component I d and 2a, which is twice the amplitude a, is the specular reflection component I s It becomes.

[0046] Next, for the diffuse reflection component of the reflected light, the process of calculating the diffuse reflection coefficient will be described using the Lambert reflection model (reflection of a diffusing surface based on Lambert's cosine law) as an example. Also, for the specular reflection component, the process of calculating the specular reflection coefficient will be described using the Phong reflection model as an example.

[0047] <Calculation of diffuse reflectance coefficient> The diffuse reflection coefficient K d The calculation process will be described. The reflection coefficient calculation unit 142 calculates the diffuse reflection coefficient K based on Lambert's cosine law, which states that the intensity of reflected light is proportional to the cosine of the incident angle. d Calculate.

[0048] As shown in FIG. 11, the diffuse reflection light of the incident light I0 incident on the subject at an angle θ is directly proportional to the cosine of the angle θ between the incident light and the normal line N of the surface, based on Lambert's cosine law. The diffuse reflection coefficient K d is the I calculated in the above-mentioned process of separating the diffuse reflection component and the specular reflection component. d If the normal N obtained from the shape information of the object, the light source position (illumination position), and the amount of incident light I0 are known, then it can be calculated using the following equation (3). I d = K d I0cosθ...Equation (3)

[0049] <Calculation of specular reflection coefficient> The specular reflection coefficient K s The calculation process will be described. FIG. 12 is an explanatory diagram of specular reflection based on the Phong reflection model. Here, if the observation (photographing) viewpoint is shifted by α from the direction in which the incident light (incident light amount I0) is reflected specularly, the intensity I s can be calculated by the following formula (4): The deviation α of the observation (photographing) viewpoint is calculated from the camera position. I s = K s I0cos n α...Equation (4)

[0050] The exponent n of the trigonometric function cos in equation (4) is a value determined according to the roughness of the surface of the subject. In this embodiment, the relationship between the exponent n and the roughness σ calculated as above is linked by the following method. Here, the exponent n and the specular reflection coefficient K s [Method 1] calculates the power n and the specular reflection coefficient K using information obtained from multiple viewpoints. s We will now explain Method 2 for finding the above.

[0051] [Method 1] The reflection coefficient calculation unit 142 uses the roughness σ (here, σ1) obtained from a single viewpoint of a part of the object to observe objects having various surface roughnesses in advance, and calculates the exponent n and the specular reflection coefficient K s Request.

[0052] (Method 1-1) The reflection coefficient calculation unit 142 calculates the exponent n by using a lookup table created in advance. In advance, α, σ1, I s , K s , n are created as a lookup table. Then, the roughness σ1 calculated by the unevenness calculation unit 141, α obtained by prior observation, and the specular reflection component I s From the lookup table, the optimal exponent n and specular reflection coefficient K are obtained. s Extract the following:

[0053] (Method 1-2) Parameter fitting to higher-order equations of three or more dimensions. In the above formula (4), α, I0, I s is known from the measurement conditions and measurement results. By using the n calculated by the following equation (5) in this equation, the specular reflection coefficient K s Calculate. n=aσ1 2 +bσ1+c...Equation (5)

[0054] (Method 1-3) Using a trained neural network s , n is estimated. Let the input be α, σ1, I s Let the output be K s Using a DNN (Deep Neural Network) with n as the weights, we train the weights using a training dataset measured in advance. This training dataset uses data acquired from various subjects, various light source positions (illumination positions), and camera measurement positions. The reflection coefficient calculation unit 142 actually calculates K s When calculating n, we use α, σ1, and I s Enter K s , n is output.

[0055] [Method 2] The reflection coefficient calculation unit 142 uses roughness σ (here, σ1 and σ2) obtained from multiple viewpoints of the object part to observe objects having various surface roughnesses in advance, and calculates the exponent n and the specular reflection coefficient K by, for example, the following methods 2-1 and 2-2. s Request.

[0056] (Method 2-1) Estimate n using a trained neural network. A nonlinear regression problem is treated as the problem, and a DNN is created with inputs α1, α2, σ1, σ2 and output n, and the weights are trained using a training dataset measured in advance. When the reflection coefficient calculation unit 142 actually calculates n, α1, α2, σ1, and σ2 are input to the trained DNN and n is output. The output n is inserted into the above equation (4) to obtain the specular reflection coefficient K s Request.

[0057] (Method 2-2) Using a trained neural network s , n is estimated. Let the input be α1, α2, σ1, σ2, I s1 , I s2 Let the output be K s For a DNN with n, weights are trained using a training dataset measured in advance. This training dataset uses data acquired from various subjects, various light source positions (illumination positions), and camera measurement positions. The reflection coefficient calculation unit 142 actually calculates K s When calculating n, we use α1, α2, σ1, σ2, I s1 , I s2 Enter K s , n is output.

[0058] In this way, the reflection coefficient calculation unit 142 calculates the diffuse reflection coefficient K d and the specular reflection coefficient K s can be calculated.

[0059] Dynamic lighting control Here, the dynamic control of the polarized illumination device 30 that the texture calculation unit 140 executes via the measurement control unit 130 will be described. In this embodiment, polarized lighting from the polarized lighting device 30 is used as a surface light source, and lighting coded using the three colors of RGB (coded lighting) is irradiated onto the subject (see Figs. 6 and 10). At this time, the measurement control unit 130 dynamically changes the lighting conditions according to the size of the subject and the scale of the unevenness of the surface to be measured. This satisfies the lighting conditions that allow the photometric stereo shape acquisition unit 143 to acquire the detailed shape of the subject surface.

[0060] With reference to FIG. 13, an example of a dynamic pattern of coded illumination is described. As shown in FIG. 13, the measurement control unit 130 changes the color, polarization direction, and scale pattern for the three colors, RGB, on a frame-by-frame basis to illuminate the subject. In the example shown in Fig. 13, the scale is changed from small to medium to large every two frames out of a total of six frames. And, in the two frames, the combination of color and polarization direction changes every other frame. Note that "black" in Fig. 13 indicates the state where the illumination is not emitting light. In this way, even when subjects of different sizes are targeted, or when the scale of the surface irregularities differs depending on the part of the subject, more accurate measurements are possible. It should be noted that the dynamic pattern is not limited to changing the pattern with a predetermined regularity as shown in Fig. 13. As long as there is no anisotropic reflection, it is also possible to randomly change the code for the three RGB colors, the polarization direction, and the scale.

[0061] FIG. 14 is a diagram showing another example of a dynamic pattern of coded illumination according to this embodiment. In the example shown in Figure 14, the surface light source of a regular hexahedral photography booth illuminates the three surfaces adjacent to the reference vertex with each of the RGB colors, and six frames make up one sequence. One set also consists of two frames, with each color moving to the opposite surface in the next frame. By changing the direction of the light source of the same color within this set, shape measurement is performed using photometric stereo, which will be described later, and detailed shape information of the subject's surface ("subject surface shape information", which will be described later) is obtained.

[0062] The photography booth according to this embodiment is not limited to a regular hexahedron, and may be, for example, a geodesic dome shape as shown in Fig. 15, or a dodecahedron as shown in Fig. 16. If the lighting is configured with a surface light source such as a flat panel display, one of the constituent surfaces may be a light source unit (see Fig. 15(a)), and by adjusting the pattern presented as lighting, it is not necessary to control the emitted color on a surface-by-surface basis, and it is also possible to make each color be reflected arbitrarily on the surface of the subject (see Fig. 15(b)).

[0063] Returning to FIG. 1, the photometric stereo shape acquisition unit (surface shape acquisition unit) 143 dynamically changes the coded illumination according to the size of the subject and the scale of the surface irregularities, as described above, and acquires information captured by the polarization imaging device (polarized RGB camera) 40. In this case, the photometric stereo shape acquisition unit 143 estimates the normal based on the luminance of the object surface photographed from a plurality of different light source directions to obtain the inclination of the surface, and obtains detailed shape information of the object as object surface shape information. s Diffuse reflection component I excluding d By using a chisel, the inclination of the surface can be found while more accurately reflecting the uneven state of the subject surface.

[0064] The three-dimensional model detailed information output unit 150 outputs the diffuse reflection coefficient K d , specular reflection coefficient K s In order to calculate each piece of information, information including the roughness σ indicating the unevenness of the subject's surface, and detailed shape information of the subject's surface (subject's surface shape information) is output to an external device such as a CG drawing device as detailed 3D model information in accordance with the time when the polarization imaging device 40 captures the image. By acquiring this detailed 3D model information, the CG drawing device is able to express the fine unevenness of the subject's surface as a texture in real time.

[0065] <Operation of the texture acquisition device> Next, the operation performed by the texture acquisition device 1 according to the present embodiment will be described with reference to FIG.

[0066] First, the initial setting information acquisition unit 110 of the texture acquisition device 1 acquires information such as the camera position, camera attitude, lighting position, light distribution characteristics, etc. of each device (depth measurement device 20, polarized lighting device 30 and polarized shooting device 40) installed in the shooting booth BH from an external device, etc., and stores the information in the memory unit 12 as initial setting information 200 (step S1).

[0067] Next, the subject shape acquisition unit 120 of the texture acquisition device 1 transmits instruction information for causing the depth measurement device 20 arranged in the shooting booth BH to perform measurement via the measurement control unit 130. Then, the subject shape acquisition unit 120 receives the measurement information from the depth measurement device 20 to acquire a (rough) subject shape (step S2).

[0068] Next, the texture calculation unit 140 (unevenness calculation unit 141) of the texture acquisition device 1 calculates roughness σ, which indicates the unevenness of the subject surface (step S3). Here, it is assumed that the unevenness calculation unit 141 calculates the roughness σ by using a principal component analysis technique based on the variance state of color vectors in the vicinity of a pixel of interest. Specifically, as shown in Fig. 6, the unevenness calculation unit 141 causes the polarized lighting device 30 to irradiate RGB polarized light via the measurement control unit 130, and causes the polarized RGB camera 40a to capture the image. Then, the unevenness calculation unit 141 applies principal component analysis to the specular reflected light color vector in the RGB space, and calculates the first principal component contribution rate and the third principal component contribution rate. The unevenness calculation unit 141 calculates the roughness σ based on the formula (1) in Fig. 8 using the calculated first principal component contribution rate and third principal component contribution rate.

[0069] Next, the texture calculation unit (reflection coefficient calculation unit 142) of the texture acquisition device 1 calculates the diffuse reflection component I d and the specular reflection component I s Then, separation is performed (step S4). Specifically, the reflection coefficient calculation unit 142 generates a sine wave from the luminance values ​​obtained from pixels with polarization directions of 0 degrees, 45 degrees, 90 degrees, and 135 degrees, as shown in Fig. 10. Then, the reflection coefficient calculation unit 142 calculates the minimum value I min Diffuse reflection component I d Let 2a, which is twice the amplitude a, be the specular reflection component I s Let us assume that. At this time, the reflection coefficient calculation unit 142, via the measurement control unit 130, illuminates the subject with coded illumination from the polarized illumination device 30 by dynamically changing the color, polarization direction, and scale pattern for the three colors, RGB, on a frame-by-frame basis, as shown in FIG. 13.

[0070] Next, the reflection coefficient calculation unit 142 calculates the diffuse reflection coefficient K d is calculated (step S5). Here, the reflection coefficient calculation unit 142 calculates the reflection coefficient based on the observed angle θ, the amount of incident light I0, and the diffuse reflection component I d Inserting this into the above equation (3) based on Lambert's cosine law, we obtain the diffuse reflection coefficient K d Calculate.

[0071] Next, the reflection coefficient calculation unit 142 calculates the specular reflection coefficient K s is calculated (step S6). The reflection coefficient calculation unit 142 calculates the relationship between the roughness σ and the exponent n in the above equation (4) based on the Phong reflection model, using the above [Method 1] (Methods 1-1 to 1-3) or [Method 2] (Method 2-1, Method 2-2).

[0072] 14, for example, the photometric stereo shape acquisition unit (surface shape acquisition unit) 143 dynamically changes the coded illumination of the polarized illumination device 30 via the measurement control unit 130, and acquires information captured by the polarized imaging device (polarized RGB camera) 40. As a result, the photometric stereo shape acquisition unit 143 uses a photometric stereo technique to estimate the normal based on the luminance of the diffuse reflection components of the subject surface captured from a plurality of different light source directions, thereby determining the inclination of the surface, and acquires detailed shape information of the subject surface as subject surface shape information (step S7).

[0073] Next, the three-dimensional model detailed information output unit 150 calculates the diffuse reflection coefficient K d , specular reflection coefficient K s In order to calculate each piece of information, information including the roughness σ indicating the uneven state of the subject's surface, and detailed shape information of the subject's surface (subject's surface shape information) is output to an external device such as a CG drawing device as detailed 3D model information in accordance with the time when the polarization imaging device 40 took the image (step S8), and the processing is then terminated.

[0074] In this way, the texture acquisition system 1000 including the texture acquisition device 1 of this embodiment can acquire in real time the accurate diffuse reflection coefficient and specular reflection coefficient of the subject surface, the uneven state of the subject surface (roughness σ), and detailed shape information of the subject surface based on photometric stereo (subject surface shape information) for expressing the fine unevenness of the subject surface to express the texture of the subject. [Explanation of symbols]

[0075] 1 Texture acquisition device 10 Control section 11 Input / output section 12 Storage section 20 Depth measurement device 30 Polarized Lighting Device 40 Polarized photography device 40a Polarized RGB camera 110 Initial setting information acquisition unit 120 Object shape acquisition section 130 Measurement control section 140 Texture calculation section 141 Unevenness calculation unit 142 Reflection coefficient calculation unit 143 Photometric Stereo Shape Acquisition Unit (Surface Shape Acquisition Unit) 150 3D model detailed information output section 200 Initial setting information 1000 Texture Acquisition System BH Photo Booth< / c>

Claims

1. a photography booth having a depth measurement device, a polarized lighting device, and a polarized photography device, the photography booth being arranged to surround a subject; a texture acquisition device that is communicatively connected to the depth measurement device, the polarized illumination device, and the polarized imaging device, and that acquires a texture represented by a state of unevenness on the surface of the subject, The texture acquisition device includes: a storage unit for storing initial setting information including an illumination position of the polarized illumination device, and camera positions and camera orientations of the depth measurement device and the polarized imaging device; a subject shape acquisition unit that acquires shape information of the subject by receiving depth information measured by the depth measurement device; a measurement control unit that controls the polarized illumination device to irradiate the subject with polarized light of a plurality of colors; an unevenness calculation unit that acquires information obtained by capturing, by the polarization imaging device, reflected light from the object irradiated with the polarized light of the plurality of colors, and calculates the roughness of the object surface from the variance of the color vector of the polarized light generated by the unevenness of the object surface; a reflection coefficient calculation unit that separates the reflected light into a diffuse reflection component and a specular reflection component, calculates a diffuse reflection coefficient of the subject surface using the diffuse reflection component, the amount of light incident on the subject, the shape information of the subject acquired by the subject shape acquisition unit, and the illumination position, and calculates a specular reflection coefficient of the subject surface using the specular reflection component, the amount of light incident on the subject, the roughness of the subject surface calculated by the unevenness calculation unit, and the camera position; a surface shape acquisition unit that irradiates the subject with the polarized light of the multiple colors from multiple different light source directions via the measurement control unit, and acquires shape information of the subject surface as subject surface shape information by estimating a normal based on the luminance of a diffuse reflection component of the subject surface photographed by the polarization imaging device; a three-dimensional model detailed information output unit that outputs information including the roughness of the object surface, the diffuse reflection coefficient, the specular reflection coefficient, and the object surface shape information as three-dimensional model detailed information in accordance with the time when the polarization imaging device captures the image in order to calculate information including the roughness of the object surface, the diffuse reflection coefficient, the specular reflection coefficient, and the object surface shape information; A texture acquisition system comprising:

2. A texture acquisition device that is communicatively connected to a depth measurement device, a polarized lighting device, and a polarized photography device that are arranged to surround a subject in a photography booth, and that acquires a texture represented by a state of unevenness on the surface of the subject, a storage unit for storing initial setting information including an illumination position of the polarized illumination device, and camera positions and camera orientations of the depth measurement device and the polarized imaging device; a subject shape acquisition unit that acquires shape information of the subject by receiving depth information measured by the depth measurement device; a measurement control unit that controls the polarized illumination device to irradiate the subject with polarized light of a plurality of colors; an unevenness calculation unit that acquires information obtained by capturing, by the polarization imaging device, reflected light from the object irradiated with the polarized light of the plurality of colors, and calculates the roughness of the object surface from the variance of the color vector of the polarized light generated by the unevenness of the object surface; a reflection coefficient calculation unit that separates the reflected light into a diffuse reflection component and a specular reflection component, calculates a diffuse reflection coefficient of the subject surface using the diffuse reflection component, the amount of light incident on the subject, the shape information of the subject acquired by the subject shape acquisition unit, and the illumination position, and calculates a specular reflection coefficient of the subject surface using the specular reflection component, the amount of light incident on the subject, the roughness of the subject surface calculated by the unevenness calculation unit, and the camera position; a surface shape acquisition unit that irradiates the subject with the polarized light of the multiple colors from multiple different light source directions via the measurement control unit, and acquires shape information of the subject surface as subject surface shape information by estimating a normal based on the luminance of a diffuse reflection component of the subject surface photographed by the polarization imaging device; a three-dimensional model detailed information output unit that outputs information including the roughness of the object surface, the diffuse reflection coefficient, the specular reflection coefficient, and the object surface shape information as three-dimensional model detailed information in accordance with the time when the polarization imaging device captures the image in order to calculate information including the roughness of the object surface, the diffuse reflection coefficient, the specular reflection coefficient, and the object surface shape information; A texture acquisition device comprising:

3. The unevenness state calculation unit performing a principal component analysis on the color vector of the deflected light, and calculating the roughness of the surface of the object using the calculated first principal component contribution rate and third principal component contribution rate; The texture acquisition device according to claim 2 .

4. The measurement control unit is The polarized light of the plurality of colors that is irradiated onto the subject by the polarized illumination device is irradiated onto the subject by changing a color, a polarization direction, and a scale on a frame-by-frame basis using a surface light source. The texture acquisition device according to claim 2 or 3,

5. A texture acquisition program for causing a computer to function as the texture acquisition device according to any one of claims 2 to 4.

Images