Information processing device and method
The use of cross-polarized and parallel-polarized images in the information processing device and method addresses the challenge of specular reflection in 3D modeling by enhancing material estimation accuracy and suppressing component deterioration, resulting in improved material information quality.
Patent Information
- Application Number
- PCT/JP2024/016191
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-17
- Filing Date
- 2024-04-25
- Publication Date
- 2024-11-21
AI Technical Summary
Conventional 3D modeling methods, such as photogrammetry and Physically Based Rendering (PBR), face challenges in accurately estimating material information due to the presence of specular reflection components, which can lead to a reduction in the quality of material information, especially when using polarization-based imaging techniques that fail to sufficiently suppress these components.
An information processing device and method that utilize cross-polarized and parallel-polarized images to estimate the roughness of a 3D object's surface, generating roughness maps and integrating material information to suppress the deterioration of material quality by effectively separating and analyzing specular and diffuse reflection components.
The proposed solution enhances the accuracy of material estimation by effectively suppressing the impact of specular reflection components, leading to improved quality of material information and reduced deterioration in roughness and metallic maps, thereby improving the overall fidelity of 3D data generation.
Smart Images

Figure JP2024016191_21112024_PF_FP_ABST
Abstract
Description
Information processing device and method
[0001] The present disclosure relates to an information processing device and method, and more particularly to an information processing device and method that can suppress a decrease in the quality of material information.
[0002] Conventionally, methods for 3D modeling of 3D objects with three-dimensional shapes, such as photogrammetry, involve capturing images of the 3D object from multiple directions and generating 3D data based on the captured images.
[0003] One method for rendering the texture of such 3D models is called Physically Based Rendering (PBR), which precisely renders the surface based on physical behavior, measuring factors such as light reflection and refraction. PBR reproduces the optical characteristics (how an object appears under light) based on the physical behavior of the 3D object's surface. For example, material information indicating this physical behavior can include attributes such as albedo, roughness, metallic, and normal. When generating 3D data, in addition to 3D modeling, which estimates the three-dimensional shape of the 3D model, material estimation is also performed to estimate this material information as part of texturing.
[0004] However, if the specular reflection component (regular reflection component) of a 3D object is included in the captured image, there is a risk that the quality of the material information will be reduced. To address this issue, there are imaging methods that use a polarization sensor or a polarization camera to capture images of 3D objects (see, for example, Patent Documents 1 and 2). By polarizing the light from the subject in a specific polarization direction, the imaging unit can receive light with the specular reflection component suppressed.
[0005] International Publication No. WO 2022 / 259466 International Publication No. WO 2019 / 069536
[0006] However, this imaging method may not be able to sufficiently suppress specular reflection components, which may result in a decrease in the quality of material information.
[0007] The present disclosure has been made in light of such circumstances, and makes it possible to suppress a decrease in the quality of material information.
[0008] An information processing device according to one aspect of the present technology includes a roughness map generation unit that performs roughness estimation to estimate the roughness of a surface of a 3D object using a cross-polarized image and a parallel-polarized image, and generates a roughness map that is distribution information of the roughness corresponding to the parallel-polarized image, wherein the cross-polarized image is a captured image generated by cross-polarized imaging, and the parallel-polarized image is a captured image generated by parallel-polarized imaging, and the captured image is generated by irradiating a subject with light from a light-emitting unit and receiving the light from the subject by an imaging unit, and in the case of the cross-polarized imaging, the light from the light-emitting unit is polarized in a first polarization direction and the light from the subject is polarized in a second polarization direction perpendicular to the first polarization direction, and in the case of the parallel-polarized imaging, the light from the light-emitting unit is polarized in the first polarization direction and the light from the subject is polarized in the first polarization direction.
[0009] An information processing method according to one aspect of the present technology performs roughness estimation to estimate the roughness of a surface of a 3D object using a cross-polarized image and a parallel-polarized image, and generates a roughness map, which is distribution information of the roughness corresponding to the parallel-polarized image, wherein the cross-polarized image is a captured image generated by cross-polarized imaging, and the parallel-polarized image is a captured image generated by parallel-polarized imaging, and the captured image is generated by irradiating a subject with light from a light-emitting unit and receiving the light from the subject by an imaging unit, and in the case of the cross-polarized imaging, the light from the light-emitting unit is polarized in a first polarization direction and the light from the subject is polarized in a second polarization direction perpendicular to the first polarization direction, and in the case of the parallel-polarized imaging, the light from the light-emitting unit is polarized in the first polarization direction and the light from the subject is polarized in the first polarization direction.
[0010] In an information processing device and method according to one aspect of the present technology, roughness estimation is performed to estimate the roughness of a 3D object surface using a cross-polarized image and a parallel-polarized image, and a roughness map, which is distribution information of roughness corresponding to the parallel-polarized image, is generated.
[0011] FIG. 1 is a diagram for explaining an overview of photogrammetry. FIG. 1 is a diagram showing an example of a material estimation method. FIG. 1 is a diagram showing an example of cross polarization imaging. FIG. 2 is a diagram showing an example of parallel polarization imaging. FIG. 2 is a diagram showing an example of roughness estimation. FIG. 3 is a diagram showing an example of metallic estimation. FIG. 3 is a diagram for explaining increasing resolution. FIG. 4 is a block diagram showing an example of the main configuration of a 3D data generation device. FIG. 4 is a flowchart showing an example of the flow of 3D data generation processing. FIG. 5 is a diagram showing an example of material information integration. FIG. 5 is a diagram showing an example of a material estimation method. FIG. 6 is a diagram showing an example of roughness map integration. FIG. 7 is a diagram showing an example of reliability derivation. FIG. 8 is a diagram showing an example of roughness map integration. FIG. 9 is a block diagram showing an example of the main configuration of a roughness estimation unit. FIG. 10 is a block diagram showing an example of the main configuration of a roughness integration unit. FIG. 11 is a flowchart showing an example of the flow of 3D data generation processing. FIG. 12 is a flowchart showing an example of roughness estimation processing. FIG. 13 is a flowchart showing an example of roughness integration processing. FIG. 14 is a block diagram showing an example of the main configuration of an imaging device. FIG. 15 is a flowchart showing an example of the flow of imaging processing. FIG. 16 is a block diagram showing an example of the main configuration of a computer.
[0012] Below, modes for carrying out the present disclosure (hereinafter referred to as embodiments) will be described. The description will be made in the following order: 1. Literature etc. supporting technical content and technical terminology 2. Material estimation 3. Use of cross-polarized images and parallel-polarized images 4. First embodiment (3D data generation device) 5. Integration of material information 6. Second embodiment (3D data generation device) 7. Third embodiment (imaging device) 8. Supplementary notes
[0013] <1. Literature, etc. supporting technical content and technical terminology> The scope of what is disclosed in the present technology includes not only the content described in the embodiments, but also the content described in the following patent documents, etc. that were publicly known at the time of filing, and the content of other documents referenced in the following patent documents.
[0014] Patent Document 1: (described above) Patent Document 2: (described above)
[0015] In other words, the contents of the above-mentioned patent documents and the contents of other documents referenced in the above-mentioned patent documents are also used as the basis for determining the support requirements.
[0016] <2. Material Estimation> <Photogrammetry> Conventionally, one method for 3D modeling of 3D objects with three-dimensional shapes, such as photogrammetry, involves capturing images of the 3D object from multiple directions and generating 3D data based on the multiple captured images.
[0017] Photogrammetry is a technique for reconstructing highly accurate three-dimensional models using the principles of triangulation from multiple images taken from various viewpoints. In this specification, the term "accuracy" in reference to 3D data (3D models) refers not only to the reproducibility (accuracy, definition, etc.) of the three-dimensional shape of the target 3D object, but also to the reproducibility (accuracy, definition, etc.) of the texture applied to the surface of the 3D model. For example, cameras 11-1 to 11-5 shown in FIG. 1 capture images of a 3D object 10 from multiple viewpoints to obtain multiple captured images. These captured images are then used to perform processes called SfM (Structure from Motion) and MVS (Multiview Stereo), followed by post-processing such as meshing and texturing to generate 3D data 15.
[0018] In SfM, for example, corresponding points are searched for between captured images, the position and orientation of the camera are derived using epipolar constraints, and the positions of each corresponding point in three-dimensional space are identified by triangulation based on the camera position and orientation. In this specification, these points in three-dimensional space are also referred to as three-dimensional points. In other words, three-dimensional points corresponding to each corresponding point are identified. The entire three-dimensional point cloud identified in this way is then optimized by bundle adjustment.
[0019] In MVS, for example, a dense corresponding point search is further performed using the 3D point group derived as described above, and 3D points are added.
[0020] As described above, photogrammetry performs a global optimization calculation called bundle adjustment to minimize errors, which allows it to obtain very accurate results, but it also places a heavy burden on the calculations.Furthermore, because it is based on geometric calculations rather than physical measurements, in principle, the higher the resolution of the images used, the more accurate the model can be restored.
[0021] <Physically Based Rendering> One method for rendering textures for such 3D models is called Physically Based Rendering (PBR), which precisely renders the surface based on physical behavior by measuring the reflection and refraction of light. In other words, physically based rendering reproduces the optical characteristics (how an object looks when exposed to light) based on the physical behavior of the surface of a 3D object. For example, material information that indicates this physical behavior can include attributes such as albedo, roughness, metallic, and normal.
[0022] Albedo is an attribute that defines the color of a material. Albedo is also called base color. Roughness is an attribute that controls the roughness (diffusion of reflected light) or smoothness of a material's surface. This attribute determines how sharp or blurry reflections on the material are. Metallic is an attribute that defines whether a material is metallic or not (non-metallic). For "pure surfaces" such as pure metal, stone, or plastic, metallic is set to either "0" or "1." Normal is an attribute that controls the roughness of an object's surface.
[0023] When generating 3D data, in addition to 3D modeling such as photogrammetry mentioned above, material estimation was performed to estimate such material information as texturing.
[0024] <Effect of Specular Reflection Components on Material Estimation> However, if specular reflection components (regular reflection components) from a 3D object are included in a captured image, there is a risk that the quality of material information will be reduced. To address this issue, there are imaging methods that use a polarization sensor or polarization camera to capture images of 3D objects, as described in Patent Documents 1 and 2, for example. By polarizing the light from the subject in a predetermined polarization direction, the imaging unit can receive light with the specular reflection components suppressed.
[0025] However, this imaging method may not be able to sufficiently suppress specular reflection components, which may result in a reduction in the quality of material information. For example, the inclusion of specular reflection components in the captured image may reduce the accuracy of roughness and metallicity estimation.
[0026] 3. Use of Cross Polarized Images and Parallel Polarized Images Method 1 Therefore, as shown in the top row of the table in FIG. 2, material estimation is performed using cross polarized images and parallel polarized images (Method 1).
[0027] <Cross-polarized imaging> A cross-polarized image is an image generated by cross-polarized imaging. This image is generated by irradiating a subject with light from a light-emitting unit and receiving the light from the subject with an imaging unit. In cross-polarized imaging, the light from the light-emitting unit and the light from the subject are polarized in directions perpendicular to each other. That is, the light from the light-emitting unit is polarized in a first polarization direction, and the light from the subject is polarized in a second polarization direction perpendicular to the first polarization direction.
[0028] For example, as shown in Fig. 3 , in the case of cross-polarized light imaging, the light emitter 111 emits light, and the light from the light emitter 111 is polarized in a predetermined direction (first direction) by a polarizing filter 112 or the like, and the polarized illumination light is irradiated onto a 3D object 113. At this time, the illumination light is irradiated so that the illumination light is dominant in the light reflected from the 3D object 113. The imaging unit 115 then receives the reflected light to generate a captured image of the 3D object 113 as a subject. At this time, the reflected light is polarized by a polarizing filter 114 or the like in a second direction perpendicular to the first direction (i.e., orthogonal to the first direction), and is then received by the imaging unit 115.
[0029] Note that the reflection of light from the 3D object 113 includes specular reflection (regular reflection) and diffuse reflection. That is, the reflected light from the 3D object 113 may include a specular reflection component and a diffuse reflection component. In the case of cross-polarized light imaging, the polarization direction of the polarizing filter 114 is orthogonal to the polarization direction of the polarizing filter 112, so the specular reflection component of the light emitted from the light-emitting unit 111 can be suppressed.
[0030] That is, in the case of such cross-polarized light imaging, the imaging section 115 mainly receives the diffuse reflection component, and therefore the cross-polarized light image mainly contains the diffuse reflection component (diffuse component).
[0031] <Parallel Polarization Imaging> In contrast, a parallel polarization image is an image generated by parallel polarization imaging. This image is generated by irradiating a subject with light from a light-emitting unit and receiving the light from the subject by an imaging unit. In the case of parallel polarization imaging, the light from the light-emitting unit and the light from the subject are polarized in parallel directions (the same direction as each other). In other words, the light from the light-emitting unit is polarized in a first polarization direction, and the light from the subject is polarized in the same first polarization direction.
[0032] For example, as shown in Fig. 4 , in the case of parallel polarization imaging, the light-emitting unit 121 emits light, and the light from the light-emitting unit 121 is polarized in a predetermined direction (first direction) by a polarizing filter 122 or the like, and the polarized irradiation light is irradiated onto a 3D object 123. At this time, the irradiation light is irradiated so that the light reflected from the 3D object 123 is dominated by the irradiation light. Then, the imaging unit 125 receives the reflected light to generate a captured image with the 3D object 123 as the subject. At this time, the reflected light is polarized in the first direction by a polarizing filter 124 or the like and received by the imaging unit 125.
[0033] In the case of parallel polarization imaging, the polarization direction of the polarizing filter 124 is perpendicular to the polarization direction of the polarizing filter 122, so the imaging unit 125 receives both the specular reflection component and the diffuse reflection component. Therefore, the parallel polarization image contains the diffuse reflection component and the specular reflection component.
[0034] <Material Estimation> By using the cross-polarized image and the parallel-polarized image, which have different characteristics, the information processing device can perform more accurate material estimation, thereby suppressing degradation of the quality of material information.
[0035] <Method 1-1> When applying Method 1, any attributes may be included in the material information. For example, as shown in the second row from the top of the table in FIG. 2, an albedo map may be generated by gain-correcting a cross-polarized image (Method 1-1). The albedo map is distribution information of the albedo corresponding to a captured image (parallel polarization image or cross-polarized image). In other words, with Method 1-1, the albedo map is obtained by multiplying each pixel value (mainly the diffuse component) of the cross-polarized image by a gain.
[0036] For example, the information processing device may include an albedo map generator that performs gain correction on a cross-polarized image to generate an albedo map. For example, the albedo map generator may calculate a gain that minimizes the difference between a rendered image and the cross-polarized image actually captured, and generate an albedo map by multiplying the cross-polarized image by the calculated gain.
[0037] In this way, the albedo map generator can generate an albedo map with higher resolution, and the information processing device can therefore suppress degradation in the quality of the albedo map (i.e., material information).
[0038] <Method 1-2> When applying Method 1, for example, as shown in the third row from the top of the table in Fig. 2, a roughness map may be generated by analyzing glossy areas in a parallel polarization image to estimate roughness (Method 1-2). The roughness map is roughness distribution information corresponding to the captured image (parallel polarization image or cross polarization image).
[0039] For example, an information processing device may include a roughness map generator that uses a cross polarized image and a parallel polarized image to perform roughness estimation to estimate the roughness of a 3D object surface, and generates a roughness map that is distribution information of roughness corresponding to the parallel polarized image.Also, for example, an information processing method executed by an information processing device may use a cross polarized image and a parallel polarized image to perform roughness estimation to estimate the roughness of a 3D object surface, and generate a roughness map that is distribution information of roughness corresponding to the parallel polarized image.
[0040] As described above, the cross-polarized image mainly contains a diffuse reflection component. The parallel-polarized image contains a diffuse reflection component and a specular reflection component. By using the cross-polarized image and the parallel-polarized image, which have different characteristics, the information processing device can perform more accurate roughness estimation. Therefore, the information processing device can suppress a reduction in the quality of the roughness map (i.e., material information).
[0041] Any method may be used for roughness estimation using the cross polarization image and the parallel polarization image. For example, the roughness map generator may use a difference image between the parallel polarization image and the cross polarization image to detect a flash area included in the parallel polarization image, analyze the detected flash area, and perform roughness estimation based on the analysis result of the flash area.
[0042] Roughness is an attribute that indicates the degree of diffusion of reflected light, and indicates the degree to which the irradiated light is specularly reflected on a 3D object (i.e., the degree to which the reflected light is diffused). In other words, if a portion of a parallel polarization image that contains the specular reflection component of the irradiated light (also called flash light) (i.e., the portion where the irradiated light is specularly reflected) is defined as a flash region, roughness has a correlation with the spread function (PSF (Point Spread Function)) of that flash region.
[0043] In FIG. 5 , parallel polarization images 131 and 133 show examples of flash areas included in parallel polarization images. In parallel polarization images 131 and 133, the pixel values (brightness values) of each pixel are indicated by black or white. Pixels closer to black indicate lower brightness, while pixels closer to white indicate higher brightness. In parallel polarization images 131 and 133, pixels near the center form flash areas containing specular reflection components, which are white (i.e., high brightness). Graph 132 shows the distribution of pixel values in the flash area of parallel polarization image 131. Graph 134 shows the distribution of pixel values in the flash area of parallel polarization image 133.
[0044] For example, when flash light is strongly specularly reflected by a 3D object (when the diffuse reflection component is small), the flash area is formed relatively small, as shown in parallel polarization image 131 in Figure 5, and the pixel value changes sharply over a relatively narrow range, as shown in graph 132. Materials with low roughness are more likely to specularly reflect. Therefore, in this case, the extent of the flash area is relatively small, as shown in the example of parallel polarization image 131, and the pixel value changes sharply over a relatively narrow range, as shown in the example of graph 132.
[0045] In contrast, when flash light is weakly specularly reflected by a 3D object (when the diffuse reflection component is large), the flash area is relatively large, as shown in parallel polarization image 133 in Figure 5, and the pixel values change gradually over a relatively wide range, as shown in graph 134. Materials with high roughness are more likely to cause diffuse reflection. Therefore, in this case, the flash area becomes relatively large, as shown in the example of parallel polarization image 133, and the pixel values change gradually over a relatively wide range, as shown in the example of graph 134.
[0046] As described above, there is a correlation between the extent of the flash area (how the flash area expands) and roughness. Therefore, if the extent of the flash area (i.e., the PSF) is known, this correlation can be used to easily and accurately estimate roughness from the PSF.
[0047] The extent of the flash area may be determined by any method. For example, a flash area may be detected in a parallel polarization image or a cross polarization image, and the extent of the flash area may be determined by analyzing the flash area. As described above, a parallel polarization image contains a diffuse reflection component and a specular reflection component, while a cross polarization image contains primarily a diffuse reflection component. Therefore, a difference image obtained by subtracting a cross polarization image from a parallel polarization image (obtained by subtracting the corresponding pixel value of the cross polarization image from each pixel value of the parallel polarization image) contains primarily a specular reflection component. As described above, since the flash area is a specular reflection component (specular component), using this difference image allows detection and analysis of the flash area while suppressing the influence of unnecessary diffuse reflection components. In other words, using a difference image allows the extent of the flash area to be determined more easily and accurately than using a parallel polarization image or a cross polarization image.
[0048] Therefore, as described above, the roughness map generator may generate a difference image between the parallel polarization image and the cross polarization image, use the difference image to detect flash areas, analyze how the flash areas spread, and estimate roughness based on the analysis results. In this way, the roughness map generator can more easily and accurately estimate roughness.
[0049] Using this principle, a relationship between the extent of a flash area when a flash light is irradiated onto a material and the roughness of the material may be learned, and the extent of the flash area may be input into a neural network corresponding to the learning results, thereby obtaining a roughness estimation result (generating a roughness map). Alternatively, a difference image obtained by subtracting a cross-polarized image from a parallel-polarized image may be used instead of the extent of the flash area. That is, a relationship between a difference image when a flash light is irradiated onto a material and the roughness of the material may be learned, and the difference image may be input into a neural network corresponding to the learning results, thereby obtaining a roughness estimation result (generating a roughness map). Alternatively, a parallel-polarized image and a cross-polarized image may be used instead of the difference image. That is, a relationship between a parallel-polarized image and a cross-polarized image when a flash light is irradiated onto a material and the roughness of the material may be learned, and the parallel-polarized image and the cross-polarized image may be input into a neural network corresponding to the learning results, thereby obtaining a roughness estimation result (generating a roughness map).
[0050] <Method 1-3> When applying Method 1, for example, as shown in the fourth row from the top of the table in Fig. 2, a metallicity map may be generated by comparing a cross-polarized image with a parallel-polarized image to estimate metallicity (Method 1-3). The metallicity map is distribution information of metallicity corresponding to the captured image (parallel-polarized image or cross-polarized image).
[0051] For example, the information processing device may include a metallic map generator that performs metallic estimation using a cross-polarized image and a parallel-polarized image to generate a metallic map. Generally, it has been difficult to perform metallic estimation from a cross-polarized image or a parallel-polarized image. By using a cross-polarized image and a parallel-polarized image, the metallic map generator can easily and accurately perform metallic estimation. Therefore, the information processing device can suppress a reduction in the quality of the metallic map (i.e., material information).
[0052] Any method may be used to estimate metallicity using the cross-polarized image and the parallel-polarized image. For example, the metallic map generator may estimate that the subject (the surface material of the 3D object) is metallic if the pixel values of the parallel-polarized image for the processing target area are sufficiently large relative to the cross-polarized image.
[0053] The type of material can be estimated based on the magnitude of the specular reflection component and the diffuse reflection component, as shown in the graph of FIG. 6 . For example, in the graph of FIG. 6 , in region 141, the pixel values of the parallel polarization image and the cross polarization image are both small. Therefore, if the combination of the pixel values of the parallel polarization image and the cross polarization image falls within region 141, the material corresponding to that pixel is estimated to be black. In contrast, in region 142, the pixel values of the parallel polarization image and the cross polarization image are approximately equal. Therefore, if the combination of the pixel values of the parallel polarization image and the cross polarization image falls within region 142, the material corresponding to that pixel is estimated to be matte. Furthermore, in region 143, the pixel values of the parallel polarization image are slightly larger than those of the cross polarization image compared to region 142. In other words, this material is glossier than a matte material. Therefore, if the combination of the pixel values of the parallel polarization image and the cross polarization image falls within region 143, the material corresponding to that pixel is estimated to be plastic. Furthermore, in region 144, the pixel values of the cross polarization image are small and the pixel values of the parallel polarization image are large. In other words, this material has a very strong luster. Therefore, when a pixel value of a parallel polarized image and a pixel value of a cross polarized image are combined and fall within this region 144, the material corresponding to that pixel is presumed to be metallic.
[0054] The metallic map generator may determine the relationship between the pixel values of the cross-polarized image and the parallel-polarized image and the quality of the material, and estimate whether the material is metallic based on that relationship. For example, the metallic map generator may estimate whether the material corresponding to that pixel is metallic based on whether the combination of the pixel values of the parallel-polarized image and the cross-polarized image is located within region 144 of the graph in Figure 6, i.e., based on whether the pixel value of the parallel-polarized image is sufficiently larger than that of the cross-polarized image. In this way, the metallic map generator can easily and accurately estimate metallicity.
[0055] The metallic estimation result may be expressed as a binary value of "0" (not metallic) or "1" (metallic). Alternatively, the metallic estimation result may be expressed as a degree of metallicity (any value between 0 and 1).
[0056] The processing target region for metallic estimation may be any region. For example, the processing target region may be a single pixel, and metallic estimation may be performed for each pixel. Alternatively, the processing region may be a region made up of multiple pixels, and metallic estimation may be performed for each region (i.e., for each set of multiple pixels).
[0057] Note that, by utilizing this principle, the relationship between metallicity and a difference image obtained by subtracting a cross-polarized image from a parallel-polarized image may be learned, and the difference image may be input into a neural network corresponding to the learning results, thereby obtaining a metallicity estimation result (generating a metallicity map). Furthermore, a parallel-polarized image and a cross-polarized image may be applied instead of the difference image. In other words, the relationship between metallicity and a parallel-polarized image and a cross-polarized image may be learned, and the relationship between metallicity and a parallel-polarized image and a cross-polarized image may be input into a neural network corresponding to the learning results, thereby obtaining a metallicity estimation result (generating a metallicity map).
[0058] <Method 1-4> The roughness map and metallic map generated as described above have low resolution. Therefore, when applying Method 1, the roughness map and metallic map may be increased in resolution using the albedo map as a guide signal, as shown in the fifth row from the top of the table in FIG. 2 (Method 1-4).
[0059] For example, the information processing device may further include a resolution increasing unit that increases the resolution of the roughness map using the albedo map as a guide signal. Also, the information processing device may further include a resolution increasing unit that increases the resolution of the metallic map using the albedo map as a guide signal. Also, the information processing device may further include a resolution increasing unit that increases the resolution of the roughness map and the metallic map using the albedo map as a guide signal. In other words, this resolution increasing unit may increase the resolution of at least one of the roughness map and the metallic map using the albedo map as a guide signal.
[0060] For example, as shown in a square 151 in FIG. 7, the following formulas (1) and (2) may be used to increase the resolution of the material information (at least one of the roughness map and the metallic map).
[0061] ...(1) ... (2)
[0062] As shown in a square 152 in FIG. 7, the parameters in equations (1) and (2) have the following values:
[0063] In this way, the resolution increasing unit can increase the resolution of at least one of the roughness map and the metallic map, thereby enabling the information processing device to suppress degradation in the quality of the material information.
[0064] <Method 1-5> When applying Method 1, for example, as shown in the bottom row of the table in FIG. 2, the albedo map, roughness map, and metallic map may be integrated for each viewpoint (Method 1-5). As described with reference to FIG. 1, in 3D modeling such as photogrammetry, a 3D model is generated using multiple images captured from different viewpoints. As described above, material information (e.g., albedo map, roughness map, metallic map) is generated corresponding to the images captured from each viewpoint (parallel polarization image and cross polarization image). By integrating this material information from each viewpoint, material information corresponding to the entire 3D model can be generated.
[0065] In this specification, "integration" refers to combining multiple pieces of material information (maps) into a single piece of material information (map). Any method for combining the materials may be used. For example, the edges of multiple maps may be connected to generate a larger map. If there are redundant portions between maps (portions corresponding to the same positions on the 3D object), those portions may be superimposed (combined). The entirety of one map may be superimposed (combined) on the entirety of the other map. The maps do not have to be the same size; the entirety of one map may be superimposed (combined) on a portion of the other map. Any method for this combining (combining) may be used. For example, the values of one of the multiple maps to be combined (combined) may be selected. The number of maps to be combined may be any number of times. Each of the above-described methods may be combined with any other method. For example, any two or more of Methods 1-1 to 1-5 may be applied. These methods may also be combined with other methods.
[0066] <4. First embodiment> <3D data generation device> Fig. 8 is a block diagram showing an example of the main configuration of a 3D data generation device, which is an embodiment of an information processing device to which the present technology described above in <3. Use of cross polarized images and parallel polarized images> is applied. The 3D data generation device 200 shown in Fig. 8 is a device that performs processing such as 3D modeling and texturing based on a cross polarized image obtained by capturing cross polarized images of a 3D object and a parallel polarized image obtained by capturing parallel polarized images of the 3D object, and generates 3D data (the geometry and attributes of a 3D model).
[0067] The cross-polarized image and the parallel-polarized image input to the 3D data generation device 200 form a pair of images from the same viewpoint. That is, at least one cross-polarized image and one parallel-polarized image are input for one viewpoint. Furthermore, any number of cross-polarized images and parallel-polarized images from any number of viewpoints (any number of pairs) may be input to the 3D data generation device 200. For example, a cross-polarized image and a parallel-polarized image for each of multiple viewpoints may be input.
[0068] As described above, the 3D data generation device 200 may apply the present technology described above in <3. Use of Cross Polarized Images and Parallel Polarized Images>. For example, the 3D data generation device 200 may apply the above-described Method 1. Alternatively, the 3D data generation device 200 may apply the above-described Method 1-1. Alternatively, the 3D data generation device 200 may apply the above-described Method 1-2. Alternatively, the 3D data generation device 200 may apply the above-described Method 1-3. Alternatively, the 3D data generation device 200 may apply the above-described Method 1-4. Alternatively, the 3D data generation device 200 may apply the above-described Method 1-5. Alternatively, the 3D data generation device 200 may apply any combination of these methods.
[0069] 8 , the 3D data generation device 200 includes a gain correction unit 211, a roughness estimation unit 212, a metallic estimation unit 213, a resolution enhancement unit 214, an albedo integration unit 215, a roughness integration unit 216, and a metallic integration unit 217. The 3D data generation device 200 also includes a corresponding point detection unit 221, an image capture posture estimation unit 222, a geometry / normal estimation unit 223, and a UV placement determination unit 224. The albedo integration unit 215, the roughness integration unit 216, and the metallic integration unit 217 may be combined into a single processing unit (integration unit 231). The corresponding point detection unit 221, the image capture posture estimation unit 222, the geometry / normal estimation unit 223, and the UV placement determination unit 224 may be combined into a single processing unit (geometry generation unit 232).
[0070] The gain correction unit 211 performs processing related to the generation of an albedo map. In other words, the gain correction unit 211 can also be considered an albedo map generation unit. For example, the gain correction unit 211 may acquire a cross-polarized image input to the 3D data generation device 200. The gain correction unit 211 may generate an albedo map by multiplying (each pixel value of) the cross-polarized image by a gain (performing gain correction). The gain correction unit 211 may also supply the generated albedo map to the resolution enhancement unit 214. The gain correction unit 211 may also supply the generated albedo map to the albedo integration unit 215.
[0071] The roughness estimation unit 212 performs processing related to the generation of a roughness map. In other words, the roughness estimation unit 212 can also be considered a roughness map generation unit. For example, the roughness estimation unit 212 may acquire a cross-polarized image and a parallel-polarized image from the same viewpoint that are input to the 3D data generation device 200. The roughness estimation unit 212 may also perform roughness estimation using the cross-polarized image and the parallel-polarized image and generate a roughness map corresponding to the parallel-polarized image. In this case, the roughness estimation unit 212 may detect a flash area included in the parallel-polarized image using a difference image between the acquired parallel-polarized image and the cross-polarized image, analyze the detected flash area, and perform roughness estimation based on the analysis result of the flash area. The roughness estimation unit 212 may also supply the generated roughness map to the resolution enhancement unit 214.
[0072] The metallic estimation unit 213 performs processing related to the generation of a metallic map. In other words, the metallic estimation unit 213 can also be considered a metallic map generation unit. For example, the metallic estimation unit 213 may acquire a cross-polarized image and a parallel-polarized image from the same viewpoint that are input to the 3D data generation device 200. The metallic estimation unit 213 may also perform metallic estimation using the cross-polarized image and the parallel-polarized image and generate a metallic map corresponding to the parallel-polarized image. In this case, the metallic estimation unit 213 may estimate that the subject (the surface material of the 3D object) is metallic if the pixel values of the parallel-polarized image for the processing target region are sufficiently larger than those of the cross-polarized image. The metallic estimation unit 213 may also supply the generated metallic map to the resolution enhancement unit 214.
[0073] The resolution increasing unit 214 performs processing related to increasing the resolution of material information. For example, the resolution increasing unit 214 may acquire an albedo map supplied from the gain correction unit 211. The resolution increasing unit 214 may also acquire a roughness map supplied from the roughness estimation unit 212. The resolution increasing unit 214 may also increase the resolution of the roughness map using the albedo map as a guide signal. For example, the resolution increasing unit 214 may increase the resolution of the roughness map using the above-described equations (1) and (2). The resolution increasing unit 214 may also supply the generated high-resolution roughness map to the roughness integration unit 216.
[0074] The resolution increasing unit 214 may also acquire a metallic map supplied from the metallic estimation unit 213. The resolution increasing unit 214 may also increase the resolution of the metallic map using the albedo map as a guide signal. For example, the resolution increasing unit 214 may increase the resolution of the metallic map using the above-described formulas (1) and (2). The resolution increasing unit 214 may also supply the generated high-resolution metallic map to the metallic integration unit 217.
[0075] The albedo integration unit 215 performs processing related to the integration of albedo maps from different viewpoints. For example, the albedo integration unit 215 may acquire albedo maps supplied from the gain correction unit 211. The albedo integration unit 215 may acquire albedo maps for multiple different viewpoints (i.e., multiple albedo maps corresponding to parallel polarization images from different viewpoints). The albedo integration unit 215 may also acquire overlap determination results between cross-polarized images (i.e., between albedo maps) supplied from the corresponding point detection unit 221. The albedo integration unit 215 may integrate multiple albedo maps from different viewpoints based on the overlap determination results. In other words, the albedo integration unit 215 may integrate albedo maps so as to combine areas determined to be overlapping areas (portions of 3D objects corresponding to the same positions). The albedo integration unit 215 may also acquire UV information supplied from the UV placement determination unit 224. This UV information indicates UV coordinates where geometry is placed on a UV plane. The albedo integration unit 215 may place the integrated albedo map on a UV plane based on the UV information. The albedo integration unit 215 may supply the albedo map (high resolution) placed on the UV plane to an external device outside the 3D data generation device 200 as material information.
[0076] The roughness integration unit 216 performs processing related to the integration of roughness maps from different viewpoints. For example, the roughness integration unit 216 may acquire roughness maps supplied from the resolution increasing unit 214. The roughness integration unit 216 may acquire roughness maps for multiple different viewpoints (i.e., multiple roughness maps corresponding to parallel polarization images from different viewpoints). The roughness integration unit 216 may also acquire overlap determination results between cross-polarized images (i.e., between roughness maps) supplied from the corresponding point detection unit 221. The roughness integration unit 216 may integrate multiple roughness maps from different viewpoints based on the overlap determination results. In other words, the roughness integration unit 216 may integrate roughness maps so as to combine areas determined to be overlapping areas (portions corresponding to the same positions on 3D objects). The roughness integration unit 216 may also acquire UV information supplied from the UV placement determination unit 224. The roughness integration unit 216 may arrange the integrated roughness map on a UV plane based on the UV information. The roughness integration unit 216 may supply the roughness map (high resolution) arranged on the UV plane to an external device outside the 3D data generation device 200 as material information.
[0077] The metallic integration unit 217 performs processing related to integration of metallic maps from different viewpoints. For example, the metallic integration unit 217 may acquire a metallic map supplied from the resolution increasing unit 214. The metallic integration unit 217 may acquire metallic maps for a plurality of different viewpoints (i.e., a plurality of metallic maps corresponding to parallel polarization images from different viewpoints). The metallic integration unit 217 may integrate the plurality of metallic maps from the different viewpoints. The metallic integration unit 217 may supply the integrated metallic map (high resolution) to an external device outside the 3D data generation device 200 as material information.
[0078] The corresponding point detection unit 221 performs processing related to the detection of corresponding points. For example, the corresponding point detection unit 221 may acquire cross-polarized images input to the 3D data generation device 200. The corresponding point detection unit 221 may acquire multiple cross-polarized images from different viewpoints. The corresponding point detection unit 221 may detect corresponding points (pixels corresponding to the same positions on a 3D object) between the cross-polarized images. Furthermore, the corresponding point detection unit 221 may detect overlapping areas (portions corresponding to the same positions on a 3D object) between the cross-polarized images based on the corresponding points. For example, the corresponding point detection unit 221 may determine whether a pixel to be processed is included in an overlapping area. The corresponding point detection unit 221 may provide information indicating the detected corresponding points, the cross-polarized images, etc. to the imaging orientation estimation unit 222. The corresponding point detection unit 221 may provide information indicating the detected corresponding points, the cross-polarized images, etc. to the geometry / normal estimation unit 223. The corresponding point detection unit 221 may also supply information indicating the detected corresponding points to the albedo integration unit 215. The corresponding point detection unit 221 may also supply information indicating the detected corresponding points to the roughness integration unit 216. Note that the information indicating the corresponding points may include information indicating the overlap region determination result.
[0079] The imaging attitude estimation unit 222 performs processing related to estimation of the imaging attitude. Note that this imaging attitude may include the position and orientation of the imaging unit in cross-polarized imaging or parallel-polarized imaging. For example, the imaging attitude estimation unit 222 may acquire information indicating the corresponding points detected by the corresponding point detection unit 221, a cross-polarized image, etc., supplied from the corresponding point detection unit 221. Furthermore, the imaging attitude estimation unit 222 may estimate the imaging attitude based on the information indicating the corresponding points, etc. Furthermore, the imaging attitude estimation unit 222 may supply information indicating the estimated imaging attitude, etc., to the geometry / normal estimation unit 223.
[0080] The geometry and normal estimation unit 223 performs processing related to estimation of the geometry of a 3D model and the surface normals of the 3D model. For example, the geometry and normal estimation unit 223 may acquire information indicating the corresponding points detected by the corresponding point detection unit 221, a cross-polarized image, and the like, supplied from the corresponding point detection unit 221. The geometry and normal estimation unit 223 may also acquire information indicating the imaging orientation, and the like, supplied from the imaging orientation estimation unit 222. The geometry and normal estimation unit 223 may perform geometry estimation based on the acquired information, thereby estimating the geometry. The geometry and normal estimation unit 223 may also perform normal estimation based on the acquired information, thereby estimating the normal. The geometry and normal estimation unit 223 may also supply information indicating the estimated geometry and normals, and the like, to the UV placement determination unit 224. The geometry and normal estimation unit 223 may also supply information indicating the estimated geometry and normals, and the like, to an external device outside the 3D data generation device 200. At this time, the geometry and normal estimation unit 223 may acquire UV information supplied from the UV placement determination unit 224. Then, the geometry and normal estimation unit 223 may place the estimated geometry and normal on the UV plane based on the UV information.
[0081] The UV placement determination unit 224 performs processing related to the placement of geometry and the like on a UV plane. For example, the UV placement determination unit 224 may acquire information supplied from the geometry / normal estimation unit 223 (e.g., geometry or normals, or both, estimated by the geometry / normal estimation unit 223). The UV placement determination unit 224 may determine a location (embedding position) at which to place the information (e.g., geometry or normals, or both) on the UV plane. The UV placement determination unit 224 may supply UV information and the like indicating the location to the geometry / normal estimation unit 223. The UV placement determination unit 224 may also supply the UV information and the like indicating the location to the albedo integration unit 215. The UV placement determination unit 224 may also supply the UV information and the like indicating the location to the roughness integration unit 216.
[0082] With the above-described configuration, the present technology described above in <3. Use of cross-polarized images and parallel-polarized images> can be applied to the 3D data generation device 200. Therefore, the 3D data generation device 200 can suppress a decrease in the quality of material information.
[0083] <Flow of 3D Data Generation Process> An example of the flow of the 3D data generation process executed by the 3D data generation device 200 will be described with reference to the flowchart of FIG.
[0084] When the 3D data generation process is started, in step S201, the 3D data generation device 200 acquires a cross-polarized image and a parallel-polarized image of the viewpoint to be processed.
[0085] In step S202, the gain correction unit 211 performs gain correction on the cross-polarized image to generate an albedo map.
[0086] In step S203, the roughness estimation unit 212 estimates roughness using the cross-polarized image and the parallel-polarized image.
[0087] In step S204, the metallic estimation unit 213 estimates metallicity using the cross polarization image and the parallel polarization image.
[0088] In step S205, the resolution increasing unit 214 increases the resolution of the roughness map generated in step S203 and the metallic map generated in step S204, using the albedo map generated in step S202 as a guide signal.
[0089] In step S206, the 3D data generation device 200 determines whether all images have been processed. That is, it determines whether cross-polarized images and parallel-polarized images have been processed for all viewpoints. If it is determined that an unprocessed image (that is, an unprocessed viewpoint) exists, the process returns to step S201, and the subsequent processes are repeated. That is, the processes of steps S201 to S206 are executed for each viewpoint. Then, if it is determined in step S206 that processing has been performed for all viewpoints, the process proceeds to step S207.
[0090] In step S207, the geometry generation unit 232 performs photogrammetry. For example, the corresponding point detection unit 221 detects corresponding points between the cross-polarized images (between viewpoints). The corresponding point detection unit 221 may also perform overlapping area determination. The imaging orientation estimation unit 222 estimates the imaging orientation based on the corresponding points, etc. The geometry and normal estimation unit 223 estimates the geometry and normals based on information such as the corresponding points and imaging orientation. The UV placement determination unit 224 determines the UV placement of the geometry and normals (placement locations of the geometry and normals on the UV plane) based on information such as the geometry and normals. The geometry and normal estimation unit 223 places the geometry and normals on the UV plane based on the UV placement (UV information).
[0091] In step S208, the albedo integration unit 215 integrates the viewpoints of the albedo map based on the overlapping area determination result, etc. The albedo integration unit 215 also places (embeds) the integrated albedo map in the UV plane based on the UV information.
[0092] In step S209, the roughness integration unit 216 integrates the viewpoints of the roughness map based on the overlapping area determination result, etc. The roughness integration unit 216 also places (embeds) the integrated roughness map in the UV plane based on the UV information.
[0093] In step S210, the metallic integration unit 217 integrates the respective viewpoints of the metallic map.
[0094] When the process of step S210 is completed, the 3D data generation process is completed.
[0095] By performing each process in this manner, the present technology described above in <3. Use of cross-polarized images and parallel-polarized images> can be applied to the 3D data generation device 200. Therefore, the 3D data generation device 200 can suppress a reduction in the quality of material information.
[0096] <5. Integration of Material Information> <Reduction in Roughness Map Quality> Conventional methods may result in a reduction in the quality of the roughness map. Roughness can be calculated from the particle spread function (PSF) of the flash region. However, in regions near the peak of the PSF, estimation accuracy deteriorates, potentially reducing the quality of the roughness map. For example, the flash region 302 formed in the roughness map 301 in FIG. 10 corresponds to the flash region (the area where flash light is specularly reflected) included in the captured image, and its roughness value may be affected by the specular reflection component. As a result, the reliability of the roughness value of this flash region 302 is reduced, potentially reducing the quality of the roughness map 301.
[0097] Furthermore, the roughness maps for each viewpoint are generated independently of each other. Therefore, when integrating multiple roughness maps from different viewpoints, there is a risk that the roughness value levels will differ between the integrated roughness maps. For example, in the roughness maps 311 to 314 of FIG. 10, the different roughness value levels are indicated by colors (white to black). Note that in the example of FIG. 10, for the sake of convenience, the roughness values of the roughness maps are shown as being uniform, but in reality, the roughness values within a roughness map may vary from pixel to pixel. The roughness value level represents the magnitude of the roughness value of the entire roughness map, such as the average roughness value.
[0098] In this way, when roughness maps having different roughness value levels are combined, there is a risk that a step in the roughness value (a difference in roughness value that is significantly more noticeable than in other parts) will be formed at the boundary portions of the roughness maps. For example, when the roughness maps 311 to 314 in Fig. 10 are combined, there is a risk that a step in the roughness value will be formed at the boundary portions of each roughness map.
[0099] For example, when there is an overlapping portion, such as the roughness maps 311 to 314 in Fig. 10, the roughness value of one of the roughness maps including the overlapping portion is selected and applied as the roughness value of the overlapping portion. Therefore, the roughness values of different roughness maps are adjacent to each other at the edge of the overlapping portion. Therefore, if the roughness value levels differ between the roughness maps to be integrated, there is a risk that the roughness value will change abruptly at such a boundary portion.
[0100] For example, even in a region where the roughness is essentially uniform, there may be a step in the roughness value at the boundary portion, which may reduce the quality of the roughness map after integration.
[0101] Such a reduction in the quality of the roughness map could result in subjective degradation of the texture.
[0102] <Method 2> Therefore, for example, as shown in the top row of the table in Fig. 11 , multiple roughness maps from the same viewpoint are integrated so as to suppress specular reflection components (Method 2). For example, multiple captured images from the same viewpoint but with different flash area positions are prepared, roughness maps corresponding to each of the captured images are generated, and the multiple roughness maps are integrated. In other words, multiple roughness maps from the same viewpoint but with different flash area (area affected by) positions are integrated.
[0103] 12 are roughness maps from the same viewpoint. In other words, the roughness maps 321 and 323 overlap each other over the entire image, and by integrating these maps, a roughness map 325 of the same size (from the same viewpoint) is generated.
[0104] Furthermore, the roughness map 321 includes a flash region 322 that is affected by flash light. Furthermore, the roughness map 323 includes a flash region 324 that is affected by flash light. However, as shown in Fig. 12 , the flash region 322 and the flash region 324 are formed in different positions.
[0105] In integrating roughness maps in this manner, for example, the roughness value of roughness map 323 is applied to flash region 322, and the roughness value of roughness map 321 is applied to flash region 324. In this way, it is possible to prevent roughness map 325 from including the roughness values of flash region 322 and flash region 324. In other words, multiple roughness maps for the same viewpoint can be integrated so as to suppress the influence of specular reflection components.
[0106] As described above, by integrating multiple roughness maps from the same viewpoint, the influence of specular reflection components in the integrated roughness map is suppressed, thereby suppressing a reduction in the quality of the integrated roughness map.
[0107] This method 2 may be applied in combination with the above-described method 1. For example, the roughness map generation unit of the above-described information processing device may integrate roughness maps corresponding to a plurality of parallel polarization images taken from the same viewpoint so as to suppress specular reflection components.
[0108] <Method 2-1> When Method 2 is applied, for example, as shown in the second row from the top of the table in FIG. 11, the reliability of the roughness value may be derived, and the roughness maps may be integrated based on that reliability (Method 2-1).
[0109] The reliability of the roughness value in the roughness map is reduced by, for example, the influence of specular reflection components. The degree of reduction in reliability depends on the magnitude of the influence of the specular reflection components. In other words, the reliability of the roughness value in the roughness map is reduced in proportion to the magnitude of the specular reflection components included in the pixel values of the captured image.
[0110] Therefore, by deriving the reliability of the roughness values and integrating the roughness maps based on the reliability, it is possible to generate a roughness map that is composed of more reliable roughness values, thereby suppressing a decrease in the quality of the integrated roughness map.
[0111] This method 2-1 may be applied in combination with the above-described method 1 (or one or more of methods 1-1 to 1-5). For example, the roughness map generation unit of the above-described information processing device may derive the reliability of the roughness and integrate multiple roughness maps based on the reliability.
[0112] <Method 2-1-1> When Method 2-1 is applied, for example, as shown in the third row from the top of the table in FIG. 11, a flash area may be detected and the reliability may be derived based on the detected flash area (Method 2-1-1).
[0113] For example, suppose a captured image contains a flash region 331 as shown in Fig. 13. In this flash region 331, the diffuse reflection component is uniform, but the brightness increases toward the center. In other words, the specular reflection component increases toward the center of the flash region 331. In such a case, the reliability of the roughness value of the region of the roughness map corresponding to the flash region 331 decreases significantly toward the center of the region, as indicated by curve 333 in graph 332.
[0114] In this way, the reliability of the roughness map is significantly reduced in flash regions. Therefore, by detecting these flash regions and deriving the reliability of the roughness map based on the flash regions, it is possible to more easily detect regions where the reliability of the roughness value is reduced.
[0115] For example, suppose that the reliability distribution of the roughness map 321 in Figure 12 and the reliability distribution of the roughness map 323 are superimposed to obtain a graph 334 as shown in Figure 13. A solid curve 335 shows the reliability distribution of the roughness map 321, with the reliability decreasing at the position of the flash region 322. A dotted curve 336 shows the reliability distribution of the roughness map 323, with the reliability decreasing at the position of the flash region 324. In this way, the position where the reliability decreases can be easily detected based on the position of the flash region.
[0116] Therefore, in such flash regions, it is sufficient to apply the roughness values of other roughness maps whose reliability has not decreased. By doing so, it is possible to suppress the decrease in the reliability of the roughness values in the integrated roughness map, and it is also possible to suppress the decrease in the quality of the roughness map.
[0117] This method 2-1-1 may be applied in combination with the above-described method 1 (or one or more of methods 1-1 to 1-5). In this case, a parallel polarization image may be used to detect flash areas. Alternatively, instead of the parallel polarization image, a difference image obtained by subtracting the pixel values of the cross polarization image from the pixel values of the parallel polarization image may be used to detect flash areas. For example, the roughness map generation unit of the above-described information processing device may generate a difference image between the parallel polarization image and the cross polarization image, use the difference image to detect flash areas included in the roughness map, and derive the reliability of the roughness value based on the detected flash areas.
[0118] As described above, the parallel polarization image contains both diffuse and specular reflection components. The cross polarization image contains primarily diffuse reflection components. Therefore, the difference image contains primarily specular reflection components.
[0119] Therefore, by detecting a flash area containing a large amount of specular reflection components using a differential image, it is possible to detect the flash area more easily and accurately than when using a parallel polarization image. Therefore, it is possible to more accurately suppress a decrease in the reliability of the roughness value in the integrated roughness map, and it is possible to suppress a decrease in the quality of the roughness map.
[0120] <Method 2-1-2> When Method 2-1 is applied, for example, as shown in the fourth row from the top of the table in Figure 11, the roughness values of each roughness map to be integrated may be blended in a ratio according to their reliability (Method 2-1-1).
[0121] In this way, by blending the roughness values of multiple roughness maps and deriving the roughness value of the integrated roughness map, the roughness maps can be integrated taking into account the roughness values of each roughness map. For example, if the area does not correspond to a flash area, the roughness values of all of the roughness maps to be integrated are highly reliable. In such a case, by blending the roughness values of each roughness map, a value that reflects the roughness values of each roughness map can be derived as the roughness value of the integrated roughness map. In this way, the change in roughness value of the integrated roughness map can be made more natural (smoother), and degradation of the quality of the roughness map can be suppressed.
[0122] This method 2-1-2 may be applied in combination with the above-mentioned method 1 (or one or more of methods 1-1 to 1-5). For example, the roughness map generation unit of the above-mentioned information processing device may blend multiple roughness maps at a ratio according to the reliability. This method 2-1-2 may also be applied in combination with the above-mentioned method 2-1-1.
[0123] <Method 2-2> When Method 2 is applied, for example, as shown in the fifth row from the top of the table in FIG. 11, the levels may be matched by adjusting the gain of the roughness values of each roughness map to be integrated (Method 2-2).
[0124] By integrating the roughness maps after level matching in this way, it is possible to suppress the influence of differences in roughness value levels between the roughness maps, and therefore to suppress a decrease in the quality of the integrated roughness maps.
[0125] This method 2-2 may be applied in combination with the above-described method 1 (or one or more of methods 1-1 to 1-5). For example, the roughness map generation unit of the above-described information processing device may combine and integrate the respective levels of the roughness maps. This method 2-2 may also be applied in combination with the above-described method 2-1 (or one or more of methods 2-1-1 and 2-1-2).
[0126] <Method 2-2-1> When Method 2-2 is applied, for example, as shown in the sixth row from the top of the table in FIG. 11, the gain may be set using the average of the high reliability region (Method 2-2-1).
[0127] That is, in each of the roughness maps to be integrated, the average value of roughness in areas other than areas where reliability is reduced, such as flash areas, may be derived, and a gain to be multiplied by each roughness map may be set so that the average values of the roughness maps approach each other (so that the difference between them is reduced). By doing so, it is possible to align the levels of each roughness map while suppressing the influence of flash areas where reliability is reduced. Therefore, it is possible to suppress a reduction in the quality of the roughness map after integration.
[0128] Note that this method 2-2-1 may be applied in combination with the above-mentioned method 1 (or one or more of methods 1-1 to 1-5). For example, the roughness map generation unit of the above-mentioned information processing device may derive the reliability of the roughness and set a gain for adjusting the level of the roughness map based on the reliability. Also, this method 2-2-1 may be applied in combination with the above-mentioned method 2-1 (or one or more of methods 2-1-1 and 2-1-2).
[0129] In addition, the roughness map generation unit may generate a difference image between the parallel polarization image and the cross polarization image, use the difference image to detect flash areas included in the roughness map, and derive reliability based on the detected flash areas.
[0130] The roughness map generating section may also set the gain based on the average roughness value in an area where the roughness map has a high reliability.
[0131] <Method 3> Furthermore, for example, as shown in the seventh row from the top of the table in FIG. 11, a plurality of roughness maps from different viewpoints are integrated by adjusting the gain and matching the levels (Method 3).
[0132] For example, suppose that the roughness value levels of multiple roughness maps from different viewpoints are different, as in the roughness maps 341-1 to 344-1 shown in Figure 14. When integrating such roughness maps 341-1 to 344-1, the levels of the roughness maps are adjusted by, for example, adjusting the gain by which the roughness value of each roughness map is multiplied, and then the roughness maps are integrated. In this way, the roughness maps can be integrated while suppressing the level differences between them, thereby suppressing the occurrence of roughness value steps in the integrated roughness map. In other words, it is possible to suppress a reduction in the quality of the integrated roughness map.
[0133] Note that Method 3 may be applied in combination with the above-described Method 1 (or one or more of Methods 1-1 to 1-5). For example, the above-described information processing device may include a roughness integration unit that integrates multiple roughness maps corresponding to parallel polarization images from different viewpoints by matching their levels. With this configuration, the information processing device can suppress a decrease in the quality of the integrated roughness map. Also, Method 3 may be applied in combination with the above-described Method 2 (or one or more of Methods 2-1 to 2-2-1).
[0134] <Method 3-1> When Method 3 is applied, for example, as shown in the eighth row from the top of the table in Figure 11, the gain of the roughness map to be processed may be adjusted so that the level of the roughness map to be processed matches that of the processed roughness map (Method 3-1).
[0135] For example, when integrating multiple roughness maps from different viewpoints, the roughness maps may be processed one by one (level adjustment). For example, as shown in FIG. 14 , when integrating roughness maps 341-1 to 344-1 by adjusting their levels, first, as indicated by the bold frame, roughness map 342-1 is set as the roughness map to be processed, and processing is performed to adjust its level to that of roughness map 341-1. Next, roughness map 343-2 is set as the roughness map to be processed, and processing is performed to adjust its level to that of roughness map 342-2 (roughness map 342-1 whose level has been adjusted to that of roughness map 341-1). Next, roughness map 344-3 is set as the roughness map to be processed, and processing is performed to adjust its level to that of roughness map 343-3 (roughness map 343-2 whose level has been adjusted to that of roughness map 342-2).
[0136] By doing this, the levels of the roughness maps to be integrated are made uniform, as in the case of roughness maps 341-4 to 344-4. By performing processing in this manner, multiple roughness maps can be integrated by repeating the same processing. In other words, the integration process can be made easier.
[0137] Note that Method 3-1 may be applied in combination with the above-described Method 1 (or one or more of Methods 1-1 to 1-5). For example, in the above-described information processing device, the roughness integration unit may set the gain of the roughness map to be processed so that the level of the roughness map to be processed matches that of the processed roughness map. Also, Method 3-1 may be applied in combination with the above-described Method 2 (or one or more of Methods 2-1 to 2-2-1).
[0138] <Method 3-1-1> When Method 3-1 is applied, for example, as shown in the ninth row from the top of the table in Figure 11, the gain may be set using the average roughness values in the overlapping area between the current roughness map to be processed and the previous roughness map to be processed (Method 3-1-1).
[0139] That is, the gain is set based on the average roughness value of the overlapping region so as to match the levels between the roughness maps. By doing so, the roughness value of the overlapping region can be made more natural, and a reduction in the quality of the combined roughness map can be suppressed.
[0140] Note that Method 3-1-1 may be applied in combination with the above-mentioned Method 1 (or one or more of Methods 1-1 to 1-5). For example, in the above-mentioned information processing device, the roughness integration unit may set the gain of the roughness map to be processed by using the average roughness of the overlapping area between the current roughness map to be processed and the previous roughness map to be processed. Also, Method 3-1-1 may be applied in combination with the above-mentioned Method 2 (or one or more of Methods 2-1 to 2-2-1).
[0141] <Method 3-1-2> When Method 3-1 is applied, for example, as shown in the bottom row of the table in FIG. 11, the gain by which the roughness value of the overlapping region is multiplied may be adjusted (Method 3-1-2).
[0142] In other words, the target of level adjustment may be an overlapping region of the roughness maps. By limiting the target region of level adjustment to a partial region in this way, it is possible to suppress a decrease in the quality of the integrated roughness map while suppressing an increase in the processing load.
[0143] Note that this method 3-1-2 may be applied in combination with the above-mentioned method 1 (or one or more of methods 1-1 to 1-5). For example, in the above-mentioned information processing device, the roughness integration unit may set a gain for an overlapping region of the roughness map to be processed with the processed roughness map. Also, this method 3-1-2 may be applied in combination with the above-mentioned method 2 (or one or more of methods 2-1 to 2-2-1). Also, this method 3-1-2 may be applied in combination with the above-mentioned method 3-1-1.
[0144] Note that any region may be the target of level adjustment. For example, the overlap region and some of the surrounding regions may be the target of level adjustment. In other words, the roughness values of the regions may be multiplied by a gain. Alternatively, the entire roughness map may be the target of level adjustment. In other words, the roughness values of the entire roughness map may be multiplied by a gain.
[0145] 6. Second Embodiment Roughness Estimation Unit When the present technology described above in 5. Integration of Material Information is applied, the configuration of the 3D data generation device 200 is the same as that in FIG.
[0146] As described above, the 3D data generation device 200 may apply the present technology described in <5. Integration of Material Information>. That is, in this case, the roughness estimation unit 212 may apply the above-described method 2. Also, the roughness estimation unit 212 may apply the above-described method 2-1. Also, the roughness estimation unit 212 may apply the above-described method 2-1-1. Also, the roughness estimation unit 212 may apply the above-described method 2-1-2. Also, the roughness estimation unit 212 may apply the above-described method 2-2. Also, the roughness estimation unit 212 may apply the above-described method 2-2-1. Also, the roughness estimation unit 212 may apply any combination of two or more of these methods. FIG. 15 is a block diagram showing an example of the main configuration of the roughness estimation unit 212 when these methods are applied.
[0147] As shown in FIG. 15, the roughness estimation unit 212 has a difference image generation unit 411, a flash area detection unit 412, a reliability derivation unit 413, a roughness estimation unit 414, a gain setting unit 415, a level adjustment unit 416, a blending ratio setting unit 417, and a blending unit 418.
[0148] A pair of a cross polarized image C1-1 and a parallel polarized image P1-1 is input to the roughness estimation unit 212. In addition, a pair of a cross polarized image C1-2 and a parallel polarized image P1-2 is input to the roughness estimation unit 212 at different times. All of these captured images are captured from the same viewpoint. In addition, flash areas are formed in different positions in the cross polarized image C1-1 and the cross polarized image C1-2 (the parallel polarized image P1-1 and the parallel polarized image P1-2).
[0149] The difference image generation unit 411 generates a difference image by subtracting the cross-polarized image from the parallel polarization image. The difference image generation unit 411 supplies the generated difference image to the flash area detection unit 412. For example, the difference image generation unit 411 generates a difference image D1-1 using the parallel polarization image P1-1 and the cross polarization image C1-1 and supplies this to the flash area detection unit 412. The difference image generation unit 411 also generates a difference image D1-2 using the parallel polarization image P1-2 and the cross polarization image C1-2 and supplies this to the flash area detection unit 412.
[0150] The flash area detection unit 412 detects a flash area included in the parallel polarization image using the supplied difference image. The flash area detection unit 412 supplies information indicating the detected flash area together with the difference image to the reliability derivation unit 413. For example, the flash area detection unit 412 detects a flash area using the difference image D1-1, and supplies information indicating the detected flash area together with the difference image D1-1 to the reliability derivation unit 413. Furthermore, the flash area detection unit 412 detects a flash area using the difference image D1-2, and supplies information indicating the detected flash area together with the difference image D1-2 to the reliability derivation unit 413.
[0151] The reliability derivation unit 413 derives the reliability of the roughness value of the roughness map based on the supplied information, i.e., based on the detected flash area. The reliability derivation unit 413 derives the reliability for each pixel and generates a reliability map that is distribution information of the reliability. The reliability derivation unit 413 supplies the generated reliability map to the gain setting unit 415 and the blending ratio setting unit 417. For example, the reliability derivation unit 413 generates a reliability map Re1-1 based on the difference image D1-1 and its flash area, and supplies it to the gain setting unit 415 and the blending ratio setting unit 417. The reliability derivation unit 413 also generates a reliability map Re1-2 based on the difference image D1-2 and its flash area, and supplies it to the gain setting unit 415 and the blending ratio setting unit 417.
[0152] The roughness estimation unit 414 estimates roughness based on the parallel polarization image and the cross polarization image, and generates a roughness map. The roughness estimation unit 414 supplies the generated roughness map to the gain setting unit 415 and the level adjustment unit 416. For example, the roughness estimation unit 414 estimates roughness based on the parallel polarization image P1-1 and the cross polarization image C1-1, generates a roughness map Ro1-1, and supplies it to the gain setting unit 415 and the level adjustment unit 416. The roughness estimation unit 414 also estimates roughness based on the parallel polarization image P1-2 and the cross polarization image C1-2, generates a roughness map Ro1-2, and supplies it to the gain setting unit 415 and the level adjustment unit 416.
[0153] The gain setting unit 415 sets a gain to be multiplied by the roughness map (roughness value) based on the supplied roughness map and reliability map. The gain setting unit 415 supplies the set gain to the level adjustment unit 416. For example, the gain setting unit 415 sets a gain G1-1 for the roughness map Ro1-1 based on the roughness map Ro1-1 and the reliability map Re1-1, and supplies the gain G1-1 to the level adjustment unit 416. Furthermore, the gain setting unit 415 sets a gain G1-2 for the roughness map Ro1-2 based on the roughness map Ro1-2 and the reliability map Re1-2, and supplies the gain G1-2 to the level adjustment unit 416.
[0154] The level adjustment unit 416 multiplies the supplied roughness map (the roughness value thereof) by a gain to perform level adjustment. The level adjustment unit 416 supplies the level-adjusted roughness map to the blending unit 418. For example, the level adjustment unit 416 multiplies the roughness map Ro1-1 by a gain G1-1 to generate a level-adjusted roughness map Ro'1-1 and supplies it to the blending unit 418. The level adjustment unit 416 also multiplies the roughness map Ro1-2 by a gain G1-2 to generate a level-adjusted roughness map Ro'1-2 and supplies it to the blending unit 418.
[0155] The blending ratio setting unit 417 sets a blending ratio, which is a ratio at which roughness values of the roughness maps to be integrated are blended, based on the supplied reliability maps. The blending ratio setting unit 417 supplies the set blending ratio to the blending unit 418. For example, the blending ratio setting unit 417 sets a blending ratio B1 based on the reliability maps Re1-1 and Re1-2, and supplies the blending ratio B1 to the blending unit 418.
[0156] The blending unit 418 blends and integrates the supplied roughness maps at the supplied blending ratio. The blending unit 418 supplies the integrated roughness map to the resolution increasing unit 214 (FIG. 8). For example, the blending unit 418 blends the roughness map Ro'1-1 and the roughness map Ro'1-2 at the blending ratio B1 (and (1-B1)), generates an integrated roughness map Ro'1, and supplies it to the resolution increasing unit 214 (FIG. 8).
[0157] With this configuration, the roughness estimation unit 212 can integrate multiple roughness maps from the same viewpoint so as to suppress the influence of specular reflection components, thereby enabling the 3D data generation device 200 to suppress degradation in the quality of the roughness maps.
[0158] <Roughness Integration Unit> As described above, the 3D data generation device 200 may apply the present technology described above in <5. Integration of Material Information>. That is, in this case, the roughness integration unit 216 may apply the above-described method 3. Alternatively, the roughness integration unit 216 may apply the above-described method 3-1. Alternatively, the roughness integration unit 216 may apply the above-described method 3-1-1. Alternatively, the roughness integration unit 216 may apply the above-described method 3-1-2. Alternatively, the roughness integration unit 216 may apply any combination of these methods.
[0159] FIG. 16 is a block diagram showing an example of the main configuration of the roughness integration unit 216 when these methods are applied.
[0160] As shown in FIG. 16, the roughness integrating unit 216 includes a gain setting unit 431 , a level adjusting unit 432 , and an integrating unit 433 .
[0161] The gain setting unit 431 acquires an overlap area map, which is a map indicating the overlap area, supplied from the corresponding point detection unit 221 ( FIG. 8 ). The gain setting unit 431 also acquires a level-adjusted roughness map. The gain setting unit 431 also acquires a processing target roughness map (a roughness map whose level is to be adjusted). For the overlap area indicated by the overlap area map, the gain setting unit 431 sets a gain (G1) for the processing target roughness map (Ro3) so that the level of the roughness value of the processing target roughness map (Ro3) matches the level-adjusted roughness map (Ro′2). The gain setting unit 431 supplies the set gain to the level adjustment unit 432.
[0162] The level adjustment unit 432 adjusts the level of the roughness map to be processed by multiplying the roughness value by the supplied gain. The level adjustment unit 432 supplies the level-adjusted roughness map to the integration unit 433.
[0163] The integration unit 433 integrates the level-adjusted roughness maps supplied sequentially in this manner by arranging them on a UV plane. At this time, the integration unit 433 arranges the level-adjusted roughness maps on the UV plane based on the UV map, xyz coordinates, etc. supplied from the UV arrangement determination unit 224 ( FIG. 8 ). The integration unit 433 outputs the integrated roughness map, a UV roughness map Ro_uv, to the outside of the 3D data generation device 200.
[0164] With this configuration, the roughness integration unit 216 can integrate the roughness maps while suppressing the level differences between them. Therefore, the roughness integration unit 216 can suppress the occurrence of steps in the roughness values in the integrated roughness map. In other words, the 3D data generation device 200 can suppress a decrease in the quality of the roughness map.
[0165] <Flow of 3D Data Generation Process> An example of the flow of the 3D data generation process executed by the 3D data generation device 200 when the present technology described above in <5. Integration of Material Information> is applied will be described with reference to the flowchart in Figure 17.
[0166] In this case, steps S401 to S404 are executed in the same manner as steps S201 to S204 in Fig. 9. However, in step S401, images captured from the same viewpoint (cross-polarized images and parallel-polarized images) are first preferentially selected. In addition, in step S403, a roughness estimation process, which will be described later, is executed.
[0167] In step S405, the 3D data generation device 200 determines whether all images of the same viewpoint have been processed. If an unprocessed image exists, the process returns to step S401, and the subsequent processes are repeated. Then, if it is determined in step S405 that all images of the same viewpoint have been processed, the process proceeds to step S406.
[0168] The processes of steps S406 to S411 are executed in the same manner as the processes of steps S205 to S210 in Fig. 9. However, in step S410, a roughness integration process, which will be described later, is executed.
[0169] When the process of step S411 ends, the 3D data generation process ends.
[0170] <Flow of Roughness Estimation Processing> An example of the flow of the roughness estimation processing executed in step S403 of FIG. 17 will be described with reference to the flowchart of FIG.
[0171] When the roughness estimation process starts, in step S431, the difference image generating unit 411 generates a difference image between the cross-polarized image and the parallel-polarized image.
[0172] In step S432, the flash area detection unit 412 detects the flash area using the difference image.
[0173] In step S433, the reliability deriving unit 413 derives the reliability of the roughness value based on the flash region detection result, and generates a reliability map.
[0174] In step S434, the roughness estimation unit 414 estimates roughness using the cross-polarized image and the parallel-polarized image, and generates a roughness map.
[0175] In step S435, the gain setting unit 415 sets a gain based on the reliability map generated in step S433 and the roughness map generated in step S434.
[0176] In step S436, the level adjustment unit 416 adjusts the level of the roughness map using the gain.
[0177] In step S437, the blending ratio setting unit 417 sets the blending ratio based on the reliability map generated in step S433.
[0178] In step S438, the blending unit 418 blends the roughness maps of the same viewpoint using the blending ratio.
[0179] When the process of step S438 ends, the process returns to FIG.
[0180] <Flow of Roughness Integration Processing> An example of the flow of the roughness integration processing executed in step S410 of FIG. 17 will be described with reference to the flowchart of FIG.
[0181] When the roughness integration process starts, in step S451, the roughness integration unit 216 selects a roughness map to be processed.
[0182] In step S452, the gain setting unit 431 determines whether or not an overlapping area exists. If it is determined that an overlapping area exists, the process proceeds to step S453.
[0183] In step S453, the gain setting unit 431 sets a gain based on the overlapping area between the roughness map to be processed and the processed roughness map.
[0184] In step S454, the level adjustment unit 432 adjusts the level of the roughness map to be processed using the gain. When the processing of step S454 ends, the process proceeds to step S455. Also, if it is determined in step S451 that no overlapping area exists, the process proceeds to step S455.
[0185] In step S455, the integration unit 433 integrates the roughness map to be processed with the processed roughness map.
[0186] In step S456, the roughness integration unit 216 determines whether all roughness maps have been processed. If an unprocessed roughness map exists, the process returns to step S451, and the subsequent processes are repeated. Then, if it is determined in step S456 that all roughness maps have been processed, the roughness integration process ends, and the process returns to FIG. 17 .
[0187] By performing each process as described above, the 3D data generation device 200 can suppress a decrease in the quality of the roughness map.
[0188] <7. Third Embodiment> <Imaging Device> Note that the present technology described above may be applied to any configuration. For example, the present technology may be applied to an imaging device that captures an image of a subject. FIG. 20 is a block diagram showing a main configuration example of an imaging device that is an embodiment of an information processing device to which the present technology described above is applied. The imaging device 500 shown in FIG. 20 can perform cross-polarized imaging and parallel-polarized imaging, and the present technology described in <3. Use of Cross-Polarized Images and Parallel-Polarized Images> may be applied. For example, the imaging device 500 may apply the above-described method 1. The imaging device 500 may also apply the above-described method 1-1. The imaging device 500 may also apply the above-described method 1-2. The imaging device 500 may also apply the above-described method 1-3. The imaging device 500 may also apply the above-described method 1-4. The imaging device 500 may also apply the above-described method 1-5. The imaging device 500 may also apply any combination of these methods.
[0189] The imaging device 500 may also apply the present technology described above in <5. Integration of Material Information>. That is, in this case, the imaging device 500 may also apply the above-described method 2. The imaging device 500 may also apply the above-described method 2-1. The imaging device 500 may also apply the above-described method 2-1-1. The imaging device 500 may also apply the above-described method 2-1-2. The imaging device 500 may also apply the above-described method 2-2. The imaging device 500 may also apply the above-described method 2-2-1. The imaging device 500 may also apply the above-described method 3. The imaging device 500 may also apply the above-described method 3-1. The imaging device 500 may also apply the above-described method 3-1-1. The imaging device 500 may also apply the above-described method 3-1-2. The imaging device 500 may also apply any combination of these methods. Of course, the imaging device 500 may apply a combination of any two or more of Methods 1 to 3 (including the methods belonging to these methods).
[0190] As shown in FIG. 20, the imaging device 500 has a control unit 511, an imaging control unit 512, a light emitting unit 513, an imaging unit 514, a 3D data generation unit 515, a memory unit 516, a display unit 517, an operation unit 518, and an output unit 519.
[0191] The control unit 511 controls each processing unit of the imaging device 500. The imaging control unit 512 controls the light emitting unit 513 and the imaging unit 514 to perform cross polarization imaging and parallel polarization imaging. The light emitting unit 513 emits light and irradiates the 3D object with the irradiated light. The light emitting unit 513 has a configuration for polarizing the irradiated light, such as a polarizing filter, and polarizes the irradiated light in a first direction. The imaging unit 514 receives light from the subject and generates a captured image. The imaging unit 514 has a configuration for polarizing the light from the subject, such as a polarizing filter, and polarizes the light in a second direction. In the case of cross polarization imaging, the second direction is perpendicular to the first direction. In the case of parallel polarization imaging, the second direction is parallel to the first direction (i.e., the same direction as the first direction).
[0192] The 3D data generation unit 515 generates 3D data using the cross-polarized image and the parallel-polarized image supplied from the imaging unit 514. The 3D data generation unit 515 has a configuration similar to that of the 3D data generation device 200 in Fig. 8 and performs similar processing. That is, the 3D data generation unit 515 may perform processing by applying the present technology described above.
[0193] The storage unit 516 stores the captured images supplied from the imaging unit 514 and the 3D data supplied from the 3D data generation unit 515. The display unit 517 displays the captured images supplied from the imaging unit 514 and images read from the storage unit 516. The operation unit 518 accepts input operations by a user or the like and supplies the information to the control unit 511. The control unit 511 may control each processing unit based on the information.
[0194] The output unit 519 supplies the captured image and 3D data read from the storage unit 516 to an external device (for example, another information processing device) outside the imaging device 500 .
[0195] By having such a configuration, the imaging device 500 can suppress a decrease in the quality of the material information, similar to the case of the 3D data generation device 200.
[0196] <Image Capture Process Flow 1> An example of the flow of image capture processes executed by the image capture device 500 will be described with reference to the flowchart of FIG.
[0197] When the imaging process starts, in step S501, the imaging control unit 512 controls the light emitting unit 513 and the imaging unit 514 to perform cross-polarized imaging and generate a cross-polarized image.
[0198] In step S502, the imaging control unit 512 controls the light emitting unit 513 and the imaging unit 514 to perform parallel polarization imaging and generate a parallel polarization image.
[0199] The processes of steps S503 to S506 are executed in the same manner as the processes of steps S202 to S205 in FIG.
[0200] In step S507, the control unit 511 determines whether or not to end image capture. If it is determined not to end image capture, the process returns to step S501, and the subsequent processes are repeated. If it is determined in step S507 to end image capture, the process proceeds to step S508.
[0201] The processes of steps S508 to S511 are executed in the same manner as the processes of steps S207 to S210 in FIG.
[0202] When the process of step S511 ends, the imaging process ends.
[0203] By performing each process as described above, the imaging device 500 can suppress a decrease in the quality of the material information, similar to the case of the 3D data generation device 200.
[0204] <Image Capture Process Flow 2> Another example of the image capture process flow executed by the image capture device 500 will be described with reference to the flowchart of FIG.
[0205] When the image capturing process is started, steps S551 to S555 are executed in the same manner as steps S501 to S505 in Fig. 21. However, in step S551, images captured from the same viewpoint (cross-polarized images and parallel-polarized images) are first preferentially selected. In addition, in step S554, the roughness estimation process described with reference to Fig. 18 is executed.
[0206] In step S556, the 3D data generation device 200 determines whether or not to end imaging from the same viewpoint. If imaging is not to be ended, the process returns to step S551, and the subsequent processes are repeated. Then, if it is determined in step S556 that imaging from the same viewpoint is to be ended, the process proceeds to step S557.
[0207] The processes of steps S557 to S562 are executed in the same manner as the processes of steps S506 to S511 in Fig. 21. However, in step S561, the roughness integration process described with reference to Fig. 19 is executed.
[0208] When the process of step S562 ends, the imaging process ends.
[0209] By performing each process as described above, the imaging device 500 can suppress a decrease in the quality of the material information, similar to the case of the 3D data generation device 200.
[0210] 8. Supplementary Notes Computer The above-described series of processes can be executed by hardware or software. When the series of processes are executed by software, the programs that make up the software are installed on a computer. Here, the term computer includes computers built into dedicated hardware, and general-purpose personal computers, for example, that can execute various functions by installing various programs.
[0211] FIG. 23 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.
[0212] In a computer 900 shown in FIG. 23, a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903 are interconnected via a bus 904.
[0213] An input / output interface 910 is also connected to the bus 904. To the input / output interface 910, an input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915 are connected.
[0214] The input unit 911 includes, for example, a keyboard, a mouse, a microphone, a touch panel, and an input terminal. The output unit 912 includes, for example, a display, a speaker, and an output terminal. The storage unit 913 includes, for example, a hard disk, a RAM disk, and a non-volatile memory. The communication unit 914 includes, for example, a network interface. The drive 915 drives removable media 921 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.
[0215] In a computer configured as described above, the CPU 901 performs the above-described series of processes by, for example, loading a program stored in the storage unit 913 into the RAM 903 via the input / output interface 910 and the bus 904 and executing the program. The RAM 903 also stores data necessary for the CPU 901 to execute various processes as appropriate.
[0216] The program executed by the computer can be applied by recording it on, for example, a removable medium 921 such as a package medium. In this case, the program can be installed in the storage unit 913 via the input / output interface 910 by inserting the removable medium 921 into the drive 915.
[0217] This program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, digital satellite broadcasting, etc. In this case, the program can be received by the communication unit 914 and installed in the storage unit 913.
[0218] Alternatively, this program can be installed in advance in the ROM 902 or the storage unit 913 .
[0219] <Application of the Present Technology> The present technology can be applied to any configuration. For example, the present technology can be applied to various electronic devices.
[0220] Furthermore, for example, the present technology can also be implemented as part of an apparatus, such as a processor (e.g., a video processor) as a system LSI (Large Scale Integration), a module using multiple processors (e.g., a video module), a unit using multiple modules (e.g., a video unit), or a set in which other functions are added to a unit (e.g., a video set).
[0221] Furthermore, for example, the present technology can also be applied to a network system configured with multiple devices. For example, the present technology may be implemented as cloud computing in which multiple devices share and collaborate on processing via a network. For example, the present technology may be implemented in a cloud service that provides image (video)-related services to any terminal, such as a computer, an AV (Audio Visual) device, a portable information processing terminal, or an IoT (Internet of Things) device.
[0222] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are housed in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.
[0223] <Fields and uses to which this technology can be applied> Systems, devices, processing units, etc. to which this technology is applied can be used in any field, for example, transportation, medical care, crime prevention, agriculture, livestock farming, mining, beauty, factories, home appliances, weather, nature monitoring, etc. In addition, the uses thereof are also arbitrary.
[0224] <Others> The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present technology.
[0225] For example, a configuration described as one device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, configurations described above as multiple devices (or processing units) may be combined and configured as one device (or processing unit). Of course, configurations other than those described above may be added to the configuration of each device (or each processing unit). Furthermore, as long as the configuration and operation of the entire system are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).
[0226] Furthermore, for example, the above-described program may be executed in any device, as long as the device has the necessary functions (functional blocks, etc.) and is able to obtain the necessary information.
[0227] Also, for example, each step of a single flowchart may be executed by a single device, or may be shared and executed by multiple devices. Furthermore, when a single step includes multiple processes, the multiple processes may be executed by a single device, or may be shared and executed by multiple devices. In other words, multiple processes included in a single step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as a single step.
[0228] For example, the steps of a program executed by a computer may be executed in chronological order in the order described herein, or may be executed in parallel or individually at the required timing, such as when a call is made. In other words, as long as no contradiction occurs, the steps may be executed in an order different from the order described above. Furthermore, the steps of this program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.
[0229] Furthermore, for example, the multiple technical elements included in the present technology can be implemented independently and individually, as long as no contradiction occurs. Of course, any multiple technical elements among them can also be implemented in combination. For example, part or all of the present technology described in any embodiment can be implemented in combination with part or all of the present technology described in another embodiment. Furthermore, part or all of any of the present technology described above can be implemented in combination with other technologies not described above.
[0230] Note that the present technology can also be configured as follows: (1) An information processing device including a roughness map generator that performs roughness estimation to estimate the roughness of a surface of a 3D object using a cross polarization image and a parallel polarization image, and generates a roughness map that is distribution information of the roughness corresponding to the parallel polarization image, wherein the cross polarization image is a captured image generated by cross polarization imaging, and the parallel polarization image is a captured image generated by parallel polarization imaging, and the captured image is generated by irradiating a subject with light from a light emitting unit and receiving the light from the subject by an imaging unit, wherein in the case of the cross polarization imaging, the light from the light emitting unit is polarized in a first polarization direction and the light from the subject is polarized in a second polarization direction perpendicular to the first polarization direction, and in the case of the parallel polarization imaging, the light from the light emitting unit is polarized in the first polarization direction and the light from the subject is polarized in the first polarization direction. (2) The information processing device according to (1), wherein the roughness map generation unit detects a flash region included in the parallel polarization image using a difference image between the parallel polarization image and the cross polarization image, analyzes the detected flash region, and performs the roughness estimation based on the analysis result of the flash region. (3) The information processing device according to (1) or (2), further comprising a resolution enhancement unit that increases the resolution of the roughness map using an albedo map as a guide signal. (4) The information processing device according to any of (1) to (3), further comprising a metallic map generation unit that performs metallic estimation using the cross polarization image and the parallel polarization image and generates a metallic map. (5) The information processing device according to (4), wherein the metallic map generation unit estimates that the subject is metallic if, for a processing target region, pixel values of the parallel polarization image are sufficiently larger than those of the cross polarization image. (6) The information processing device according to (4) or (5), further comprising a resolution enhancement unit that increases the resolution of the metallic map using an albedo map as a guide signal. (7) The information processing device according to any one of (1) to (6), further comprising: an albedo map generator that performs gain correction on the cross polarization image to generate an albedo map.(8) The information processing device according to any one of (1) to (7), wherein the roughness map generation unit integrates the roughness maps corresponding to the plurality of parallel polarization images of the same viewpoint so as to suppress specular reflection components. (9) The information processing device according to (8), wherein the roughness map generation unit derives a reliability of the roughness and integrates the plurality of roughness maps based on the reliability. (10) The information processing device according to (9), wherein the roughness map generation unit generates a difference image between the parallel polarization image and the cross polarization image, detects flash areas included in the roughness map using the difference image, and derives the reliability based on the detected flash areas. (11) The information processing device according to (9) or (10), wherein the roughness map generation unit blends the plurality of roughness maps at a ratio according to the reliability. (12) The information processing device according to any one of (8) to (11), wherein the roughness map generation unit integrates the roughness maps by matching their respective levels. (13) The information processing device according to (12), wherein the roughness map generation unit derives a reliability of the roughness and sets a gain for adjusting the level of the roughness map based on the reliability. (14) The information processing device according to (13), wherein the roughness map generation unit generates a difference image between the parallel polarization image and the cross polarization image, detects flash areas included in the roughness map using the difference image, and derives the reliability based on the detected flash areas. (15) The information processing device according to (13) or (14), wherein the roughness map generation unit sets the gain based on an average value of the reliabilities of the areas of the roughness map with high reliability. (16) The information processing device according to any of (1) to (15), further comprising a roughness integration unit that integrates multiple roughness maps corresponding to the parallel polarization images from different viewpoints by adjusting their levels. (17) The information processing device according to (16), wherein the roughness integration unit sets a gain of the roughness map to be processed so that a level of the roughness map to be processed matches a level of the processed roughness map.(18) The information processing device according to (17), wherein the roughness integration unit sets the gain using an average of the roughness in an overlapping area between the current roughness map to be processed and the previous roughness map to be processed. (19) The information processing device according to (17) or (18), wherein the roughness integration unit sets the gain in an overlapping area between the roughness map to be processed and the processed roughness map. (20) An information processing method comprising: performing roughness estimation to estimate the roughness of a surface of a 3D object using a cross-polarized image and a parallel-polarized image; and generating a roughness map, which is distribution information of the roughness corresponding to the parallel-polarized image; wherein the cross-polarized image is a captured image generated by cross-polarized imaging; and the parallel-polarized image is the captured image generated by parallel-polarized imaging; and the captured image is generated by irradiating a subject with light from a light-emitting unit and receiving the light from the subject by an imaging unit; in the case of the cross-polarized imaging, the light from the light-emitting unit is polarized in a first polarization direction, and the light from the subject is polarized in a second polarization direction perpendicular to the first polarization direction; and in the case of the parallel-polarized imaging, the light from the light-emitting unit is polarized in the first polarization direction, and the light from the subject is polarized in the first polarization direction.
[0231] 200 3D data generation device, 211 Gain correction unit, 212 Roughness estimation unit, 213 Metallic estimation unit, 214 High resolution unit, 215 Albedo integration unit, 216 Roughness unit, 217 Metallic integration unit, 221 Corresponding point detection unit, 222 Imaging attitude estimation unit, 223 Geometry / normal estimation unit, 224 UV placement determination unit, 231 Integration unit, 232 Geometry generation unit, 411 Difference image generation unit, 412 Flash area detection unit, 413 Reliability derivation unit, 414 Roughness estimation unit, 415 Gain setting unit, 416 Level adjustment unit, 417 Blending ratio setting unit, 418 Blending unit, 431 Gain setting unit, 432 Level adjustment unit, 433 Integration unit, 500 Imaging device, 511 control unit, 512 imaging control unit, 513 light emitting unit, 514 imaging unit, 515 3D data generating unit, 516 storage unit, 517 display unit, 518 operation unit, 519 output unit, 900 computer
Claims
1. An information processing device comprising: a roughness map generation unit that performs roughness estimation to estimate the roughness of a 3D object surface using a cross polarized image and a parallel polarized image, and generates a roughness map that is distribution information of the roughness corresponding to the parallel polarized image, wherein the cross polarized image is a captured image generated by cross polarized imaging, and the parallel polarized image is a captured image generated by parallel polarized imaging, and the captured image is generated by irradiating a subject with light from a light emitting unit and receiving the light from the subject by an imaging unit, and in the case of the cross polarized imaging, the light from the light emitting unit is polarized in a first polarization direction, and the light from the subject is polarized in a second polarization direction perpendicular to the first polarization direction, and in the case of the parallel polarized imaging, the light from the light emitting unit is polarized in the first polarization direction, and the light from the subject is polarized in the first polarization direction.
2. The information processing device of claim 1, wherein the roughness map generation unit detects a flash area contained in the parallel polarization image using a difference image between the parallel polarization image and the cross polarization image, analyzes the detected flash area, and performs the roughness estimation based on the analysis result of the flash area.
3. The information processing device according to claim 1, further comprising a resolution increasing unit that increases the resolution of the roughness map using an albedo map as a guide signal.
4. The information processing device according to claim 1, further comprising a metallic map generation unit that performs metallic estimation using the cross polarization image and the parallel polarization image and generates a metallic map.
5. The information processing device according to claim 4, wherein the metallic map generation unit estimates that the subject is metallic when the pixel values of the parallel polarization image are sufficiently large relative to the cross polarization image for a processing target area.
6. The information processing device according to claim 4, further comprising a resolution increasing unit that increases the resolution of the metallic map using an albedo map as a guide signal.
7. The information processing device according to claim 1, further comprising an albedo map generator that performs gain correction on the cross-polarized image to generate an albedo map.
8. The information processing device according to claim 1, wherein the roughness map generating unit integrates the roughness maps corresponding to the plurality of parallel polarization images from the same viewpoint so as to suppress specular reflection components.
9. The information processing device according to claim 8, wherein the roughness map generating section derives a reliability of the roughness and integrates a plurality of the roughness maps based on the reliability.
10. The information processing device of claim 9, wherein the roughness map generation unit generates a difference image between the parallel polarization image and the cross polarization image, detects a flash area included in the roughness map using the difference image, and derives the reliability based on the detected flash area.
11. The information processing device according to claim 9, wherein the roughness map generating section blends a plurality of the roughness maps at a ratio according to the reliability.
12. The information processing device according to claim 8, wherein the roughness map generating section combines and integrates the respective levels of the roughness maps.
13. The information processing device according to claim 12, wherein the roughness map generating section derives a reliability of the roughness, and sets a gain for adjusting the level of the roughness map based on the reliability.
14. The information processing device of claim 13, wherein the roughness map generation unit generates a difference image between the parallel polarization image and the cross polarization image, detects a flash area included in the roughness map using the difference image, and derives the reliability based on the detected flash area.
15. The information processing device according to claim 13, wherein the roughness map generating section sets the gain based on an average value of the reliability in an area of the roughness map where the reliability is high.
16. The information processing device according to claim 1, further comprising a roughness integration unit that integrates a plurality of the roughness maps corresponding to the parallel polarization images from different viewpoints by matching their respective levels.
17. The information processing device according to claim 16, wherein the roughness integration section sets a gain for the roughness map to be processed so that the level of the roughness map to be processed matches that of a processed roughness map.
18. The information processing device according to claim 17, wherein the roughness integration section sets the gain by using an average of the roughness in an overlapping area between the current roughness map to be processed and the previous roughness map to be processed.
19. The information processing device according to claim 17, wherein the roughness integration section sets the gain in an overlapping area of the roughness map to be processed and the processed roughness map.
20. An information processing method comprising: performing roughness estimation to estimate the roughness of a 3D object surface using a cross polarized image and a parallel polarized image; and generating a roughness map, which is distribution information of the roughness corresponding to the parallel polarized image; wherein the cross polarized image is an image generated by cross polarized imaging; and wherein the parallel polarized image is an image generated by parallel polarized imaging; and wherein the image is generated by irradiating a subject with light from a light emitting unit and receiving the light from the subject by an imaging unit; and, in the case of the cross polarized imaging, the light from the light emitting unit is polarized in a first polarization direction, and the light from the subject is polarized in a second polarization direction perpendicular to the first polarization direction; and, in the case of the parallel polarized imaging, the light from the light emitting unit is polarized in the first polarization direction, and the light from the subject is polarized in the first polarization direction.
Citation Information
Patent Citations
Information processing device, information processing method, and recording medium
WO2019069536A1
Surface inspection device, surface inspection method, surface inspection program, and computer-readable recording medium
JP2013033017A
Image processing device, image processing method and program
JP2018017611A
Scanning and measuring of surface features
WO2005047816A1
Image processing device, and method for acquiring material information
WO2022259466A1
Cited By
Plant management system
US20240202922A1
