Information processing device and information processing method

Abstract

An information processing device (inference device) according to the present invention includes a VI component prediction unit and a VD component prediction unit. The VI component prediction unit predicts, as VI components, color components of an object that do not change depending on the photographing viewpoint or illumination conditions. The VD component prediction unit predicts, as VD components, color components of the object that change depending on the photographing viewpoint and illumination conditions. Another information processing device (learning device) according to the present invention includes a VI component learning unit and a VD component learning unit. The VI component learning unit uses a plurality of photography data sets having different illumination conditions to learn, as VI components, color components of an object that do not change depending on the photographing viewpoint or illumination conditions. The VD component learning unit uses the plurality of photography data sets to learn, as VD components, color components of the object that change depending on the photographing viewpoint and illumination conditions.

Description

Information processing device and information processing method

[0001] The present invention relates to an information processing device and an information processing method.

[0002] NeRF (Neural Radiance Field) represents complex 3D scenes as radiance fields. NeRF learns the parameters of the radiance field from multi-viewpoint images using a neural network. The radiance field is represented by a function that takes position coordinates and viewpoint direction in 3D space as arguments and outputs color and volume density. NeRF learns the function to minimize the error between the color of the rendered image and the color of the captured image. Images from arbitrary viewpoints are inferred using the trained neural network (NeRF model).

[0003] Numerous improvements and advancements to NeRF have been announced. Ref-NeRF, proposed in 2022, represents color by adding diffuse and specular colors. Ref-NeRF learns the specular color as a View-Dependent component (VD component) that changes depending on the viewing angle (viewpoint direction). Ref-NeRF learns the diffuse color as a View-Independent component (VI component) that does not change regardless of the viewing angle (viewpoint direction). This enables high reconstruction quality. Here, we will tentatively refer to the NeRF method that renders color using a combination of VI and VD components as Viewing-NeRF.

[0004] Mildenhall, et al. “NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis,” Communications of the ACM 65 “1”, pp. 99-106, 2021. Verbin, et al. “Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields,” IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp. 5481-5490, 2022.

[0005] One of the features of NeRF is that it can learn using only multi-view images of a scene and their poses (camera position and orientation), without requiring any prior knowledge. On the other hand, because NeRF learns to reproduce (reconstruct) the captured images as faithfully as possible, it cannot reconstruct information that was not included in the capture. Therefore, if multiple scenes with different lighting conditions are anticipated, it is necessary to prepare a separate learning model for each lighting condition. It would be convenient if the NeRF model could be trained simultaneously from multiple scenes with various lighting conditions, and rendering could be performed by switching between scenes as presets.

[0006] Therefore, this disclosure proposes an information processing device and an information processing method capable of reproducing multiple scenes with different lighting conditions.

[0007] According to this disclosure, an information processing device (inference device) is provided, comprising: a VI component prediction unit that predicts the color components of a subject that do not change depending on the shooting viewpoint and lighting conditions as VI components; and a VD component prediction unit that predicts the color components of the subject that change according to the shooting viewpoint and lighting conditions as VD components. According to this disclosure, an information processing device (learning device) is provided, comprising: a VI component learning unit that learns the color components of a subject that do not change depending on the shooting viewpoint and lighting conditions as VI components using a plurality of shooting datasets with different lighting conditions; and a VD component learning unit that learns the color components of the subject that change according to the shooting viewpoint and lighting conditions as VD components using the plurality of shooting datasets. Furthermore, according to this disclosure, an information processing method is provided in which the information processing of the information processing device is performed by a computer.

[0008] This is an explanatory diagram of the conventionally proposed Viewing NeRF. This diagram shows an example of generating an independent NeRF model for each lighting condition. This is an explanatory diagram of the Relighting NeRF proposed in this disclosure. This diagram shows an example of the configuration of an information processing device for implementing the Relighting NeRF of this disclosure. This diagram shows an example of NeRF model training. This diagram explains the image inference process based on the training results. This diagram shows an information processing device related to a modified example. This diagram shows an example of the hardware configuration of the information processing device.

[0009] Embodiments of the present disclosure will be described in detail below with reference to the drawings. In each of the following embodiments, the same parts will be denoted by the same reference numerals, and redundant descriptions will be omitted.

[0010] The explanation will proceed in the following order: [1. Conventionally proposed Viewing NeRF] [2. Relighting NeRF proposed in this disclosure] [3. Configuration of the information processing device] [3-1. NeRF model training] [3-2. Image inference based on training results] [4. Modification examples] [5. Hardware configuration examples] [6. Effects]

[0011] [1. Conventional Viewing NeRF] Figure 1 is an explanatory diagram of the conventionally proposed Viewing NeRF.

[0012] Viewing NeRF learns combinations of color components that change depending on the viewpoint (VD component) and color components that remain constant regardless of the viewpoint (VI component) from a set of images taken from various viewpoints (multi-view images). When the viewpoint position and line of sight direction are input to a trained NeRF model, the radiance field is reconstructed, and an image of the subject as seen from the specified position and direction (reconstructed image) is output.

[0013] Viewing NeRF learns to reproduce (reconstruct) captured images as faithfully as possible. Therefore, it cannot render (relight) images with different lighting conditions (color, position, direction, shape, number of light sources, etc.) than those at the time of capture. To perform relighting, it is necessary to train the NeRF model for each desired lighting condition. As a result, the data size of the NeRF model becomes large.

[0014] Figure 2 shows an example of generating independent NeRF models for each lighting condition.

[0015] In the example in Figure 2, "Multiview Image A" and "Multiview Image B" represent groups of images taken under different lighting conditions. To faithfully reproduce the captured images, training is performed separately for "Multiview Image A" and "Multiview Image B". Using the VI and VD components trained on "Multiview Image A", an image under the same lighting conditions as "Multiview Image A" ("Reconstructed Image A") can be estimated with high accuracy. Using the VI and VD components trained on "Multiview Image B", an image under the same lighting conditions as "Multiview Image B" ("Reconstructed Image B") can be estimated with high accuracy.

[0016] However, with this method, the VI and VD components are learned independently for each lighting condition, so the results learned under one lighting condition cannot be applied to another. Even with the same subject, a unified approach is not possible due to differences in lighting conditions, requiring separate NeRF models for each lighting condition.

[0017] Improvements to NeRF aimed at relighting are also being considered. NeRV (Neural Reflectance and Visibility Fields), proposed in 2021, is an extension of NeRF that aims to render images under arbitrary lighting conditions (see Non-Patent Document 3 below).

[0018] [Non-patent Document 3] Srinvasan, et al. “NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis,” IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp. 7495-7504, 2021.

[0019] NeRV approximates a 3D scene in line with the physically based rendering concept in the Computer Graphics field. NeRV takes the position coordinates and viewpoint direction in 3D space as arguments and learns scene properties such as volume density, surface normals, material parameters, distance to the first surface intersection, and ambient light visibility. By combining these properties, it can render new arbitrary viewpoint images under specified lighting conditions.

[0020] The lighting conditions specified at this time are light source information manually created using Computer Graphics, and are assumed to differ from the actual lighting conditions during shooting. Therefore, this differs from NeRF's philosophy of faithfully reproducing the captured 3D scene. Furthermore, the parameters representing the scene characteristics are obtained approximately through learning and do not guarantee physical accuracy. Therefore, due to the difficulty of simultaneously optimizing many parameters, it is challenging to obtain them stably. If the model based on the learned parameters contradicts the geometry of the actual 3D scene, visual artifacts will occur in the rendered image.

[0021] This disclosure was made in view of the above-mentioned issues. This disclosure proposes a method for simultaneously learning an NeRF model from multiple scenes with various lighting conditions and rendering while switching between scenes as presets. This provides a method (Relighting NeRF) that enables relighting while reducing the data size of the NeRF model. The details are explained below.

[0022] [2. Relieving NeRF proposed in this disclosure] Figure 3 is an explanatory diagram of the Relieving NeRF proposed in this disclosure.

[0023] The NeRF model disclosed herein learns common VI components from multiple multi-view images with different lighting conditions. In the example shown in Figure 3, by using both "Multi-view Image A" and "Multi-view Image B" as training data, the color components of the subject that do not change depending on the shooting viewpoint and lighting conditions are obtained as VI components.

[0024] The VD component is optimized as the difference from the VI component. The VD component is learned for each lighting condition and acquired as the color component of the subject that changes according to the shooting viewpoint and lighting conditions. In the example in Figure 3, based on "Multi-view image A" and the VI component, the VD component of "Multi-view image A" under the lighting conditions is calculated. Ａ Based on "Multiview Image B" and the VI component, the VD component "VD" under the illumination conditions of "Multiview Image B" is obtained. Ｂ " can be obtained.

[0025] The color of the subject is obtained, for example, by adding the VI component and the VD component. "Reconstructed image A" is obtained by adding the VI component and the VD component "VD Ａ The "reconstructed image B" is inferred using the VI component and the VD component "VD Ｂ This is inferred using "[...]".

[0026] The method disclosed herein allows for the reproduction of various lighting conditions using a single NeRF model. For example, changing the lighting conditions (color, position, direction, shape, number of light sources, etc.) in a photographic dataset not only changes the apparent color of the photographed subject, but also alters various lighting effects. For instance, a glossy subject may exhibit strong specular reflection when photographed from a certain viewpoint, or a shadow may be visible when photographed from a certain viewpoint depending on the relative position of the light source and the subject. Furthermore, when there are multiple subjects, the indirect lighting effect that alters how they appear to each other also changes depending on the lighting conditions.

[0027] Predicting the VD component for each of multiple image datasets captured under varying lighting conditions means extracting colors that fluctuate with changes in lighting conditions. On the other hand, the shape of the subject itself does not change even when lighting conditions are changed. Furthermore, many subjects have object-specific colors that do not change with lighting conditions. Therefore, this disclosure proposes extracting these lighting-condition-independent components from a large number of image datasets.

[0028] The NeRF model is trained by minimizing the error between the RGB values ​​drawn by rendering and the pixel values ​​of the captured image. It is based on the VI component common to multiple captured datasets, and added to it the VD component "VD Ａ " or VD component "VD Ｂ By adding these values, RGB values ​​are calculated for each lighting condition. The error between the calculated RGB values ​​and the colors of the captured image is calculated, and by minimizing this error, the VI component and 3D shape are optimized.

[0029] This method leverages NeRF's characteristic of photorealistically reproducing captured 3D scenes while simultaneously reconstructing subjects under various lighting conditions, effectively achieving relighting. Conventional NeRF models required training a separate model for each lighting condition, resulting in a large overall data size. However, the disclosed method allows a single model to represent all lighting conditions, thus reducing the NeRF model's data size. Furthermore, increasing the variety of captured datasets is expected to improve the prediction accuracy of the VI component and 3D shape.

[0030] This method allows for changing the intensity of illumination light by adjusting the size of the VD component and adding it to the VI component. Furthermore, by assigning weights to multiple VD components and adding the weighted individual VD components to the VI component, the synthesis of illumination light can be represented. Moreover, even for subjects that are difficult to model in 3D (e.g., glossy or translucent objects), high accuracy in predicting the 3D shape can be achieved by combining images taken under various lighting conditions. Therefore, this technology can be used not only for rendering purposes but also for 3D shape prediction.

[0031] [3. Configuration of Information Processing Device] Figure 4 is a diagram showing an example configuration of the information processing device 1 for implementing Relighting NeRF according to the present disclosure. Below, an example will be described in which the information processing device 1 performs both the learning process of the NeRF model and the inference process based on the learning results. However, the learning process and the inference process may be performed by separate information processing devices.

[0032] Figure 4 shows an example configuration of a learning device for training a NeRF model. The information processing device 1 includes a 3D shape learning unit 10, a VI component learning unit 20, a VD component learning unit 30, and an RGB value prediction unit 40.

[0033] The information processing device 1 uses multiple image datasets DS with different lighting conditions as training datasets. Each image dataset DS includes a group of images IM (see Figure 5) taken under the corresponding lighting conditions. The image group IM includes multiple images of the subject taken from different viewpoints as a multi-view image. The VI component learning unit 20 and the VD component learning unit 30 perform training on the VI component and VD component, respectively, using the image group IM of multiple subjects with different lighting conditions.

[0034] The VI component represents the basic color of the subject itself, independent of the lighting conditions and the shooting position. The VD component represents the color superimposed on the basic color according to the lighting conditions and the shooting position. For example, the VI component represents the color of the diffused light radiated isotropically (diffused color). The VD component represents the color of the specular reflection light radiated anisotropically (specular reflection color). The specular reflection light can be calculated based on the three-dimensional shape of the subject. Therefore, the three-dimensional shape learning unit 10 learns the three-dimensional shape of the subject using a plurality of shooting data sets DS.

[0035] Based on the learning result, the three-dimensional shape learning unit 10 acquires information regarding the three-dimensional shape of the subject as shape information. The acquisition of the shape information can be performed using a neural network. The VI component learning unit 20 and the VD component learning unit 30 respectively perform learning of the VI component and the VD component using the shape information of the subject. The prediction of the VI component and the VD component can be performed using a neural network.

[0036] The RGB value prediction unit 40 predicts the color (RGB value) of the subject from the acquired VI component and VD component. The VI component learning unit 20 and the VD component learning unit 30 perform learning so that the error between the predicted RGB value and the color of the actual captured image is minimized.

[0037] Through learning using a plurality of shooting data sets DS, the VI component learning unit 20 learns the color component of the subject that does not change depending on the shooting viewpoint and the lighting conditions as the VI component. The VD component learning unit 30 learns the color component of the subject that changes according to the shooting viewpoint and the lighting conditions as the VD component in the form of the difference from the VI component. The VI component learning unit 20 learns the VI component common to each shooting data set DS. The VD component learning unit acquires individual VD components corresponding to individual lighting conditions based on the learning results for each shooting data set DS.

[0038] For example, the VD component learning unit 30 has a plurality of individual VD component learning units 30A corresponding to each shooting data set DS. The RGB value prediction unit 40 has an individual RGB value prediction unit 40A for each individual VD component learning unit 30A.

[0039] The individual VD component learning unit 30A learns the VD components under the illumination conditions of the corresponding captured data set DS. The individual RGB value prediction unit 40A calculates the RGB values based on the VI components and the VD components predicted by the corresponding individual VD component learning unit 30A. The individual VD component learning unit 30A is optimized so that the error between the RGB values and the color of the captured image of the corresponding captured data set DS is minimized.

[0040] In the example of FIG. 4, as the captured data set DS, for example, the first captured data set DS1 and the second captured data set DS2 are prepared. The first captured data set DS1 includes an image group IM captured under the first illumination conditions. The second captured data set DS2 includes an image group IM captured under the second illumination conditions different from the first illumination conditions.

[0041] The VD component learning unit 30 includes the first VD component learning unit 31 and the second VD component learning unit 32 as the individual VD component learning units 30A, respectively. The first VD component learning unit 31 learns the VD components (first VD components) under the first illumination conditions. The second VD component learning unit 32 learns the VD components (second VD components) under the second illumination conditions.

[0042] The RGB value prediction unit 40 includes the first RGB value prediction unit 41 and the second RGB value prediction unit 42 as the individual RGB value prediction units 40A, respectively. The first RGB value prediction unit 41 predicts the RGB values (first RGB values) under the first illumination conditions based on the VI components and the first VD components. The second RGB value prediction unit 42 predicts the RGB values (second RGB values) under the second illumination conditions based on the VI components and the second VD components.

[0043] The VI component learning unit 20 and the first VD component learning unit 31 are optimized so that the error between the first RGB values and the color of the captured images of the first captured data set DS1 is minimized. The VI component learning unit 20 and the second VD component learning unit 32 are optimized so that the error between the second RGB values and the color of the captured images of the second captured data set DS2 is minimized.

[0044] In the example shown in Figure 4, learning was performed for the first and second lighting conditions. However, the number of lighting conditions to be learned is not limited to two. Depending on the number of anticipated scenes (lighting conditions), learning may be performed for three or more lighting conditions.

[0045] [3-1. NeRF Model Training] Figure 5 shows an example of training a NeRF model.

[0046] The image dataset DS includes an image group IM, three-dimensional position coordinates PC, and a direction vector DV. The image group IM includes images from various viewpoints (multi-viewpoint images) taken under the same lighting conditions. The three-dimensional position coordinates PC indicate the coordinates of the three-dimensional position of each sampling point set in the radiance field. The direction vector DV indicates the vector in the line of sight direction from the shooting viewpoint to the sampling point. The direction vector DV is obtained from the camera's position and orientation information (pose information) at the time of shooting.

[0047] The learning methods for the 3D shape learning unit 10, the VI component learning unit 20, and the individual VD component learning unit 30A are the same as those for the conventional Viewing NeRF. In this disclosure, as in the conventional, the 3D shape, VI component, and VD component are learned from a group of images IM with known pose information.

[0048] The difference from conventional methods is that the 3D shape learning unit 10 and the VI component learning unit 20 are optimized not from a single image data set DS, but from multiple image data sets DS with different lighting conditions. By using data with various lighting conditions for learning, universal color components that apply to all lighting conditions are obtained as VI components. In addition, the learning accuracy of the 3D shape is also improved because the amount of data used for learning is increased.

[0049] For example, the 3D shape learning unit 10 includes a volume density learning unit 11 and a normal vector learning unit 12. The volume density learning unit 11 learns the volume density in 3D space using 3D position coordinates PC as input. The normal vector learning unit 12 learns the normal vector of each sampling point using the volume density and direction vector DV as input.

[0050] The volume density may be expressed as a probability distribution representing the spatial occupancy or opacity of the subject at each sampling point, or as binary information representing the presence or absence of the subject. The normal vector may be obtained by spatially differentiating the volume density, or it may be obtained using a neural network with the volume density as input. The 3D shape learning unit 10 acquires the volume density and normal vector as information indicating the 3D shape of the subject.

[0051] The VI component learning unit 20 takes volume density as input and learns the VI component that does not change with respect to the direction vector DV. The individual VD component learning unit 30A takes volume density, normal vector, and direction vector DV as input and learns the VD component that changes with respect to the direction vector DV. The VI component learning unit 20 and the individual VD component learning unit 30A can each be configured as neural networks. The individual RGB value prediction unit 40A calculates the RGB value by adding the VI component and the VD component.

[0052] The RGB values ​​are compared with pixel values ​​extracted from the image group IM in the captured dataset DS. The parameters of the VI component learning unit 20 and the individual VD component learning unit 30A are optimized to minimize the error obtained from the comparison result.

[0053] As described above, in the Relighting NeRF of this disclosure, the 3D shape learning unit 10 learns the 3D shape of the subject from multiple shooting datasets DS with different lighting conditions. Because it learns using multiple shooting datasets DS, highly accurate shape information can be obtained. The VI component learning unit 20 learns the VI component from the acquired shape information as a subject-specific color independent of lighting conditions. The individual VD component learning unit 30A takes the shape information and the VI component as input and learns the VD component as a color that varies depending on the lighting conditions.

[0054] The individual RGB value prediction unit 40A takes the VI component and the VD component as input and predicts the RGB values. The RGB values ​​are calculated, for example, by adding the VD component to the VI component. The form of addition includes both addition without weighting the VD component and addition with some weighting applied to the VD component.

[0055] For each lighting condition, the VD component learning unit 30 calculates the difference between the color (RGB value) indicated by the addition result of the VD component and the VI component calculated by the individual VD component learning unit 30A and the color of the subject obtained from the captured data set DS corresponding to the lighting condition. The VD component learning unit 30 learns the VD component so that the calculated difference is minimized.

[0056] As described above, the VD component learning unit 30 can obtain, as an addition result, a value obtained by adding the weighted VD component to the VI component. When weighting the VD component, the magnitude of the weight can be set for each individual VD component learning unit 30A. For example, in the example of FIG. 4, the first RGB value and the second RGB value can be obtained using the following equations (1) and (2). First RGB value = VI component + first weight × first VD component (1) Second RGB value = VI component + second weight × first VD component (2)

[0057] The calculation of the RGB value can also be performed using a neural network. The neural network to be used can be set for each individual VD component learning unit 30A. For example, in the example of FIG. 4, the first RGB value and the second RGB value can be obtained using the following equations (3) and (4). In the following equations, "F １ " and "F ２ " represent a neural network. First RGB value = F １ (VI component, first VD component) (3) Second RGB value = F ２ (VI component, second VD component) (4)

[0058] The first weight and the neural network F １ are optimized so that the error between the first RGB value and the color of the captured image included in the first captured data set DS1 is minimized. The second weight and the neural network F ２ are optimized so that the error between the second RGB value and the color of the captured image included in the second captured data set DS2 is minimized.

[0059] The neural network F １The neural network F is trained to represent the first image dataset DS1. ２ The neural network F learns to represent the second image dataset DS2. The first image dataset DS1 is used to train the neural network F. ２ It was used for training, and the second image dataset DS2 was used for the neural network F １ It will not be used for learning purposes.

[0060] On the other hand, the neural network of the VI component learning unit 20 learns using information from both the first shooting dataset DS1 and the second shooting dataset DS2. Changing the shooting dataset DS changes the lighting conditions (color, position, direction, shape, number of light sources, etc.), but the subject being photographed remains unchanged. Therefore, the neural network of the VI component learning unit 20 is optimized to represent the basic color of the subject itself, which is not affected by the changes in lighting conditions caused by the change in the shooting dataset DS.

[0061] The universality of the VI component is expected to increase as the number of imaging datasets DS increases. The individual VD component learning unit 30A learns to predict the VD component in the form of the difference from the universal VI component. As the universality of the VI component increases, the individual VD component learning unit 30A is expected to be optimized to capture changes in lighting conditions with higher accuracy by specializing in the corresponding imaging datasets DS.

[0062] [3-2. Image Inference Based on Learning Results] Figure 6 is a diagram illustrating the image inference process based on learning results. Figure 6 shows an example configuration of an inference device for performing inference processing using a trained NeRF model. The information processing device 1 has a VI component prediction unit 70, a VD component prediction unit 80, and a color addition unit 50.

[0063] Image inference processing is performed using the trained VI component learning unit 20 and VD component learning unit 30. For example, the trained parameters of the VI component learning unit 20 are applied to the VI component prediction unit 70. The trained parameters of the VD component learning unit 30 are applied to the VD component prediction unit 80.

[0064] The VD component prediction unit 80 has a plurality of individual VD component prediction units 80A, each corresponding to an individual VD component learning unit 30A. The learned parameters of each individual VD component learning unit 30A are applied to the corresponding individual VD component prediction unit 80A. In the example of Figure 6, the VD component prediction unit 80 includes a first VD component prediction unit 81 and a second VD component prediction unit 82 as individual VD component prediction units 80A, respectively. The first VD component prediction unit 81 acquires the learned parameters learned by the first VD component learning unit 31. The second VD component prediction unit 82 acquires the learned parameters learned by the second VD component learning unit 32.

[0065] The VI component prediction unit 70 uses the learned parameters of the VI component learning unit 20 to predict the color components of the subject that do not change depending on the shooting viewpoint and lighting conditions as the VI component. The VD component prediction unit 80 uses the learned parameters of the VD component learning unit 30 to predict the color components of the subject that change according to the shooting viewpoint and the lighting conditions as the VD component.

[0066] The VI component prediction unit 70 predicts a common VI component for different lighting conditions. The VD component prediction unit 80 uses multiple individual VD component prediction units 80A to individually predict, for each lighting condition, the color component corresponding to the difference between the subject's color and the VI component as a VD component. When a combination of multiple lighting conditions used for training is used as the lighting condition for relighting, the VD components corresponding to each lighting condition are weighted according to the combination ratio and added to the VI component. This addition process is performed by the color addition unit 50.

[0067] The lighting conditions to be combined and the combination ratio can be arbitrarily specified by the user using a keyboard or the like. The color addition unit 50 acquires the combination ratio of multiple lighting conditions to be combined based on the user input information. The color addition unit 50 combines the individual VD components predicted for each lighting condition by each individual VD component learning unit 30A based on the combination ratio. The color addition unit 50 predicts the color components obtained by the combination as the VD components in the relighting environment in which each lighting condition has been combined.

[0068] In the example shown in Figure 6, the composite ratio of the first lighting condition is set as "α", and the composite ratio of the second lighting condition is set as "β". The specific values ​​of "α" and "β" can be arbitrarily set by the user. The composite value (RGB output value) output from the color addition unit 50 is expressed by the following formula (5): RGB output value = VI component + α × first VD component + β × second VD component (5)

[0069] [4. Modified Version] Figure 7 shows an information processing device 2 according to a modified version. The following explanation will focus on the differences from the information processing device 1 shown in Figure 4.

[0070] The difference between this modified example and the information processing device 1 is that the VI component is calculated using the albedo, which is an eigenvalue of the object. The VI component learning unit 60 has an albedo learning unit 61 and a gain learning unit 62. The albedo learning unit 61 learns the albedo, which is the intrinsic reflectance of the subject. The gain learning unit 62 learns the gain of the illumination intensity. The VI component learning unit 60 obtains the VI component as the product of the albedo and the gain.

[0071] For example, the gain learning unit 62 has a plurality of individual gain learning units 65 corresponding to each image data set DS. Each individual gain learning unit 65 learns the gain of the corresponding image data set DS under the lighting conditions. The gain may be obtained as a single coefficient multiplied by the albedo, or it may be obtained using a neural network "F" as shown in equation (6) below. Gain = F (VI component, 3D shape) (6)

[0072] The product of gain and albedo becomes the VI component under the corresponding lighting conditions. The VI component learning unit 60 learns the VI component for each lighting condition using each individual gain learning unit 65. In the example in Figure 7, the VI component learning unit 60 includes a first gain learning unit 63 and a second gain learning unit 64 as individual gain learning units 65. The first gain learning unit 63 learns the gain (first gain) under the first lighting condition. The second gain learning unit 64 learns the gain (second gain) under the second lighting condition.

[0073] The first RGB value prediction unit 41 predicts the first VI component by multiplying the albedo by the first gain. Based on the first VI component and the first VD component, the first RGB value prediction unit 41 predicts the RGB value (first RGB value) under the first lighting conditions. The second RGB value prediction unit 42 predicts the second VI component by multiplying the albedo by the second gain. Based on the second VI component and the second VD component, the second RGB value prediction unit 42 predicts the RGB value (second RGB value) under the second lighting conditions.

[0074] The first gain learning unit 63 and the first VD component learning unit 31 are optimized to minimize the error between the first RGB values ​​and the color of the captured image in the first captured dataset DS1. The second gain learning unit 64 and the second VD component learning unit 32 are optimized to minimize the error between the second RGB values ​​and the color of the captured image in the second captured dataset DS2.

[0075] [5. Hardware Configuration Example] Figure 8 shows an example of the hardware configuration of the information processing devices 1 and 2.

[0076] The information processing of the information processing devices 1 and 2 is realized, for example, by a computer 1000. The computer 1000 has a CPU (Central Processing Unit) 1100, RAM (Random Access Memory) 1200, ROM (Read Only Memory) 1300, HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. The various parts of the computer 1000 are connected by a bus 1050.

[0077] The CPU 1100 operates based on programs (program data 1450) stored in the ROM 1300 or HDD 1400, and controls each part. For example, the CPU 1100 loads the programs stored in the ROM 1300 or HDD 1400 into the RAM 1200 and executes processing corresponding to various programs.

[0078] ROM 1300 stores boot programs such as the BIOS (Basic Input Output System) that are executed by the CPU 1100 when the computer 1000 starts up, as well as programs that depend on the computer 1000's hardware.

[0079] The HDD 1400 is a computer-readable non-temporary recording medium that non-temporarily records programs executed by the CPU 1100 and data used by such programs. Specifically, the HDD 1400 is a recording medium that records an information processing program according to the embodiment, which is an example of program data 1450.

[0080] The communication interface 1500 is an interface for the computer 1000 to connect to an external network 1550 (for example, the Internet). For example, the CPU 1100 can receive data from other devices or transmit data it has generated to other devices via the communication interface 1500.

[0081] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from input devices such as a keyboard or mouse via the input / output interface 1600. The CPU 1100 also transmits data to output devices such as a display device, speaker, or printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs recorded on a predetermined recording medium (media). Examples of media include optical recording media such as DVDs (Digital Versatile Discs) and PDs (Phase Change Rewritable Discs), magneto-optical recording media such as MOs (Magneto-Optical Discs), tape media, magnetic recording media, or semiconductor memory.

[0082] For example, when computer 1000 functions as information processing devices 1 and 2 according to the embodiment, the CPU 1100 of computer 1000 realizes the functions of each of the parts described above by executing an information processing program loaded on RAM 1200. The HDD 1400 stores the information processing program, various models, and various data according to this disclosure. The CPU 1100 reads and executes the program data 1450 from HDD 1400, but as another example, these programs may be obtained from other devices via an external network 1550.

[0083] The CPU 1100 is an example of a processing circuit for realizing the functions of the information processing devices 1 and 2. The functions of the information processing devices 1 and 2 may also be realized by other processing circuits such as MPUs, GPUs, APUs, ASICs, and FPGAs.

[0084] [6. Effects] The information processing devices 1 and 2 have a VI component prediction unit 70 and a VD component prediction unit 80. The VI component prediction unit 70 predicts the color components of a subject that do not change depending on the shooting viewpoint and lighting conditions as the VI component. The VD component prediction unit 80 predicts the color components of a subject that change depending on the shooting viewpoint and lighting conditions as the VD component. In the information processing method of this disclosure, the processing of the information processing devices 1 and 2 is executed by a computer 1000. The program of this disclosure causes the computer 1000 to implement the processing of the information processing devices 1 and 2.

[0085] In this configuration, the VI and VD components represent the color changes of the subject under different lighting conditions. Therefore, it is possible to reproduce multiple scenes with different lighting conditions.

[0086] The VI component prediction unit 70 predicts a common VI component for different lighting conditions. The VD component prediction unit 80 individually predicts the color component corresponding to the difference between the color of the subject and the VI component as the VD component for each lighting condition.

[0087] In this configuration, the VD component represents the color difference according to the lighting conditions.

[0088] The information processing devices 1 and 2 each have a color addition unit 50. The color addition unit 50 acquires the synthesis ratio of multiple lighting conditions to be synthesized. The color addition unit 50 synthesizes the individual VD components predicted for each lighting condition based on the synthesis ratio. The color addition unit 50 predicts the color components obtained by synthesis as the VD components in the relighting environment where each lighting condition is synthesized.

[0089] This configuration allows for the prediction of colors under different lighting conditions than those used during training.

[0090] The information processing devices 1 and 2 include a VI component learning unit 20 and a VD component learning unit 30. The VI component learning unit 20 uses multiple shooting datasets DS with different lighting conditions to learn the color components of a subject that do not change depending on the shooting viewpoint and lighting conditions as the VI component. The VD component learning unit 30 uses multiple shooting datasets DS to learn the color components of a subject that change according to the shooting viewpoint and lighting conditions as the VD component.

[0091] In this configuration, the VI and VD components represent the color changes of the subject under different lighting conditions. Therefore, it is possible to reproduce multiple scenes with different lighting conditions.

[0092] The VI component learning unit 20 learns the VI component common to each of the multiple shooting datasets DS using the multiple shooting datasets DS. The VD component learning unit 30 acquires individual VD components corresponding to individual lighting conditions based on the learning results for each shooting dataset DS.

[0093] In this configuration, the VD component represents the color difference according to the lighting conditions. The VI component is trained using multiple image datasets DS. Because a large number of images are used for training, the prediction accuracy of the VI component is improved. The VD component is predicted in the form of the difference from the VI component. Because the VI component is predicted accurately, the prediction accuracy of the VD component is also improved.

[0094] The VD component learning unit 30 calculates the difference between the color shown by the sum of the VD component and the VI component for each lighting condition, and the color of the subject obtained from the shooting dataset corresponding to the lighting condition. The VD component learning unit 30 learns the VD component so that the calculated difference is minimized.

[0095] This configuration allows for the acquisition of highly accurate VD components.

[0096] The VD component learning unit 30 obtains a value obtained by adding the weighted VD component to the VI component as the summation result.

[0097] With this configuration, the accuracy of the learning process can be adjusted using weights.

[0098] The information processing devices 1 and 2 each have a 3D shape learning unit 10. The 3D shape learning unit 10 acquires information about the 3D shape of an object as shape information based on the learning results using multiple image data sets DS. The VI component learning unit 20 and the VD component learning unit 30 each learn the VI component and the VD component, respectively, using the shape information.

[0099] This configuration allows for the acquisition of highly accurate 3D shapes based on multiple image datasets (DS). As a result, the prediction accuracy of the VI and VD components is improved.

[0100] The VI component learning unit 20 acquires the VI component as the product of the albedo, which is the intrinsic reflectance of the subject, and the gain of the illumination intensity.

[0101] With this configuration, the VI component is accurately obtained based on physical quantities specific to the subject.

[0102] Furthermore, the effects described herein are merely illustrative and not limiting, and other effects may also occur.

[0103] [Note] The technology may also be configured as follows: (1) An information processing device having: a VI component prediction unit that predicts the color components of a subject that do not change depending on the shooting viewpoint and lighting conditions as VI components; and a VD component prediction unit that predicts the color components of the subject that change according to the shooting viewpoint and lighting conditions as VD components. (2) The information processing device according to (1) above, wherein the VI component prediction unit predicts the common VI components for different lighting conditions, and the VD component prediction unit individually predicts the color components corresponding to the difference between the color of the subject and the VI components as VD components for each lighting condition. (3) The information processing device according to (1) or (2) above, having a color addition unit that acquires the synthesis ratio of a plurality of lighting conditions to be synthesized, synthesizes the individual VD components predicted for each lighting condition based on the synthesis ratio, and predicts the color components obtained by synthesis as the VD components in a relighting environment in which each lighting condition is synthesized. (4) A computer-based information processing method comprising predicting the color components of a subject that do not change depending on the shooting viewpoint and lighting conditions as VI components, and predicting the color components of the subject that change depending on the shooting viewpoint and lighting conditions as VD components. (5) The information processing method according to (4) above, comprising predicting the VI components common to different lighting conditions, and individually predicting the color components corresponding to the difference between the color of the subject and the VI components as VD components for each lighting condition. (6) The information processing method according to (4) or (5) above, comprising obtaining the synthesis ratio of a plurality of lighting conditions to be synthesized, synthesizing the individual VD components predicted for each lighting condition based on the synthesis ratio, and predicting the color components obtained by synthesis as the VD components in a relighting environment in which each lighting condition is synthesized. (7) An information processing device comprising: a VI component learning unit that learns the color components of a subject that do not change depending on the shooting viewpoint and the lighting conditions as VI components using multiple shooting datasets with different lighting conditions; and a VD component learning unit that learns the color components of the subject that change according to the shooting viewpoint and the lighting conditions as VD components using the multiple shooting datasets.(8) The information processing device according to (7) above, wherein the VI component learning unit learns the VI component common to each shooting dataset using the plurality of shooting datasets, and the VD component learning unit acquires individual VD components corresponding to individual lighting conditions based on the learning results for each shooting dataset. (9) The information processing device according to (8) above, wherein the VD component learning unit calculates the difference between the color shown by the sum of the VD component and the VI component for each lighting condition and the color of the subject acquired from the shooting dataset corresponding to the lighting condition, and learns the VD component so as to minimize the calculated difference. (10) The information processing device according to (9) above, wherein the VD component learning unit acquires a value obtained by adding the weighted VD component to the VI component as the sum result. (11) The information processing device according to any one of (7) to (10) above, comprising a three-dimensional shape learning unit that acquires information relating to the three-dimensional shape of the subject as shape information based on the learning results using the plurality of shooting datasets, wherein the VI component learning unit and the VD component learning unit each learn the VI component and the VD component using the shape information. (12) The information processing device according to any one of (7) to (11) above, wherein the VI component learning unit acquires the VI component as the product of the albedo, which is the intrinsic reflectance of the subject, and the gain of the illumination intensity. (13) The information processing device according to any one of (7) to (12) above, wherein each shooting dataset includes a plurality of shooting images taken of the subject from different shooting viewpoints under the lighting conditions corresponding to the shooting dataset, as a multi-view image. (14) A computer-based information processing method comprising: learning the color components of a subject that do not change with respect to the shooting viewpoint and the lighting conditions as VI components using multiple shooting datasets with different lighting conditions; and learning the color components of the subject that change according to the shooting viewpoint and the lighting conditions as VD components using the multiple shooting datasets.(15) The information processing method according to (14), comprising: (16) For each lighting condition, calculating the difference between the color shown by the sum of the VD component and the VI component and the color of the subject obtained from the lighting condition corresponding to the lighting condition, and learning the VD component so as to minimize the calculated difference. (17) The information processing method according to (16), comprising: obtaining a value obtained by adding the weighted VD component to the VI component as the sum of the VD component and the VI component as the sum of the VD component and the color of the subject obtained from the lighting condition corresponding to the lighting condition. (19) The information processing method according to any one of (14) to (18) above, wherein the VI component is obtained as the product of the albedo, which is the intrinsic reflectance of the subject, and the gain of the illumination intensity. (20) The information processing method according to any one of (14) to (19) above, wherein each shooting dataset includes a plurality of shooting images taken of the subject from different shooting viewpoints under the illumination conditions corresponding to the shooting dataset, as a multi-view image.

[0104] 1,2 Information Processing Device 10 3D Shape Learning Unit 20 VI Component Learning Unit 30 VD Component Learning Unit 50 Color Addition Unit 70 VI Component Prediction Unit 80 VD Component Prediction Unit DS Image Dataset

Claims

1. An information processing device comprising: a VI component prediction unit that predicts the color components of a subject that do not change depending on the shooting viewpoint and lighting conditions as the VI component; and a VD component prediction unit that predicts the color components of the subject that change according to the shooting viewpoint and lighting conditions as the VD component.

2. The information processing apparatus according to claim 1, wherein the VI component prediction unit predicts a common VI component for different lighting conditions, and the VD component prediction unit individually predicts a color component corresponding to the difference between the color of the subject and the VI component as the VD component for each lighting condition.

3. The information processing apparatus according to claim 1, comprising a color addition unit that obtains the synthesis ratio of a plurality of lighting conditions to be synthesized, synthesizes the individual VD components predicted for each lighting condition based on the synthesis ratio, and predicts the color component obtained by synthesis as the VD component in a relighting environment in which each lighting condition is synthesized.

4. A computer-based information processing method comprising predicting the color components of a subject that do not change depending on the shooting viewpoint and lighting conditions as the VI component, and predicting the color components of the subject that change depending on the shooting viewpoint and lighting conditions as the VD component.

5. The information processing method according to claim 4, comprising predicting a common VI component for different lighting conditions, and individually predicting a color component corresponding to the difference between the color of the subject and the VI component as the VD component for each lighting condition.

6. The information processing method according to claim 4, comprising: obtaining the synthesis ratio of multiple lighting conditions to be synthesized; synthesizing the individual VD components predicted for each lighting condition based on the synthesis ratio; and predicting the color components obtained by synthesis as the VD components in a relighting environment in which each lighting condition is synthesized.

7. An information processing device comprising: a VI component learning unit that learns the color components of a subject that do not change depending on the shooting viewpoint and the lighting conditions as VI components using multiple shooting datasets with different lighting conditions; and a VD component learning unit that learns the color components of the subject that change according to the shooting viewpoint and the lighting conditions as VD components using the multiple shooting datasets.

8. The information processing apparatus according to claim 7, wherein the VI component learning unit learns the VI component common to each of the multiple shooting datasets using the plurality of shooting datasets, and the VD component learning unit acquires individual VD components corresponding to individual lighting conditions based on the learning results for each of the shooting datasets.

9. The information processing apparatus according to claim 8, wherein the VD component learning unit calculates the difference between the color shown by the sum of the VD component and the VI component for each lighting condition and the color of the subject obtained from the shooting dataset corresponding to the lighting condition, and learns the VD component so as to minimize the calculated difference.

10. The information processing apparatus according to claim 9, wherein the VD component learning unit obtains a value obtained by adding the weighted VD component to the VI component as the summation result.

11. The information processing apparatus according to claim 7, comprising a three-dimensional shape learning unit that acquires information relating to the three-dimensional shape of the subject as shape information based on the learning results using the plurality of image datasets, wherein the VI component learning unit and the VD component learning unit each learn the VI component and the VD component using the shape information.

12. The information processing apparatus according to claim 7, wherein the VI component learning unit acquires the VI component as the product of the albedo, which is the intrinsic reflectance of the subject, and the gain of the illumination intensity.

13. The information processing apparatus according to claim 7, wherein each shooting dataset includes a plurality of shooting images as a multi-view image, each of which the subject is photographed from different shooting viewpoints under the lighting conditions corresponding to the shooting dataset.

14. A computer-based information processing method comprising: learning the color components of a subject that do not change depending on the shooting viewpoint and the lighting conditions as the VI component using multiple shooting datasets with different lighting conditions; and learning the color components of the subject that change depending on the shooting viewpoint and the lighting conditions as the VD component using the multiple shooting datasets.

15. The information processing method according to claim 14, comprising: learning the VI component common to each of the multiple shooting datasets using the plurality of shooting datasets; and obtaining individual VD components corresponding to individual lighting conditions based on the learning results for each of the shooting datasets.

16. The information processing method according to claim 15, comprising: calculating the difference between the color shown by the sum of the VD component and the VI component for each lighting condition and the color of the subject obtained from the shooting dataset corresponding to the lighting condition; and learning the VD component so as to minimize the calculated difference.

17. The information processing method according to claim 16, comprising obtaining a value obtained by adding the weighted VD component to the VI component as the summation result.

18. The information processing method according to claim 14, comprising: acquiring information regarding the three-dimensional shape of the subject as shape information based on the learning results using the plurality of image datasets; and performing learning of the VI component and the VD component, respectively, using the shape information.

19. The information processing method according to claim 14, comprising obtaining the VI component as the product of the albedo, which is the intrinsic reflectance of the subject, and the gain of the illumination intensity.

20. The information processing method according to claim 14, wherein each shooting dataset includes a plurality of shooting images as a multi-view image obtained by shooting the subject from different shooting viewpoints under the lighting conditions corresponding to the shooting dataset.

Images