Information processing device, information processing method, and program

Abstract

Accurately estimate spatial information. [Solution] The information processing device 100 according to the present disclosure acquires a plurality of captured images obtained by capturing an object from a plurality of viewpoints and camera parameters corresponding to each of the plurality of viewpoints in the capturing, acquires information indicating the transparent region in each of the plurality of captured images, sets a plurality of different background colors, generates a training image corresponding to each of the plurality of background colors based on the captured images and the transparent region, and learns the spatial information based on the difference between the color value of the composite color obtained by accumulating spatial information based on the camera parameters and compositing each of the plurality of background colors for each background color and the color value of the training image corresponding to each of the plurality of background colors for each background color.

Description

[Technical Field]

[0001] This disclosure relates to information processing technology for modeling a target space. [Background technology]

[0002] There is a technique for estimating spatial information about an object in a target space based on multiple images obtained by capturing from multiple viewpoints (hereinafter referred to as "multi-view images"). Here, spatial information refers to, for example, the radiance field, which represents the volume density of an object relative to its position in space and its color in each direction. By using the estimated radiance field, it is possible to generate an image (hereinafter referred to as "virtual viewpoint image") that corresponds to how the object would appear when viewed from an arbitrary virtual viewpoint (hereinafter referred to as "virtual viewpoint"). Hereafter, the space for which the radiance field is estimated will be referred to as a "scene".

[0003] Patent Document 1 discloses a technique for estimating the radiance field using deep learning with multi-viewpoint images as training data, and then calculating the pixel values ​​of a virtual viewpoint image by integrating colors along light rays starting from an arbitrary viewpoint based on the estimated radiance field. Such a pixel value calculation process is generally called volume rendering. In the deep learning disclosed in Patent Document 1, first, volume rendering is performed to generate a virtual viewpoint image in which the virtual viewpoint is the same viewpoint as the viewpoint from which the captured image was taken (hereinafter referred to as the "imaging viewpoint"). Subsequently, the deep learning is performed using the difference between the pixel values ​​of the generated virtual viewpoint image and the pixel values ​​of the captured image as the loss. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Special Publication No. 2023-543538 [Overview of the Initiative] [Problems that the invention aims to solve]

[0005] The technology disclosed in Patent Document 1 (hereinafter referred to as "prior art") originally learns the entire space of the object to be imaged as a whole, and deep learning is performed on the space including not only the target object but also its background and other objects. Furthermore, the accuracy of the virtual viewpoint image generated by the prior art decreases as the space to be learned (hereinafter referred to as "learning domain") widens, assuming that the number of parameters representing the radiance field and the number of light rays sampling the space remain constant. On the other hand, the processing time and memory capacity required for learning the radiance field increase as the number of parameters and light rays increases. Therefore, it is desirable that the learning domain be limited to the narrowest possible space that contains the target object to be reproduced as an image on the virtual viewpoint image.

[0006] When the learning domain is limited to a narrow space, the background and other objects other than the target object are excluded from the learning domain. As a result, when volume rendering is performed using the same viewpoint as the imaging viewpoint as the virtual viewpoint, there may be instances where rays with no volume density and color exist at all points within the learning domain on the rays corresponding to pixels in the virtual viewpoint image. In contrast, the captured image contains images of the background or other objects that are not included in the learning domain. Therefore, a gap may occur between the spatial information corresponding to the learning domain and the captured image. As a way to resolve such a gap, it is conceivable to replace the color of the region within the learning domain that does not contain the target object (hereinafter referred to as the "transparent region") with the background color in the virtual space (hereinafter referred to as the "learning background color") for each captured image constituting the multi-view image, and then perform the learning process.

[0007] However, in conventional techniques, regardless of whether no objects exist on the light ray or whether an object of the same color as the training background exists on the light ray, the color of the pixels in the virtual viewpoint image corresponding to the light ray becomes equal to the training background color. Therefore, there may be no difference in learning loss between a radiance field where the transmission region is correctly learned and a radiance field where the transmission region is incorrectly learned as if an object of the same color as the training background exists within the transmission region. In other words, in such cases, learning may converge to the incorrect radiance field, which leads to the problem of artifacts occurring in the virtual viewpoint image when a viewpoint different from the imaging viewpoint is used as the virtual viewpoint.

[0008] Therefore, this disclosure aims to provide a technology that can accurately estimate spatial information corresponding to the learning domain. [Means for solving the problem]

[0009] The information processing device according to this disclosure includes: an imaging data acquisition means for acquiring a plurality of captured images obtained by imaging an object from a plurality of viewpoints and camera parameters corresponding to each of the plurality of viewpoints in the imaging; a region acquisition means for acquiring information indicating a transparent region in each of the plurality of captured images; a background color setting means for setting a plurality of different background colors; a teacher generation means for generating a teacher image corresponding to each of the plurality of background colors based on the captured images and the transparent region; and a learning means for learning the spatial information based on the difference between the color value of a composite color obtained by accumulating spatial information based on the camera parameters and each of the plurality of background colors for each background color and the color value of the teacher image corresponding to each of the plurality of background colors for each background color. [Effects of the Invention]

[0010] The technology disclosed herein can accurately estimate spatial information related to the learning domain. [Brief explanation of the drawing]

[0011] [Figure 1] It is a block diagram showing an example of the hardware configuration of an information processing apparatus according to the first embodiment. [Figure 2] It is a block diagram showing an example of the logical configuration of an information processing apparatus according to the first embodiment. [Figure 3] It is a diagram showing an example of the arrangement of a target object, a background object, a learning area, and an imaging device according to the first embodiment. [Figure 4] It is a diagram showing an example of a captured image according to the first embodiment. [Figure 5] It is a flowchart showing an example of the processing flow of an information processing apparatus according to the first embodiment. [Figure 6] It is a diagram showing an example of a GUI according to the first embodiment. [Figure 7] It is a diagram showing an example of an object area mask according to the first embodiment. [Figure 8] It is a diagram showing an example of a teacher image according to the first embodiment. [Figure 9] It is a diagram showing an example of a light ray r according to the first embodiment. [Figure 10] It is a diagram showing an example of an estimated image calculated based on an incorrect radiance field. [Figure 11] It is a diagram showing an example of a GUI according to the first embodiment.

Mode for Carrying Out the Invention

[0012] Hereinafter, embodiments according to the technology of the present disclosure will be described with reference to the accompanying drawings. Note that the following embodiments do not limit the means for solving the problems according to the present disclosure, and not all combinations of the features described in the following embodiments are essential for the means for solving the problems according to the present disclosure. The same components will be described with the same reference numerals.

[0013] [First Embodiment] In this embodiment, a method is described in which multiple training background colors are set, a training image and a virtual viewpoint image are generated for each training background color, and the loss in training is calculated based on the difference between the training image and the virtual viewpoint image for each training background color.

[0014] <Hardware configuration of the information processing device> Figure 1 is a block diagram showing an example of the hardware configuration of an information processing device 100 according to the first embodiment. The information processing device 100 has a CPU 101, RAM 102, ROM 103, serial I / F (interface) 104, VC (video card) 105, and general-purpose I / F 106 as its hardware configuration. Each part of the information processing device 100's hardware configuration is connected to each other so as to be able to communicate via a system bus 107. The CPU 101 uses RAM 102 as work memory to execute the OS (operating system) and various programs stored in ROM 103 or storage device 111, etc. The CPU 101 controls the entire information processing device 100 via the system bus 107 by executing various programs. The processing of each step shown in the flowchart described later is realized by the program code stored in ROM 103 or storage device 111, etc. being loaded into RAM 102, and then executed by the CPU 101.

[0015] The serial I / F 104 is an interface configured with serial ATA or the like, and connects the information processing device 100 and the storage device 111 via the serial bus 108. The storage device 111 is a large-capacity storage device such as an HDD (hard disk drive) or SSD (solid state drive). In this embodiment, the storage device 111 is described as an external device of the information processing device 100, but the information processing device 100 may include the storage device 111 internally. The VC 105 receives control signals from the CPU 101 and outputs signals related to the display image to the display device 112 via the serial bus 109. The display device 112 is configured with a liquid crystal display or the like and displays the display image based on the signals related to the display image output by the information processing device 100. The general-purpose I / F 106 is connected to an input device 113 such as a mouse or keyboard via the serial bus 110 and receives input signals from the input device 113.

[0016] The CPU 101 displays a GUI (Graphical User Interface) provided by a program on the display device 112 via the VC 105 and receives input signals indicating user instructions via the input device 113. The information processing device 100 is implemented, for example, by a desktop PC (personal computer). The information processing device 100 may also be implemented by a notebook PC integrated with the display device 112, or a tablet PC, etc. The storage device 111 can also be implemented by a media (portable storage medium) and a drive such as a disk drive or a reader such as a memory card reader for accessing the media. The media may include FD (Flexible Disk), CD (Compact Disc)-ROM, DVD (Digital Versatile Disc), USB (Universal Serial Bus) memory, MO (magneto-optical) disk, or flash memory, etc.

[0017] <Logical configuration of information processing device> Figure 2 is a block diagram showing an example of the logical configuration of the information processing device 100 according to the first embodiment. The information processing device 100 has, as a logical configuration, an imaging data acquisition unit 201, a learning area setting unit 202, a transparency area acquisition unit 203, a background color setting unit 204, a teacher image generation unit 205, a learning unit 206, and an output unit 207. Each unit of the logical configuration of the information processing device 100 is realized by the CPU 101 executing a program stored in ROM 103 or the like, using RAM 102 as work memory. Note that not all of the processes shown below necessarily need to be executed by the CPU 101, and the information processing device 100 may be configured so that some or all of the processes are executed by one or more processing circuits other than the CPU 101.

[0018] The imaging data acquisition unit 201 acquires data from multiple captured images (multi-view images) obtained by capturing objects present in the scene from various imaging viewpoints, based on instructions from the user input via the input device 113. Hereinafter, the data of the captured images acquired by the imaging data acquisition unit 201 will be described as image data in RGB image format. The imaging data acquisition unit 201 may acquire multi-view image data by directly acquiring the data of the captured images output by the imaging device from the imaging device, or it may acquire multi-view image data by reading the captured image data that has been stored in advance from the storage device 111 or the like. The acquired multi-view image data is transmitted to the training image generation unit 205.

[0019] Furthermore, the imaging data acquisition unit 201 acquires the camera parameters of the imaging device that captured each image constituting the multi-view image. Hereinafter, the camera parameters acquired by the imaging data acquisition unit 201 will be described as including the internal parameters, external parameters, and distortion parameters of the imaging device. Internal parameters are parameters that represent the position of the principal point of the imaging device and the focal length of the lens of the imaging device. External parameters are parameters that represent the position of the imaging device and the optical axis direction of the imaging device, i.e., the orientation of the imaging device. Distortion parameters are parameters that represent the distortion of the optical system, such as the lens of the imaging device. The imaging data acquisition unit 201 may acquire the camera parameters held by each imaging device by requesting them from the imaging device, or it may acquire the camera parameters by reading camera parameters that have been stored in advance in the storage device 111 or the like. The acquired camera parameters of each imaging device are transmitted to the learning unit 206.

[0020] The learning area setting unit 202 sets the position and size of the learning area based on instructions from the user input via the input device 113. The learning area setting unit 202 may also set the position and size of the learning area by reading information indicating the position and size of the learning area that has been previously stored in the storage device 111 or the like. Hereinafter, the shape of the learning area will be described as a rectangular parallelepiped composed of planes orthogonal to the directions of the three coordinate axes representing three-dimensional space. The information indicating the set position and size of the learning area is transmitted to the learning unit 206.

[0021] Figure 3 is a diagram showing an example of the arrangement of a target object 301, a background object 302, a learning area 303, and an imaging device 310 that images these objects 301 and 302, all located within a scene according to the first embodiment. Figure 4 is a diagram showing an example of an image obtained by imaging with the imaging device 310 according to the first embodiment. Specifically, Figure 4(a) shows an example of an image 410 obtained by imaging with imaging device 311. Figure 4(b) shows an example of an image 420 obtained by imaging with imaging device 312. Figure 4(c) shows an example of an image 430 obtained by imaging with imaging device 313. Images 410, 420, and 430 include, in order, images 411, 421, and 431 of the target object 301 located within the learning area 303, and images 412, 422, and 432 of the background object 302 located outside the learning area 303.

[0022] The transparency region acquisition unit 203 acquires object region masks corresponding to each captured image that makes up the multi-view image. An object region mask is an image that shows the region (hereinafter referred to as "object region") corresponding to the image of an object present within the learning region in the captured image. Details of the object region mask acquisition process in the transparency region acquisition unit 203 will be described later. The acquired object region mask data is transmitted to the teacher image generation unit 205. The background color setting unit 204 sets multiple learning background colors. Details of the learning background color setting process in the background color setting unit 204 will be described later. Information indicating the set multiple learning background colors is transmitted to the teacher image generation unit 205 and the learning unit 206.

[0023] The training image generation unit 205 generates training images to be used for learning, as described later, based on the multiple captured images constituting the multi-view image acquired by the imaging data acquisition unit 201 and the object region masks corresponding to each captured image acquired by the transparency region acquisition unit 203. Specifically, the training image generation unit 205 generates training images corresponding to each captured image constituting the multi-view image and each learning background color set by the background color setting unit 204, based on the multi-view image and the object region masks. Details of the training image generation process in the training image generation unit 205 will be described later. The generated training image data is transmitted to the learning unit 206 and the output unit 207.

[0024] The learning unit 206 estimates spatial information corresponding to the learning area set by the learning area setting unit 202. Hereinafter, the learning unit 206 will be described as estimating the radiance field corresponding to the learning area as spatial information corresponding to the learning area, as an example. Specifically, the learning unit 206 estimates the radiance field corresponding to the learning area based on camera parameters acquired by the imaging data acquisition unit 201, multiple learning background colors set by the background color setting unit 204, and a teacher image generated by the teacher image generation unit 205. Details of the radiance field estimation process in the learning unit 206 will be described later. Information indicating the radiance field estimated by the learning unit 206 is transmitted to the output unit 207 along with the teacher image data used to estimate the radiance field and information indicating the multiple learning background colors. The output unit 207 outputs the radiance field estimated by the learning unit 206, as well as the teacher image used to estimate the radiance field and information regarding the multiple learning background colors, to a display device 112 or a storage device 111, etc. Details of the output processing in the output unit 207 will be described later.

[0025] <Operation of the Information Processing Device> Figure 5 is a flowchart showing an example of the processing flow of the information processing device 100 according to the first embodiment. The processing shown in the flowchart in Figure 5 is realized by the CPU 101 loading a program stored in the ROM 103 or the like into the RAM 102 and executing it. In the following description, the symbol "S" means step. First, at S501, the imaging data acquisition unit 201 acquires data of multiple imaging images constituting a multi-view image and camera parameters used to capture each imaging image (hereinafter referred to as "camera parameters of imaging images") based on instructions from the user. Next, at S502, the learning area setting unit 202 sets the learning area based on instructions from the user.

[0026] Figure 6 shows an example of the GUI 600 displayed on the display device 112 according to the first embodiment. User instructions in S501 and S502 are received via the GUI 600 shown as an example in Figure 6. The GUI 600 includes data path setting fields 601 and 602, a learning area setting field 603, and a button 604. The data path setting field 601 is a field where the user inputs the data path to the data of the multi-view image. The data path setting field 602 is a field where the user inputs the data path to the camera parameters of each captured image.

[0027] The learning area setting field 603 is a field where the user inputs the coordinates corresponding to the center position of the learning area and the lengths of the sides in each coordinate axis direction of the rectangular prism to be set as the learning area. Hereafter, it will be explained assuming that the user has prior knowledge of the approximate position and size of objects in the scene. Note that the position and size of objects in the scene may be estimated by the learning area setting unit 202, etc., based on multi-view images, and the learning area setting unit 202 may set the position and size of the learning area according to the results of the estimation. In this case, object shape estimation techniques such as the viewing volume cross-eyed method or the stereo matching method can be used to estimate the position and size of objects. Button 604 is a button that is pressed to instruct the execution of each process in the information processing device 100. When button 604 is pressed by the user, processes S501 and S502 are executed.

[0028] Following S502, in S503, the transparency region acquisition unit 203 acquires an object region mask corresponding to each of the captured images acquired in S501. Specifically, first, for each captured image, the transparency region acquisition unit 203 acquires a difference image showing the difference between the captured image and the background image. The background image is an image prepared in advance, for example, by capturing a scene in which no objects exist within the learning region, and the background image data is acquired by reading it from the storage device 111 or the like based on instructions from the user. Next, the transparency region acquisition unit 203 extracts the region containing pixels whose pixel values ​​in the difference image are above a predetermined threshold as the object region. Subsequently, the transparency region acquisition unit 203 generates an image in which, for example, the pixel values ​​of pixels included in the object region are set to 1 and the pixel values ​​of pixels outside the object region are set to 0, and acquires the generated image as the object region mask.

[0029] Figure 7 shows an example of an object region mask according to the first embodiment. Specifically, Figure 7(a) shows an example of an object region mask 710 corresponding to the captured image 410 shown in Figure 4(a). Figure 7(b) shows an example of an object region mask 720 corresponding to the captured image 420 shown in Figure 4(b). Figure 7(c) shows an example of an object region mask 730 corresponding to the captured image 430 shown in Figure 4(c). In the captured images 410, 420, and 430 shown in Figure 7, pixels in the object region with a pixel value of 1 are represented in white, and pixels outside the object region with a pixel value of 0 are represented in black. Furthermore, the object regions 711, 721, and 731 in the object region masks 710, 720, and 730 correspond to the regions of images 411, 421, and 431 of the object 301 that exist within the learning region in the captured images 410, 420, and 430. The transparent area acquisition unit 203 may acquire the data by reading pre-generated object area mask data from the storage device 111 or the like, based on instructions from the user.

[0030] Following S503, in S504, the background color setting unit 204 sets multiple colors that are as far apart as possible from each other in a predetermined color space such as the RGB space as learning background colors. Hereafter, the background color setting unit 204 will be described as setting K (K≧2) learning background colors, and the k-th learning background color among the K (K≧2) learning background colors will be referred to as "background color k". Furthermore, hereafter, as an example, the background color setting unit 204 will be described as setting two learning background colors, with white set as background color 1 and black set as background color 2 so that the distance between the two colors in the RGB space is maximized.

[0031] Next, in S505, the training image generation unit 205 generates a training image based on the multi-view image acquired in S501, the object region masks corresponding to each captured image constituting the multi-view image acquired in S503, and the multiple training background colors set in S504. Specifically, the training image generation unit 205 generates a training image by replacing the color of the image region corresponding to the transparent region in each captured image with the training background color, based on the object region masks corresponding to each captured image constituting the multi-view image and the multiple training background colors. More specifically, the training image generation unit 205 generates a training image based on the RGB values ​​c of each pixel in the training image corresponding to the captured image obtained by the nth imaging device 310 (hereinafter referred to as the "nth captured image"). GTk (n,u,v) are determined, for example, using formula (1).

[0032]

number

[0033] Figure 8 shows an example of a training image according to the first embodiment. Specifically, Figure 8(a) shows an example of a training image 811 with background color 1 corresponding to the captured image 410 in Figure 4(a). Figure 8(b) shows an example of a training image 812 with background color 2 corresponding to the captured image 410 in Figure 4(a). Figure 8(c) shows an example of a training image 821 with background color 1 corresponding to the captured image 420 in Figure 4(b). Figure 8(d) shows an example of a training image 822 with background color 2 corresponding to the captured image 420 in Figure 4(b). Figure 8(e) shows an example of a training image 831 with background color 1 corresponding to the captured image 430 in Figure 4(c). Figure 8(f) shows an example of a training image 832 with background color 2 corresponding to the captured image 430 in Figure 4(c).

[0034] Following S505, in S506, the learning unit 206 estimates the radiance field using multiple learning background colors set in S504 for the learning region set in S502, based on the camera parameters acquired in S501 and the training image generated in S505. Hereafter, the learning unit 206 uses the function F shown as an example in equation (2). Θ This will be explained as estimating the radiance field modeled using [a specific method / tool].

[0035]

number

[0036] As an example, consider the function F Θ This model is implemented using an MLP (Multi-layer perceptron), and the learning unit 206 is described as estimating the radiance field by performing machine learning on this model. In this case, the radiance field is expressed as a parameter of the MLP, that is, as a weight coefficient related to the nodes that constitute the MLP. The estimated MLP parameters are stored in a memory area allocated in RAM 102 or elsewhere. Note that the function F Θ This is not limited to what is realized by MLP, but may also be realized by, for example, a sparse voxel grid represented by the coefficients of spherical harmonics representing color and the volume density.

[0037] In the processing of S506, first, the learning unit 206 calculates the pixel values ​​of virtual viewpoint images, for each learning background color set in S504, based on the output from the model described above, where the virtual viewpoint is the same viewpoint from which each image acquired in S501 was captured. Hereafter, these virtual viewpoint images will be referred to as "estimated images". Next, the learning unit 206 optimizes the MLP parameters so that the pixel values ​​of the estimated images calculated for each learning background color approach the pixel values ​​of the training image generated in S505. Specifically, the learning unit 206 uses the Loss shown in equation (3) as the loss, and learns the model described above by repeatedly performing loss calculation and MLP parameter updates using backpropagation.

[0038]

number

[0039] [Number] Here, i and N represent, in order, the index and the number of sampling points arranged within the learning region on the ray r. c i and σ i represent, in order, the RGB value and the volume density output from the above-described model for the i-th sampling point. δ i represents the distance from the i-th sampling point to the (i + 1)-th sampling point. In these Equations (4) to (7), c VR (r) and α VR (r) represent, in order, the RGB value and the opacity as the integrated value of color, obtained by performing volume rendering based on the above-described model for the ray r. Also, T i represents the cumulative transmittance from the position of the imaging device 310 to the sampling point. Hereinafter, the estimated image calculated for the background color k will be described as the "estimated image of the background color k".

[0040] Figure 10 shows an example of an estimated image calculated based on an incorrect radiance field. Specifically, Figure 10(a) shows an example of an object indicated by the incorrect radiance field. It includes object 301 that actually exists within the learning region 303, and objects 1001 and 1002 that do not actually exist. Figure 10(b) shows an example of an estimated image 1011 with background color 1, corresponding to the captured image 410 in Figure 4(a). Figure 10(c) shows an example of an estimated image 1012 with background color 2, corresponding to the captured image 410 in Figure 4(a). Figure 10(d) shows an example of an estimated image 1021 with background color 1, corresponding to the captured image 420 in Figure 4(b). Figure 10(e) shows an example of an estimated image 1022 with background color 2, corresponding to the captured image 420 in Figure 4(b). Figure 10(f) shows an example of an estimated image 1031 with background color 1, corresponding to the captured image 430 in Figure 4(c). Figure 10(g) shows an example of an estimated image 1032 with background color 2, corresponding to the captured image 430 in Figure 4(c).

[0041] As shown in Figures 10(b), (d), and (f), the images 1013, 1023, and 1033 of object 1002, which is a color close to black (background color 2), are included in the white, transparent regions of the estimated images 1011, 1021, and 1031 for background color 1. In such cases, a large difference in pixel values ​​occurs between the estimated images 1011, 1021, and 1031 and the training images 811, 821, and 831 for background color 1, which do not include the images 1013, 1023, and 1033, as shown in Figures 8(a), (c), and (e). Also, as shown in Figures 10(c), (e), and (g), the images 1014, 1024, and 1034 of object 1001, which is a color close to white (background color 1), are included in the black, transparent regions of the estimated images 1012, 1022, and 1032 for background color 2. In such cases, a large difference in pixel values ​​occurs between the estimated images 1012, 1022, and 1032 and the training images 812, 822, and 832 with background color 2 that do not include the images 1014, 1024, and 1034, as shown in Figures 8(b), (d), and (f).

[0042] In other words, if the space represented by the radiance field includes objects that do not actually exist, the difference in pixel values ​​between the estimated image and the training image becomes large in one of the multiple training background colors, and the loss in equation (3) becomes a large value. Therefore, by training in a way that reduces the loss in equation (3), the above model becomes less likely to converge to an incorrect state. This makes it possible to accurately estimate the correct radiance field, that is, a radiance field in which objects that do not actually exist in the space corresponding to the transmission region do not exist.

[0043] Following S506, in S507, the output unit 207 outputs difference information showing the difference between the estimated image obtained by volume rendering based on the radiance field estimated in S506 and the training image used to estimate the radiance field. Specifically, for example, the output unit 207 outputs a signal showing the difference information to the display device 112, causing the display device 112 to display the difference information.

[0044] Figure 11 shows an example of a GUI 1100 displayed on a display device 112 by the output unit 207 according to the first embodiment. The GUI 1100 shown as an example in Figure 11 includes background color display fields 1101, 1102, background color score display fields 1103, 1104, background color image display areas 1105, 1106, and camera ID setting field 1107. Background color display fields 1101 and 1102 are fields that display information about the training background color used to estimate the radiance field in S506. Background color score display fields 1103 and 1104 are fields that display information (difference information) showing the difference between the training image and the estimated image, calculated for each training background color. In background color score display fields 1103 and 1104, the difference information displays, for example, a value indicating the magnitude of the mean squared error of the pixel values.

[0045] The camera ID setting field 1107 is a UI (user interface) component for the user to select the desired imaging device from among multiple imaging devices 310. The background color image display areas 1105 and 1106 are areas where estimated images of virtual viewpoints corresponding to the imaging viewpoint of the imaging device 310 selected by the user via the camera ID setting field 1107 are displayed, and these areas display estimated images of different learning background colors. The content of the difference information to be displayed and the method of displaying the difference information are not limited to the examples described above. For example, the information processing device 100 may generate and display difference images showing the difference between the teacher image and the estimated image for each learning background color, or it may display the difference images side by side with at least one of the teacher image and the estimated image. In addition, the information processing device 100 may display difference information as needed during the learning process in S506. With such a display, the user can easily grasp whether or not there are any objects that do not actually exist in the space indicated by the radiance field, based on the displayed difference information.

[0046] After S507, in S508, the output unit 207 outputs information regarding the radiance field estimated in S506, and stores this information in the storage device 111, for example. The destination of the output information regarding the radiance field is not limited to the storage device 111; for example, the output unit 207 may output the information to another external device different from the information processing device 100. After S508, the information processing device 100 completes the processing shown in the flowchart in Figure 5.

[0047] With the information processing device 100 configured as described above, the radiance field within the learning domain can be estimated with high accuracy. As a result, artifacts that occur in the virtual viewpoint image obtained by rendering using the radiance field can be suppressed.

[0048] In this embodiment, the case where the captured image is an RGB image was described as an example, but the captured image may be an image represented in other formats such as a grayscale image, an XYZ image, or a YUV image.

[0049] Furthermore, in this embodiment, as an example, the case in S504 where the background color setting unit 204 sets two colors, white and black, as the learning background colors has been described. However, the background color setting unit 204 may set two or more other colors as the learning background colors. More preferably, it is desirable for the background color setting unit 204 to set multiple colors as the learning background colors such that the sum or minimum value of the distances between them is maximized in the color space used when calculating the loss, i.e., the color space in which the teacher image and the estimated image are represented. In that case, if there is an object that does not actually exist but has a color close to one of the learning background colors in the space indicated by the radiance field, the difference between the estimated image based on the other learning background colors and the teacher image will appear larger. Therefore, it becomes more difficult to converge to an incorrect radiance field.

[0050] Furthermore, the background color setting unit 204 may set multiple learning background colors of different colors for each pixel position or for each captured image. For example, the background color setting unit 204 may use the color of each pixel in the background image described above as one of the multiple learning background colors. In that case, the learning unit 206 can use the captured image as is as part of the training image.

[0051] Furthermore, the background color setting unit 204 may set different numbers of learning background colors for each pixel position or for each captured image. For example, the background color setting unit 204 may set multiple learning background colors for pixels in the transparent region as described above, and set the color of each pixel in the background image as the learning background color for pixels in the object region. In this case, the information processing device 100 can accurately estimate the radiance field of the learning region, which includes semi-transparent objects that allow the background color to show through, while suppressing the occurrence of objects that do not actually exist in the transparent region. Also, for example, if it is known that the objects included in the learning region are opaque, the background color setting unit 204 may set a color that is significantly different from the color of each pixel in the captured image, such as the complementary color of that color, as the learning background color for pixels in the object region. In this case, if the object indicated by the radiance field is semi-transparent or transparent, the difference between the training image and the estimated image will be large. Therefore, the information processing device 100 can suppress the object indicated by the estimated radiance field from being incorrectly semi-transparent or missing.

[0052] [Second Embodiment] In the first embodiment, a method was described in which all of the set learning background colors were used each time the loss calculation was performed during learning. In this embodiment, a method is described in which the learning background color used for loss calculation is changed each time the loss calculation is performed during learning.

[0053] The hardware configuration, logical configuration, and overall flow of processing of the information processing device 100 according to this embodiment are equivalent to those of the information processing device 100 according to the first embodiment. However, the method of calculating the loss in S506 of the information processing device 100 according to this embodiment differs from that of the information processing device 100 according to the first embodiment. The following will mainly describe the differences between this embodiment and the first embodiment. Components identical to those in the first embodiment will be denoted by the same reference numerals.

[0054] In this embodiment, when the learning unit 206 repeatedly calculates the loss during learning, it selects one background color k from a set of multiple learning background colors each time. t Select this option and, for example, use formula (8) to calculate the loss shown as Loss'.

[0055]

number

[0056] The learning unit 206 ensures that the number of times each learning background color is selected is not biased during the repeated loss calculation process in learning, by selecting the background color k in each iteration. t Select the following. For example, if there are two background colors for learning, white and black, the learning unit 206 will use white for odd-numbered times and black for even-numbered times. t You can select it as such. Furthermore, the learning unit 206 randomly selects a background color k from among several learning background colors each time, based on a random number table, etc. t You may also select this option. For example, the background color k selected using formula (8) t When the loss' is calculated only for the background colors used for training, the computational complexity and memory requirements are lower compared to when the loss' is calculated for all training background colors using formula (3).

[0057] With the information processing device 100 configured as described above, the amount of computation and memory required for each calculation of the loss can be reduced compared to the information processing device 100 according to the first embodiment, while accurately estimating the radiance field within the learning domain.

[0058] In this embodiment, the learning unit 206 has been described as selecting one learning background color from among multiple learning background colors and using it in the loss calculation process. However, the learning unit 206 may select two or more colors from among multiple learning background colors and use them in the loss calculation process. Furthermore, the number of learning background colors used in the loss calculation process may differ for each iteration of the calculation process. For example, if the learning unit 206 uses two or more selected learning background colors in the loss calculation process, the learning unit 206 may use, for example, formula (3) to calculate the loss as the sum of the differences between the estimated image and the training image obtained for each selected learning background color.

[0059] [Other embodiments] In the embodiments described above, a method for estimating a radiance field representing volume density relative to a position and color for each direction as spatial information was explained. However, the information represented by spatial information is not limited to a radiance field. For example, spatial information may represent volume density relative to a position and color independent of direction, or it may be a Signed Distance Field representing color relative to a position and the distance to the surface of an object. Furthermore, the technology of this disclosure is applicable to various methods such as Gaussian Splatting, in which the pixel values ​​of the estimated image are determined based on the color and opacity indicated by the spatial information.

[0060] The technology of this disclosure can also be implemented by supplying a program that implements one or more of the functions of the embodiments described above to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. Furthermore, the technology of this disclosure can also be implemented by a processing circuit such as an ASIC that implements one or more functions.

[0061] Furthermore, within the scope of this disclosure, the technologies described herein allow for free combination of each embodiment, modification of any component of each embodiment, or omission of any component in each embodiment.

[0062] [Structure of this disclosure] This disclosure includes the following configurations, methods, and programs.

[0063] <Configuration 1> An imaging data acquisition means that acquires multiple images obtained by imaging an object from multiple viewpoints, and camera parameters corresponding to each of the multiple viewpoints in the imaging, A region acquisition means for acquiring information indicating the transparent region in each of the plurality of captured images, A background color setting means for setting multiple background colors that are different from each other, A teacher generation means that generates a teacher image corresponding to each of the plurality of background colors based on the captured image and the transparent region, A learning means that learns the spatial information based on the difference between the color value of a composite color obtained by accumulating spatial information based on the camera parameters and each of the multiple background colors, and the color value of the training image corresponding to each of the multiple background colors for each background color. An information processing device characterized by having the following features.

[0064] <Configuration 2> The system further includes output means for outputting information regarding the difference for each background color. An information processing device as described in Configuration 1, characterized by the above.

[0065] <Structure 3> The learning means learns the spatial information such that the sum of the differences for each background color becomes smaller. An information processing device according to configuration 1 or 2, characterized by the above.

[0066] <Structure 4> The learning means repeatedly performs the process of calculating the difference between the color value of the composite color and the color value of the training image, and each time the difference calculation process is repeated, it selects the background color to be used in the difference calculation process from among the plurality of background colors. An information processing device according to any one of configurations 1 to 3 characterized by the above.

[0067] <Composition 5> The background color setting means sets a plurality of colors with large intervals between them in the color space as the plurality of background colors. An information processing device according to any one of configurations 1 to 4 characterized by the above.

[0068] <Composition 6> The background color setting means sets a plurality of colors as the plurality of background colors, the plurality of colors having the maximum interval between colors in the color space. The information processing device described in configuration 5, characterized by the above.

[0069] <Composition 7> The background color setting means sets the color of the background image obtained by imaging when the object is not present in the learning area set within the space where the object exists as one of the plurality of background colors. An information processing device according to any one of configurations 1 to 6 characterized by the above.

[0070] <Structure 8> The background color setting means sets a color different from the color of the captured image as one of the plurality of background colors. An information processing device according to any one of configurations 1 to 7 characterized by the above.

[0071] <Method> A process for acquiring imaging data, which involves acquiring multiple images obtained by imaging an object from multiple viewpoints, and camera parameters corresponding to each of the multiple viewpoints in the imaging, A region acquisition step is to acquire information indicating the transmission region for each of the plurality of captured images, A background color setting process involves setting multiple background colors that are different from each other, A teacher generation step that generates a teacher image corresponding to each of the plurality of background colors based on the captured image and the transparent region, A learning process in which spatial information is learned based on the difference between the color value of a composite color obtained by accumulating spatial information based on the camera parameters and each of the multiple background colors, and the color value of the training image corresponding to each of the multiple background colors for each background color. An information processing method characterized by including

[0072] <Program> A program for causing a computer to function as an information processing device described in any one of configurations 1 through 8. [Explanation of Symbols]

[0073] 100 Information Processing Devices 201 Imaging Data Acquisition Unit 203 Transparent area acquisition unit 204 Background Color Settings Section 205 Training Image Generation Unit 206 Learning Department

Claims

1. An imaging data acquisition means that acquires multiple images obtained by imaging an object from multiple viewpoints, and camera parameters corresponding to each of the multiple viewpoints in the imaging, A region acquisition means for acquiring information indicating the transparent region in each of the plurality of captured images, A background color setting means for setting multiple background colors that are different from each other, A teacher generation means that generates a teacher image corresponding to each of the plurality of background colors based on the captured image and the transparent region, A learning means that learns the spatial information based on the difference between the color value of a composite color obtained by accumulating spatial information based on the camera parameters and each of the multiple background colors, and the color value of the training image corresponding to each of the multiple background colors for each background color. An information processing device characterized by having the following features.

2. The system further includes output means for outputting information regarding the difference for each background color. The information processing apparatus according to claim 1, characterized by the following:

3. The learning means learns the spatial information such that the sum of the differences for each background color becomes smaller. The information processing apparatus according to claim 1, characterized by the following:

4. The learning means repeatedly performs the process of calculating the difference between the color value of the composite color and the color value of the training image, and each time the difference calculation process is repeated, it selects the background color to be used in the difference calculation process from among the plurality of background colors. The information processing apparatus according to claim 1, characterized by the following:

5. The background color setting means sets a plurality of colors with large intervals between them in the color space as the plurality of background colors. The information processing apparatus according to claim 1, characterized by the following:

6. The background color setting means sets a plurality of colors as the plurality of background colors, the plurality of colors having the maximum interval between colors in the color space. The information processing apparatus according to claim 5, characterized by the following:

7. The background color setting means sets the color of the background image obtained by imaging when the object is not present in the learning area set within the space where the object exists as one of the plurality of background colors. The information processing apparatus according to claim 1, characterized by the following:

8. The background color setting means sets a color different from the color of the captured image as one of the plurality of background colors. The information processing apparatus according to claim 1, characterized by the following:

9. A process for acquiring imaging data, which involves acquiring multiple images obtained by imaging an object from multiple viewpoints, and camera parameters corresponding to each of the multiple viewpoints in the imaging, A region acquisition step is to acquire information indicating the transmission region for each of the plurality of captured images, A background color setting process involves setting multiple background colors that are different from each other, A teacher generation step that generates a teacher image corresponding to each of the plurality of background colors based on the captured image and the transparent region, A learning process in which spatial information is learned based on the difference between the color value of a composite color obtained by accumulating spatial information based on the camera parameters and each of the multiple background colors, and the color value of the training image corresponding to each of the multiple background colors for each background color. An information processing method characterized by including

10. A program for causing a computer to function as an information processing device according to any one of claims 1 to 8.

Images