Information processing device, information processing method, and program

Abstract

To provide an information processing device that accurately estimates a radiance field in a learning area even when the learning area is limited to a small space.SOLUTION: An information processing device acquires data on a plurality of photographed images obtained by photographing images of an object present in a predetermined imaging space from a plurality of viewpoints, camera parameters corresponding to the respective viewpoints at the time of photographing the images, and color information on an out-of-area object that is an object present outside a learning area set in a three-dimensional space corresponding to the predetermined photographing space; and on the basis of the data on the plurality of photographed images, the camera parameters corresponding to the respective viewpoints, and the color information related to the out-of-area object, learns space information which is information related to color and volume density at each position within the learning area. The learning of the space information is performed on the basis of a difference between a color of the photographed image and a rendering color including a cumulative color obtained by accumulating the space information and a color of the out-of-area object.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to an image processing technique for generating an image corresponding to an image seen from an arbitrary viewpoint using a plurality of images obtained by capturing images from multiple viewpoints. [Background technology]

[0002] There is a technology for generating an image corresponding to an image of a target object viewed from an arbitrary viewpoint based on a plurality of captured images (hereinafter referred to as "multi-viewpoint images") obtained by capturing images from a plurality of viewpoints with known camera parameters. Hereinafter, an arbitrary viewpoint will be referred to as a "virtual viewpoint," and an image corresponding to an image viewed from the virtual viewpoint will be referred to as a "virtual viewpoint image." Patent Document 1 discloses a technology for estimating radiance fields related to a target object using multi-viewpoint images, and generating a virtual viewpoint image corresponding to the appearance of the object when viewed from a virtual viewpoint using the estimated radiance fields.

[0003] Specifically, the technology disclosed in Patent Document 1 uses deep learning, which uses pixel values of a captured image as training, to train a neural network (hereinafter referred to as "NeRF") that responds to the color and volume density of an object with respect to its position and direction in three-dimensional space. Furthermore, pixel values of the virtual viewpoint image are calculated by weighting and accumulating color values output from the NeRF along a ray determined based on the pixel position of the virtual viewpoint image, the position of the virtual viewpoint, and the line of sight direction at the virtual viewpoint. Hereinafter, information including information indicating the position of the virtual viewpoint and information indicating the line of sight direction at the virtual viewpoint will be referred to as virtual camera parameters.

[0004] The process of calculating pixel values is generally called volume rendering. In NeRF training, a virtual viewpoint image is generated by volume rendering, with the same viewpoint as the viewpoint from which the captured image was captured. The difference between the pixel values of the generated virtual viewpoint image and the pixel values of the captured image is used as the loss function. NeRF originally trains the entire three-dimensional space of the target image captured from multiple viewpoints at once, learning not only the main object but also its background and other objects. The accuracy of the virtual viewpoint image generated by NeRF decreases as the three-dimensional space to be trained (hereinafter referred to as the "training domain") becomes larger, assuming the number of neural network parameters and the number of light rays sampling the space are constant. On the other hand, the processing time and memory capacity costs required for NeRF training increase with the number of parameters or light rays. Therefore, it is desirable to limit the training domain to as small a space as possible that contains the object to be reproduced as an image in the virtual viewpoint image. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] U.S. Patent No. 1,130,8659 Summary of the Invention [Problem to be solved by the invention]

[0006] If the learning area is limited to a small space, the background and other objects will not be included in the learning area, and the colors and volume densities of the background and other objects will not be present within the learning area. As a result, the necessary colors and volume densities may not be present on the light rays corresponding to the pixels of the virtual viewpoint image within the learning area. In contrast, the captured image contains images of the background or other objects not included in the learning area, and the pixels of the captured image always have some color. As such, the information corresponding to the three-dimensional space of the learning object may not contain information indicating color, resulting in a gap between the information corresponding to the three-dimensional space of the learning object and the information corresponding to the captured image, which always contains color information. This makes it difficult to accurately estimate the radiance field within the learning area.

[0007] An object of the present disclosure is to provide a technology that can accurately estimate a radiance field within a learning area even when the learning area is limited to a small space. [Means for solving the problem]

[0008] The information processing device according to the present disclosure includes an imaging data acquisition means for acquiring data on a plurality of captured images obtained by capturing images of an object existing in a predetermined imaging space from a plurality of viewpoints and camera parameters corresponding to each of the plurality of viewpoints at the time of capturing the images; an outside-area color data acquisition means for acquiring color information on out-of-area objects, which are objects existing outside a learning area set in a three-dimensional space corresponding to the predetermined imaging space; and a learning means for learning spatial information, which is information regarding color and volume density at each position within the learning area, based on the data on the plurality of captured images, the camera parameters corresponding to each of the plurality of viewpoints, and the color information on the out-of-area objects, the learning means learning the spatial information based on the difference between the color of the captured image and a rendering color including a cumulative color obtained by accumulating the spatial information and the color of the out-of-area objects. [Effects of the Invention]

[0009] According to the present disclosure, even when the learning area is limited to a small space, the radiance field within the learning area can be estimated with high accuracy. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of a virtual viewpoint image generated by a conventional NeRF. [Figure 2] 1 is a block diagram showing an example of a hardware configuration of an information processing device according to a first embodiment. [Figure 3] 1 is a block diagram showing an example of a logical configuration of an information processing device according to a first embodiment. [Figure 4] 3A to 3C are diagrams for explaining an example of a captured image acquired by an imaging data acquisition section according to the first embodiment. [Figure 5] FIG. 3 is a diagram showing an example of a background image acquired by a background data acquisition unit according to the first embodiment. [Figure 6] 4 is a flowchart showing an example of a processing flow in the information processing device according to the first embodiment. [Figure 7] FIG. 2 is a diagram showing an example of a GUI displayed and output by the information processing device according to the first embodiment. [Figure 8] 3A to 3C are diagrams showing an example of an object shape acquired by a shape acquisition unit according to the first embodiment. [Figure 9] 10A and 10B are diagrams illustrating an example of light rays in the generation process of an image outside the learning area according to the first embodiment. [Figure 10] 10 is a flowchart showing an example of the flow of processing for generating an image outside the learning area in the outside-area color data generating unit according to the first embodiment. [Figure 11] 4 is a diagram showing an example of an outside-of-learning-area image generated by an outside-area color data generating unit according to the first embodiment; FIG. [Figure 12] FIG. 3 is a diagram showing an example of a virtual viewpoint image generated by a learning unit according to the first embodiment. [Figure 13] 10 is a flowchart showing an example of the flow of processing for generating an image outside the learning area in an outside-area color data generating unit according to the second embodiment. [Figure 14] 10 shows an example of the positional relationship between an imaging device and an object according to the second embodiment. [Figure 15] 10A and 10B are diagrams illustrating an example of light rays in the generation process of an image outside the learning area according to the second embodiment. [Figure 16] 10A to 10C are diagrams showing examples of a captured image, a background image, an image outside the learning area, and a virtual viewpoint image according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. Note that the following embodiments do not limit the means for solving the problems of the present disclosure, and not all of the combinations of features described in the following embodiments are necessarily essential to the means for solving the problems of the present disclosure. Note that the same components will be described with the same reference numerals.

[0012] [First embodiment] Before describing the specific embodiments, the problem to be solved by the present disclosure will be described in detail. As described above, if the learning area is limited to a small space and, as a result, there is no color or volume density on a ray corresponding to at least some pixels in a virtual viewpoint image, the color of the pixel obtained by accumulating along the ray in volume rendering will be transparent. Hereinafter, the color obtained by accumulating along the ray will be referred to as "accumulated color." Furthermore, if there is some color or volume density on the ray, the accumulated color will be translucent. These transparent or translucent accumulated colors are generally overlaid with an opaque color (hereinafter referred to as "virtual background color") that is separately set as a background to generate a virtual viewpoint image.

[0013] FIG. 1 is a diagram illustrating an example of a virtual viewpoint image generated by a conventional NeRF. Specifically, FIG. 1(a) shows an example of the positional relationship between an object 101, an object 102, and a viewpoint 103. Hereinafter, a semi-transparent object 101 is placed in front of a background object 102, such as a wall, and a learning area 104 is set so that the object 101 in front of the viewpoint 103 is included but the object 102 behind the object 102 is not included. Hereinafter, the object 102 is described as including a portion 102a and a portion 102b that have different colors or patterns. FIGS. 1(b) and 1(c) show an example of an image in the case of the positional relationship shown in FIG. 1(a). Specifically, FIG. 1(b) shows an example of a captured image 110 obtained by capturing an image with an imaging device placed at the viewpoint 103, and FIG. 1(c) shows an example of a virtual viewpoint image 120 generated by a conventional NeRF when the viewpoint 103 is used as a virtual viewpoint.

[0014] The captured image 110 includes an image 111 corresponding to the object 101 and an image 112 corresponding to the object 102 that forms the background of the image 111; specifically, an image 112a corresponding to the portion 102a and an image 112b corresponding to the portion 102b. Since the object 101 is semi-transparent, the image 111 corresponding to the object 101 allows the image of the object 102 to pass through, and the image 111 corresponding to the object 101 is an image in which the color of the object 101 and the color of the object 102 are superimposed. Meanwhile, the virtual viewpoint image 120 includes an image 121 corresponding to the object 101 and, in the background of the image 111, an image 122 in a virtual background color that is prepared in advance. Since the object 101 is semi-transparent, the image 121 corresponding to the object 101 allows the image in the virtual background color to pass through, and the image 111 corresponding to the object 101 is an image in which the color of the object 101 and the virtual background color are superimposed.

[0015] Therefore, when the color of object 102, i.e., the color of the object outside the learning area, differs from the virtual background color, the difference between the virtual viewpoint image generated by the conventional NeRF and the captured image becomes large, making it difficult for the NeRF learning to converge to a correct state. Therefore, in this embodiment, an image in which only the image of the object outside the learning area is drawn is generated as color information related to the object outside the learning area, and the pixel value of this image is used as the virtual background color.

[0016] <Hardware configuration> FIG. 2 is a block diagram showing an example of the hardware configuration of an information processing device 200 according to the first embodiment. The information processing device 200 includes, as its hardware configuration, a CPU 201, a RAM 202, a ROM 203, a serial I / F (interface) 204, a VC (video card) 205, and a general-purpose I / F 206. The various components of the information processing device 200 are communicably connected to one another via a system bus 207. The CPU 201 uses the RAM 202 as a work memory and executes an OS (operating system) and various programs stored in the ROM 203, a storage device 211, or the like. The CPU 201 controls the entire information processing device 200 via the system bus 207 by executing the various programs. Note that the processing of each step shown in a flowchart described below is realized by the CPU 201 expanding program code stored in the ROM 203, the storage device 211, or the like into the RAM 202 and executing the expanded program code.

[0017] The serial I / F 204 is an interface configured by a serial ATA or the like, and is connected to a storage device 211 via a serial bus 208. The storage device 211 is a large-capacity storage device such as an HDD (hard disk drive) or an SSD (solid state drive). In the first embodiment, the storage device 211 is described as an external device of the information processing device 200, but the information processing device 200 may include the storage device 211 internally. The VC 205 receives a control signal from the CPU 201 and outputs a signal related to a display image to a display device 212 via a serial bus 209. The display device 212 is configured by a liquid crystal display or the like, and displays the display image based on the signal related to the display image output by the information processing device 200. The general-purpose I / F 206 is connected to an input device 213 such as a mouse or a keyboard via a serial bus 210, and receives an input signal from the input device 213.

[0018] The CPU 201 displays a GUI (Graphical User Interface) provided by a program on a display device 212 via a VC 205, and receives an input signal indicating an instruction from a user obtained via an input device 213. The information processing device 200 is realized, for example, by a desktop PC (Personal Computer). The information processing device 200 may also be realized by a notebook PC or a tablet PC integrated with the display device 212. The storage device 211 may also be realized by a medium (portable storage medium) and a drive such as a disk drive for accessing the medium, or a reader such as a memory card reader. The medium may be a flexible disk (FD), CD-ROM, DVD, USB memory, MO, flash memory, or the like.

[0019] <Logical configuration> 3 is a block diagram showing an example of the logical configuration of the information processing device 200 according to the first embodiment. The information processing device 200 has, as its logical configuration, an imaging data acquisition unit 301, a region setting unit 302, an outside-region color data acquisition unit 303, and a learning unit 304. The outside-region color data acquisition unit 303 also has a background data acquisition unit 331, a shape acquisition unit 332, and an outside-region color data generation unit 333. Each unit of the information processing device 200 as its logical configuration is realized by the CPU 201 executing a program stored in the ROM 203 using the RAM 202 as a work memory. Note that not all of the processes described below necessarily need to be executed by the CPU 201; the information processing device 200 may be configured so that some or all of the processes are executed by one or more processing circuits other than the CPU 201.

[0020] The imaging data acquisition unit 301 acquires data of multiple captured images (multi-viewpoint images) obtained by capturing images of an object in an imaging space from various viewpoints, and camera parameters corresponding to each captured image. The acquisition process of the imaging data acquisition unit 301 is executed based on instructions from a user input via the input device 213. Hereinafter, as an example, each captured image will be described as an RGB image. Hereinafter, as an example, the camera parameters will include internal parameters, external parameters, and distortion parameters of the imaging device that captures the captured image. The internal parameters are parameters that represent the position of the principal point of the imaging device and the focal length of the lens. The external parameters are parameters that represent the position of the imaging device and the direction of the optical axis of the imaging device (also referred to as the "attitude of the imaging device"). The distortion parameters are parameters that represent the distortion of the lens of the imaging device. The multi-viewpoint image data acquired by the imaging data acquisition unit 301 and the camera parameters corresponding to each captured image are transmitted to the shape acquisition unit 332, the outside-of-area color data generation unit 333, and the learning unit 304.

[0021] FIG. 4 shows a captured image I acquired by the image data acquisition unit 301 according to the first embodiment. CAM4A is a diagram for explaining an example of the positional relationship between the image capture device CAMk (k=1, 2, ..., K) and objects 401, 402, and 403. In the following description, the object 401 is assumed to be a sphere having a translucent color, the object 402 is assumed to be an ellipsoid having an opaque color, and the object 403 is assumed to be a planar wall having an opaque color. Note that the rectangular parallelepiped indicated by the dashed line in FIG. 4A represents the learning area 404.

[0022] Figure error! Bookmark not defined. (b) to (d) are images I obtained by imaging devices CAMk to CAMk+2. CAM An example of the captured image I is shown. CAMk The captured image I includes images 411 to 413 corresponding to the objects 401 to 403. Note that since the object 401 is semi-transparent, CAMk In the image I, the images 412 and 413 corresponding to the objects 402 and 403 are shown as being transparent through the image 411 corresponding to the object 401. CAMk+1 includes images 421 to 423 corresponding to the objects 401 to 403, and the captured image I CAMk+1 In the image I, an image 423 corresponding to the object 403 is shown as being visible through an image 421 corresponding to the object 401. CAMk+2 includes images 431 to 433 corresponding to the objects 401 to 403, and the captured image I CAMk+2 In the figure, an image 433 corresponding to object 403 is displayed in a transparent state through an image 431 corresponding to object 401.

[0023] The area setting unit 302 sets a learning area 404 in a three-dimensional space (virtual space) corresponding to the imaging space, based on instructions from the user input via the input device 213. Information indicating the learning area 404 set by the area setting unit 302 (hereinafter referred to as "learning area information") is transmitted to the outside-area color data acquisition unit 303 and the learning unit 304. The outside-area color data acquisition unit 303 acquires outside-area color data, which is information indicating colors outside the learning area 404.

[0024] Specifically, the background data acquisition unit 331 acquires background image data obtained by capturing an image of only a background object (hereinafter referred to as a "background object") based on an instruction from the user input via the input device 213. The background object is, for example, a distant view, a floor such as the ground, a wall surface of a structure such as a building, or a built-in facility. In the following description, the background image is assumed to be obtained by capturing an image using the same camera parameters as the imaging device that captured each captured image acquired by the imaging data acquisition unit 301. In the following description, the background image data is assumed to be an RGB image, like each captured image acquired by the imaging data acquisition unit 301, and to be prepared by capturing an image in advance in a state where no objects other than the background object are present within the angle of view.

[0025] FIG. 5 shows a background image I acquired by the background data acquisition unit 331 according to the first embodiment. BG Specifically, Fig. 5(a) to (c) are images of the captured image I shown in Fig. 5(b) to (d), respectively. CAMk ~I CAMk+2 Background image I corresponding to BGk ~I BGk+2 More specifically, the background image I shown in FIGS. BGk ~I BGk+2 is a state in which the objects 401 and 402 shown in FIG. 4(a) do not exist, the captured image I CAMk ~I CAMk+2 That is, the background image I shown in FIGS. BGk ~I BGk+2 In the image, the background object is the object 403. CAMk ~I CAMk+2 includes only images 413 to 433 corresponding to the object 403. The background image data acquired by the background data acquisition unit 331 is sent to the shape acquisition unit 332 and the outside-area color data generation unit 333.

[0026] The shape acquisition unit 332 acquires data on the three-dimensional shape of an object other than a background object (hereinafter referred to as "object shape") based on data on each captured image, camera parameters corresponding to each captured image, and data on a background image corresponding to each captured image. Details of the object shape acquisition process will be described later. The object shape data acquired by the shape acquisition unit 332 is sent to the outside-area color data generation unit 333.

[0027] The outside-area color data generation unit 333 generates an image in which only the image of the object outside the learning area is drawn, based on the data of each captured image, the camera parameters corresponding to each captured image, the background image data corresponding to each captured image, and the object shape data. The details of the process of generating this image (hereinafter referred to as the "outside-learning-area image") will be described later. The data of the outside-learning-area image generated by the outside-area color data generation unit 333 is transmitted to the learning unit 304.

[0028] The learning unit 304 sets a virtual background color based on the image outside the learning area, and estimates the color and volume density of the object in the learning area using the set virtual background color, the data of each captured image, and the camera parameters corresponding to each captured image. Specifically, for example, the learning unit 304 uses a function F Θ The color and volume density of a three-dimensional space that is a model of a three-dimensional space set as a learning area are estimated using the above method. F Θ :(x,y,z,θ,φ)→(R,G,B,σ)...Equation (1)

[0029] Here, (x, y, z) represent the position coordinates in three-dimensional space, (θ, φ) represent the direction in three-dimensional space, (R, G, B) represent the RGB values that indicate the color of the object determined by the position and direction, and σ represents the volume density of the object determined by the position. Θ is a function that outputs values indicating color and volume density for values indicating position and direction in three-dimensional space. ΘThe color information and volume density information in the three-dimensional space expressed by the function F Θ The following description will be given assuming that the radiance field is estimated by implementing the above using an MLP (Multi-layer Perceptron) and training the MLP using deep learning. In this case, the radiance field is expressed as MLP parameters, i.e., weight parameters related to the nodes that make up the MLP. Details of the radiance field estimation process will be described later. Data related to the radiance field estimated by the training unit 304 is output to and stored in the storage device 211 or the like.

[0030] <Operation of information processing device 200> 6 is a flowchart showing an example of a processing flow in the information processing device 200 according to the first embodiment. Note that "S" prefixed to a reference symbol indicates a step (process). First, in S601, the imaging data acquisition unit 301 acquires data of a plurality of captured images (multi-viewpoint images) and camera parameters corresponding to each captured image, for example, by reading them from the storage device 211, based on an instruction from the user. Next, in S602, the background data acquisition unit 331 acquires data of background images corresponding to each captured image acquired in S601, for example, by reading them from the storage device 211, based on an instruction from the user.

[0031] 7A and 7B are diagrams showing examples of GUIs 700 and 710 displayed by the information processing device 200 according to the first embodiment. Specifically, FIG. 7A shows an example of the GUI 700 for receiving instructions from the user in steps S601 and S602. In FIG. 7A, data path setting fields 701 to 703 are areas for inputting data paths for accessing captured image data, camera parameter data, and background image data, respectively. A button 704 is a button that the user presses to instruct continuation of processing. When the user presses the button 704, steps S601 and S602 are executed. The GUI 710 shown in FIG. 7B will be described later.

[0032] After S601 and S602, in S603, the shape acquisition unit 332 acquires object shape data based on the multi-view image data and camera parameters acquired in S601 and the background image data acquired in S602. The object shape data acquired in S603 only needs to represent the position and general shape of the object in three-dimensional space, and does not necessarily represent the fine irregularities in the object or the color of the object. Specifically, for example, the shape acquisition unit 332 first acquires a difference image between the captured image and the background image corresponding to the captured image. Next, the shape acquisition unit 332 acquires object shape data by estimating the three-dimensional shape of the object based on the difference images corresponding to each captured image and the camera parameters. Known three-dimensional shape estimation techniques, such as volume intersection or stereo matching, can be used to estimate the object shape. In this embodiment, the shape acquisition unit 332 acquires three-dimensional shape data represented by a set of voxels as object shape data using volume intersection.

[0033] FIG. 8 is a diagram showing an example of object shapes 801, 802 acquired by the shape acquisition unit 332 according to the first embodiment. Specifically, the object shapes 801, 802 shown in FIG. 8 are obtained based on the captured images shown as examples in (b) to (d) and the background images shown as examples in FIGS. 5(a) to (c). More specifically, the object shape 801 represents the position and general shape of the object 401 shown in FIG. 4(a), and the object shape 802 represents the position and general shape of the object 402 shown in FIG. 4(a). Also, the position P shown in FIG. CAMk This represents the position of the imaging device CAMk shown in Figure Error! Bookmark not defined. (a).

[0034] After S603, in S604, the region setting unit 302 sets a learning region based on instructions from the user. FIG. 7(b) shows an example of a GUI 710 for receiving instructions from the user in S604. Specifically, for example, the information processing device 200 presents learning region options via the GUI 710 shown as an example in FIG. 7(b). In FIG. 7(b), a learning region setting field 711 is an area in which candidate learning regions are presented as options, and is also an area in which the user selects an area to be used as a learning region from the presented options. An image display field 712 is an area in which the learning regions presented as options are visualized and displayed. In the example shown in FIG. 7(b), bounding boxes circumscribing each of the multiple object shapes acquired in S603 are presented as options, and the learning region setting field 711 shows a state in which "Region 1" is selected as the learning region. In the example shown in FIG. 7(b), the edges that make up the learning area (bounding box) presented as an option in the image display field 712 are projected and drawn on the background image together with the object shape.

[0035] Button 713 is a button that the user presses when instructing to continue processing. When button 713 is pressed by the user, the area selected in the learning area setting field 711 is set as the learning area. Note that the method of setting the learning area is not limited to the example described above. For example, the information processing device 200 may preliminarily associate information indicating a three-dimensional space corresponding to an imaging space in which an object may exist for each imaging environment, such as a studio, with information indicating the object, and present this information as options. Also, for example, the user may directly specify values related to the position and shape of the learning area. Note that in this embodiment, as an example, the shape of the learning area is described as a rectangular parallelepiped, but the shape of the learning area is not limited to a rectangular parallelepiped and may be any three-dimensional shape, such as a cylinder or a sphere.

[0036] After S604, in S605, the outside-area color data generation unit 333 generates an outside-learning-area image corresponding to each captured image. Specifically, the outside-learning-area image is generated based on the captured image data and camera parameters acquired in S601, the background image data acquired in S602, the object shape acquired in S603, and the learning area set in S604. Details of the outside-learning-area image generation process in S605 will be described later.

[0037] Next, in S606, the learning unit 304 estimates a radiance field for the learning region set in S604 based on the captured image data and camera parameters acquired in S601 and the image outside the learning region generated in S605. Specifically, for example, the learning unit 304 performs the following process in S606 as a process for estimating a radiance field for the learning region. First, the learning unit 304 determines a virtual background color based on the image outside the learning region. Next, the learning unit 304 generates a virtual viewpoint image by calculating a weighted sum (cumulative color) of the determined virtual background color and the color of the object in the learning region based on the radiance field. Next, the learning unit 304 performs MLP training using a signal consisting of RGB values of all pixels of the generated virtual viewpoint image as a predicted signal and a signal consisting of RGB values of all pixels of the captured image corresponding to the virtual viewpoint image as a teacher signal. This training is performed, for example, by an error backpropagation algorithm in which the squared Euclidean distance between the predicted signal and the teacher signal is used as a loss function. Details of the virtual viewpoint image generation process will be described later.

[0038] Next, in S607, the learning unit 304 outputs information about the radiance field estimated in S606 to the storage device 211, etc. After S607, the information processing device 200 ends the processing of the flowchart shown in FIG.

[0039] <Generation of images outside the learning area> In this embodiment, as an example, the RGB values of the image outside the learning area are determined based on the captured image and the background image in accordance with the intersection state between the object shape and the ray determined based on the pixel position (hereinafter referred to as "pixel position") and the camera parameters. Fig. 9 shows the RGB values of the image outside the learning area according to the first embodiment, which are determined based on the pixel position (hereinafter referred to as "pixel position") and the object shape. u,v FIG. 9(a) shows an example of a ray r u,v , the position P of the imaging device CAMk 9 shows a schematic example of the positional relationship between an image plane 901, a pixel 902 whose pixel position is (u, v), a learning area 903, and object shapes 801 and 802.

[0040] FIG. 9(b) is a schematic diagram of FIG. 9(a) viewed from the y-axis direction, and the light ray r u0,v0 〜r u3,v3 Each of these represents a ray whose corresponding pixel position is different from each other. u0,v0 intersects only with the object shape 801. Also, the ray r u1,v1 does not intersect with any object shape. u0,v0 and ray r u1,v1 For pixels corresponding to rays such as these, i.e., pixels corresponding to rays that do not intersect with the object shape outside the learning area 903, the RGB values of the background image, which contains only the image of the background object, are used as the RGB values of the image outside the learning area.

[0041] Also, ray r u2,v2 intersects only with the object shape 802. u2,v2 The pixel corresponding to ray r contains an image of an object 402 outside the training region, which is not a background object. u2,v2 For pixels corresponding to such rays, the RGB values of the corresponding pixels in the captured image are used as the RGB values of the image outside the learning area. u3,v3 intersects with both the object shape 801 and the object shape 802. Therefore, the ray r u3,v3The pixel corresponding to the ray r contains the image of the object 401 in the learning area in front of the image of the object 402 outside the learning area, overlapping each other. u3,v3 For pixels corresponding to such rays, the color of object 402 outside the learning area that is occluded by object 401 inside the learning area is estimated, and the RGB value indicating the estimated color is used as the RGB value of the image outside the learning area.

[0042] 10 is a flowchart showing an example of the flow of processing for generating an image outside the learning area in the outside-area color data generating unit 333 according to the first embodiment, and is a flowchart showing an example of the flow of processing in S605 shown in the flowchart of FIG. 6. This flowchart is executed after the processing of S604. First, in S901, the outside-area color data generating unit 333 generates a captured image I for which a corresponding outside-learning area image has not yet been generated. CAM Next, in S902, the outside-of-area color data generating unit 333 selects one of the captured images I selected in S901. CAM A pixel position (u, v) whose RGB value has not yet been determined is selected in the image outside the learning area corresponding to the pixel position (u, v).

[0043] Next, in S903, the outside-region color data generation unit 333 determines whether the image of the object 401 inside the learning region and the image of the object 402 outside the learning region overlap each other at the pixel position selected in S902. That is, in S903, the outside-region color data generation unit 333 determines whether the ray corresponding to the pixel position selected in S902 intersects with both the object shape 801 inside the learning region and the object shape 802 outside the learning region. In S1003, the outside-region color data generation unit 333 determines whether the ray intersects with the ray r shown in FIG. 9(b). u3,v3 If it is determined that the object shape 801 intersects with both the object shapes 801 and 802 inside and outside the learning area, the process of S904 is executed, and otherwise the process of S905 is executed.

[0044] In S904, the outside-of-region color data generation unit 333 determines the pixel value of the pixel position (u, v) selected in S902 in the outside-of-learning-region image corresponding to the captured image selected in S1001, based on the object shape and the captured image. Specifically, the outside-of-region color data generation unit 333 determines the pixel value of the pixel position (u, v) selected in S902 by estimating the color of the object 402 outside the learning region that is occluded by the object 401 in the learning region. For example, the outside-of-region color data generation unit 333 determines the pixel value of the pixel position (u, v) by executing the following process in S904.

[0045] First, the outside-of-area color data generator 333 calculates the color data for the ray r u,v Next, the outside-area color data generating unit 333 identifies the object shape (object shape 802 in the example of FIG. 9B) outside the learning area whose intersection point with the ray r is closest to the position of the image capturing device corresponding to the captured image selected in S901. u,v and obtains the three-dimensional coordinates of the intersection point p with the identified object shape. Next, the outside-of-area color data generation unit 333 determines whether or not an image of an object corresponding to the object shape is included in the pixel corresponding to the intersection point p for each captured image not selected in S901. Specifically, the outside-of-area color data generation unit 333 calculates the three-dimensional coordinates of the intersection point p of the ray r u,v The outside-region color data generating unit 333 determines whether the voxel that is the shortest distance from the position of the imaging device among the voxels that make up the object shape that intersects with the position is the voxel that makes up the object shape outside the learning region that is closest to the position. If the voxel is a voxel that makes up the object shape, the outside-region color data generating unit 333 determines that the image of the object corresponding to the object shape is included. In this case, the outside-region color data generating unit 333 calculates the average RGB value of the pixels that are determined to include the image of the object as the RGB value I of the outside-region image. OS Determined as (u,v).

[0046] In S905, the outside-area color data generation unit 333 determines whether or not the pixel position (u, v) selected in S902 includes an image of the object 402 outside the learning area. For example, if the ray corresponding to the pixel position is the ray r u2,v2 If the pixel position intersects with a voxel that constitutes the object shape outside the learning region as shown in S1005, it is determined that the pixel position contains an image of an object outside the learning region. If it is determined in S1005 that the pixel position contains an image of an object outside the learning region, the outside-region color data generation unit 333 executes the process of S906, and otherwise executes the process of S907.

[0047] In S906, the outside-of-area color data generating unit 333 calculates the RGB value I of the captured image selected in S901. CAM (u,v) is the RGB value of the image outside the learning area I OS In step S907, the outside-of-area color data generating unit 333 determines the RGB values I of the background image corresponding to the captured image selected in step S901. BK (u,v) is the RGB value of the image outside the learning area I OS Determined as (u,v).

[0048] After S1005, S1006, or S1007, in S908, the outside-of-area color data generating unit 333 generates the outside-of-area color data for the captured image I selected in S901. CAM In S908, it is determined whether all pixel positions have been selected in the image outside the learning area corresponding to the selected pixel position. If it is determined in S908 that at least some pixel positions of the image outside the learning area have not been selected, the outside-area color data generation unit 333 repeatedly executes the processes from S902 to S1008 until it is determined in S908 that all pixel positions have been selected. If it is determined in S908 that all pixel positions of the image outside the learning area have been selected, the outside-area color data generation unit 333 executes the process of S909.

[0049] In S909, the outside-of-area color data generation unit 333 determines whether or not all of the captured images have been selected in S901. If it is determined in S909 that at least some of the captured images have not been selected, the outside-of-area color data generation unit 333 repeatedly executes the processes from S901 to S1009 until it is determined in S909 that all of the captured images have been selected. If it is determined in S909 that all of the captured images have been selected, the outside-of-area color data generation unit 333 ends the process of the flowchart shown in Fig. 10, i.e., the process of generating images outside the learning area in S605 shown in Fig. 6.

[0050] FIG. 11 shows an outside-of-learning-area image I generated by the outside-of-area color data generating unit 333 according to the first embodiment. OS Specifically, FIG. 11 shows an example of the captured image I shown in (b) to (d) when the learning area is set to the learning area 404 shown in FIG. 4(a). CAM and background image I shown in Figs. 5(a) to (c). BG and the out-of-field image I generated based on OS More specifically, each of Figures 11(a) to 11(c) shows an example of the captured image I shown in Figures 11(b) to 11(d) in order. CAMk ~I CAMk+2 The corresponding out-of-domain image I OSk ~I OSk+2 An example of image I outside the learning area is shown. OSk ~I OSk+2 are images that include only the images 1102 to 1104 corresponding to the object 402 outside the learning area, and the image of the object 403.

[0051] 11(a), an area 1101 surrounded by a dashed line is an area in the captured image where the image of object 401 within the learning area overlaps with the image of object 402 outside the learning area, and the RGB values in area 1101 are the values determined in S904. Also, an area of images 1102 to 1104 surrounded by a double line is an area in the captured image where the image of object 402 does not overlap with the image of object 401, and the RGB values in the areas of images 1102 to 1104 are the same as those in the captured image. The remaining areas are areas in the captured image that include images other than the image of object 402, and the RGB values in these areas are the same as those in the background image.

[0052] <Virtual viewpoint image generation process> RGB value of virtual viewpoint image I VR (u, v) is calculated as a weighted sum of the RGB values in the learning area and the RGB values of the virtual background color, using the opacity in the learning area as a weighting coefficient, according to the following equations (2) to (6). I VR (u,v)=C A (r u,v )+(1-Opa(r u,v ))·C B ...Equation (2) TIFF2025117277000002.tif15150 TIFF2025117277000003.tif18150α i =1-exp(-σ i δ i )...Equation (5) TIFF2025117277000004.tif16150

[0053] where C A (r u,v ) is the RGB value of the object in the learning area indicated by the radiance field, and Opa(r u,v ) is the opacity, C B is the RGB value of the virtual background color. Furthermore, equations (3) to (5) correspond to known volume rendering, and i is the ray r u,vis the index of the sampling point on the i is the cumulative transmittance from the position of the image capture device to the sampling point. i is the opacity of the sampling point, and c i is the above function F Θ is the output RGB value, and σ i is a function F Θ is the volume density to be output. j is the distance from the jth sampling point to the j+1th sampling point.

[0054] Within the learning area, ray r u,v If there is an opaque object on top, the opacity of the object image will be 1, and the weight related to the virtual background color will be 0. Therefore, in this case, the color of the pixel in the virtual viewpoint image is not affected by the virtual background color. Also, within the learning area, the ray r u,v If no object exists on the virtual background, the opacity of the object image is 0, and the weight of the virtual background color is 1. Therefore, in this case, the color of the pixel in the virtual viewpoint image is equal to the virtual background color. Also, if the RGB value of the virtual background color is (0,0,0), then equation (2) becomes I VR (u,v)=C A (r u,v ) is equivalent to the opacity of the image of the object. u,v ) regardless of the RGB values of the objects in the learning area C A (r u,v ) is the RGB value of the virtual viewpoint image I VR The result is (u,v).

[0055] In this embodiment, the RGB value I OS (u,v) is used as the RGB value of the virtual background color. At this time, the RGB value I of the virtual viewpoint image VR (u, v) can be determined using, for example, the following equation (7). I VR(u,v)=C A (r u,v )+(1-Opa(r u,v ))·I OS (u,v)...Equation (7)

[0056] FIG. 12 shows a virtual viewpoint image I generated by the learning unit 304 according to the first embodiment. VR Specifically, FIG. 12 shows an example of the function F Θ 12(a) to 12(c) show an example of a virtual viewpoint image generated when the virtual background color is set to white. CAMk ~I CAMk+2 The virtual viewpoint image I VRk ~I VRk+2 These virtual viewpoint images I VRk ~I VRk+2 does not include the image of the object 402 outside the learning region. Therefore, even though the radiance field inside the learning region is accurately represented, the captured image I CAMk ~I CAMk+2 Therefore, as mentioned above, the difference between the captured image I CAMk ~I CAMk+2 and virtual viewpoint image I VRk ~I VRk+2 If the radiance field in the learning region is estimated using the difference as a loss, it becomes difficult for learning to converge to a correct state.

[0057] In contrast, (d) to (f) in FIG. 12 are images I outside the learning area shown in (a) to (c) in FIG. 11, respectively. OSk ~I OSk+2 Virtual viewpoint image I generated when the RGB values of VRk ~I VRk+2 12(d) to (f) show examples of virtual viewpoint images I VRk ~I VRk+2 In the virtual viewpoint image I, the color of the object 402 outside the learning area is reflected in the virtual background color.VRk ~I VRk+2 and captured image I CAMk ~I CAMk+2 This reduces the difference between the two, making it difficult for the learning to converge to the correct state.

[0058] According to the information processing device 200 configured as above, even when the learning area is limited to a small space, the radiance field within the learning area can be estimated with high accuracy.

[0059] In the present embodiment, the captured image and the background image have been described as RGB images, but the captured image and the background image may be images expressed in other formats such as grayscale images, XYZ images, YUV images, etc. In addition, in the present embodiment, the color of an object has been described as being determined by its position and direction, but the color of an object may be determined only by its position, regardless of its direction.

[0060] Furthermore, if it is clear in advance that the image of the object 401 inside the learning area and the image of the object 402 outside the learning area do not overlap in any of the captured images, the following processing may be performed in the processing for generating the image outside the learning area in S605. Specifically, in this case, after selecting the pixel position in S902, the processing of S903 and S904 may be skipped and the determination processing of S905 may be performed.

[0061] Furthermore, if there are no objects outside the learning area other than the background object (object 403 in the example of FIG. 4), the background image may be used as the image outside the learning area in the process of generating the image outside the learning area in S605. In this case, the shape acquisition unit 332 and the process of acquiring data on the three-dimensional shape of the object in S603 are not essential.

[0062] Furthermore, if the three-dimensional shapes of all objects included as images in the captured image are estimated in the process of acquiring three-dimensional shape data related to the objects in S603, the following process may be performed in the process of generating an image outside the learning area in S605. Specifically, in this case, the processes of S905 and S907 may be skipped. In that case, the background data acquisition unit 331 and the process of acquiring background image data in S602 are no longer essential.

[0063] The captured image may be a moving image made up of a plurality of frames, in which case the background image may be generated based on the plurality of frames using a known dynamic background estimation method.

[0064] [Second embodiment] In the first embodiment, a radiance field is estimated based on captured images, assuming that no object outside the learning area is in front of any object within the learning area, as viewed from the imaging device. However, depending on the positional relationship between the imaging device and the object, it may be difficult to capture an image in a state where no object outside the learning area is in front of any object within the learning area, as viewed from the imaging device. If, as viewed from the positions of at least some of the imaging devices, an object outside the learning area is in front of an object within the learning area, information about an object that is not a learning target (i.e., not in the learning area) is mixed into the teacher signal. Therefore, in such a case, it is difficult to correctly learn the radiance field within the learning area using the radiance field learning method in the information processing device 200 according to the first embodiment. In the second embodiment, a radiance field learning method that can accurately estimate the radiance field even in such a case will be described.

[0065] The hardware configuration and logical configuration of the information processing device 200 according to the second embodiment are the same as those of the information processing device 200 according to the first embodiment, and therefore, in the second embodiment, only the differences from the first embodiment will be described. Furthermore, the overall flow of processing executed in the information processing device 200 according to the second embodiment is the same as that of the information processing device 200 according to the first embodiment. However, the processing for generating an image outside the learning area in S605 and the processing for estimating a radiance field in S606 shown in FIG. 6 are different from those of the information processing device 200 according to the first embodiment. Therefore, in the second embodiment, mainly the processing differences from the first embodiment will be described. Note that, in the following, the same configurations and processing steps as those in the first embodiment will be described using the same reference numerals.

[0066] <Generation process of image outside learning area according to the second embodiment> 13 is a flowchart showing an example of the flow of processing for generating an image outside the learning area in the outside-of-area color data generation unit 333 according to the second embodiment, and is a flowchart showing an example of the flow of processing in S605 shown in the flowchart of FIG. 6. This flowchart is executed after the processing of S604. First, the outside-of-area color data generation unit 333 according to the second embodiment (hereinafter simply referred to as the "outside-of-area color data generation unit 333") executes the processing from S1001 to S1003. The processing from S1001 to S1003 has been described above, so a description thereof will be omitted.

[0067] At S1003, the light ray is ray r shown in FIG. 9(b). u3,v3 If it is determined that the object shape 801 intersects with both the object shapes 801, 802 inside and outside the learning area, as shown in S1304, the outside-area color data generation unit 333 executes the processing of S1304. In other cases, the outside-area color data generation unit 333 executes the processing of S1005 to S1007 as appropriate. The processing of S1005 to S1007 has been described above, so a description thereof will be omitted. In S1304, the outside-area color data generation unit 333 determines whether or not the pixel position (u, v) selected in S1002 includes an image of an object within the learning area.

[0068] Specifically, the outside-of-area color data generator 333 calculates the color data for the ray r u,v It is determined whether or not the voxel that is the shortest distance from the position of the imaging device among the voxels that make up the object shape that intersects with is present within the learning area. If the voxel that is the shortest distance from the position of the imaging device among the voxels that make up the object shape is present within the learning area, the outside-area color data generation unit 333 determines that the image of the object in the learning area is included at pixel position (u, v). If it is determined in S1304 that the image of the object in the learning area is included at pixel position (u, v), the outside-area color data generation unit 333 executes the processing of S1004, and otherwise executes the processing of S1306. The processing of S904 has been described above, so a description thereof will be omitted.

[0069] In S1306, the outside-area color data generation unit 333 sets a value indicating that the image of the object in the learning area is not included at the pixel position (u, v) selected in S1002 to the RGB value I of the image outside the learning area. OS The image data is determined as (u, v). Here, a value indicating that the image of an object within the learning area is not included is, for example, a negative value that cannot be taken as an RGB value, such as (-1, -1, -1). After S1004, S1306, S1006, or S1007, the outside-of-area color data generation unit 333 executes the processes of S1008 and S1009 as appropriate. The processes of S908 and S909 have been described above, so their description will be omitted. If it is determined in S909 that all captured images have been selected, the outside-of-area color data generation unit 333 ends the process of the flowchart shown in FIG. 13, i.e., the process of generating images outside the learning area at S605 shown in FIG. 6.

[0070] <Radiance field estimation process according to the second embodiment> As in the first embodiment, the learning unit 304 according to the second embodiment generates a virtual viewpoint image using a virtual background color determined based on an image outside the learning area, and performs learning by using the distance between a predicted signal consisting of RGB values of the virtual viewpoint image and a teacher signal consisting of RGB values of the captured image as a loss. Hereinafter, the learning unit 304 according to the second embodiment will be simply referred to as the "learning unit 304." Hereinafter, the learning unit 304 performs learning by calculating the loss using pixels included in areas other than areas where images of objects in the learning area are not captured due to objects in the learning area being occluded by objects outside the learning area. In other words, the areas other than areas where images of objects in the learning area are not captured refer to the following areas in the captured image: areas where images of objects in the learning area are visible, and areas where images of objects outside the learning area are visible and in directions in which the objects in the learning area are not occluded by the objects outside the learning area as viewed from the position of the imaging device. Specifically, the learning unit 304 references the image outside the learning area generated in S605 and excludes pixels whose RGB values indicate that they do not contain images of objects in the learning area from the predicted signal and the teacher signal.

[0071] FIG. 14 shows an example of the positional relationship between the image capture device CAMk (k=1, 2, ..., K) and objects 401, 402, 403, and 1401 according to the second embodiment. In the following description, the object 1401 is an ellipsoid with an opaque color, and is assumed to be located in front of the learning area 404 when viewed from the position of the image capture device CAMk. FIG. 15 shows the relationship between the ray r corresponding to the pixel position (u, v) in the process of generating an image outside the learning area according to the second embodiment. u,v ray r u0,v0 〜r u3,v3 represent rays whose corresponding pixel positions are different from each other. Also, object shape 1503 shown in FIG. 15 represents the object shape corresponding to object 1401. Note that data of object shape 1503 is acquired by shape acquisition unit 332, similar to object shapes 801 and 802. In FIG. 15, ray r u0,v0is closer to the position P of the image capture device than the object shape 801 in the learning area. CAMk At a position close to the intersection, the object shape 1503 outside the learning area intersects with the object shape 1503.

[0072] FIG. 16 shows a captured image I according to the second embodiment. CAMk , background image I BGk , image I outside the learning domain OSk , and virtual viewpoint image I VRk Specifically, FIG. 16(a) shows an example of a captured image I obtained by imaging using an imaging device CAMk. CAMk An example of the captured image I is shown. CAMk In FIG. 16(b), a part of the image 411 of the object 401 in the learning area is hidden by the image 1611 of the object 1401 outside the learning area and is therefore not visible. CAMk Background image I corresponding to BGk The background image I shown in FIG. BGk is the background image I shown in Figure 5(a). BGk is the same as

[0073] FIG. 16(c) shows the captured image I of FIG. 16(a). CAMk The corresponding out-of-domain image I OSk In FIG. 16(c), an area 1621 indicated by a texture with diagonal lines is an area where the object 1401 outside the learning area overlaps in front of the object 401 inside the learning area. Therefore, the RGB values of the pixels in the area 1621 are the same as those of the outside learning area image I. OSk , the value indicates that the image of the object 401 in the learning area is not included. FIG. 16(d) shows the captured image I in FIG. 16(a), which is obtained when the radiance field in the learning area is accurately expressed. CAMk The virtual viewpoint image I VRk In the area corresponding to the area 1621 shown in FIG. 16(c), the virtual viewpoint image I VRkand the captured image, the pixels in this region 1621 are excluded when calculating the loss, and therefore do not contribute to the estimation of the radiance field, i.e., the training of the MLP described above.

[0074] As described above, in the second embodiment, the information processing device 200 is configured to calculate the loss by excluding pixels that do not contain images of objects within the learning region when estimating the radiance field, i.e., when learning the above-described MLP. According to the information processing device 200 configured in this manner, it is possible to accurately estimate the radiance field within the learning region even when an object outside the learning region appears in front of an object within the learning region in the captured image.

[0075] [Other embodiments] The present disclosure can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0076] It should be noted that within the scope of the present disclosure, the embodiments may be freely combined, any component of each embodiment may be modified, or any component of each embodiment may be omitted.

[0077] [Configuration of the present disclosure] The present disclosure includes the following configurations, methods, and programs.

[0078] <Configuration 1> an imaging data acquisition means for acquiring data of a plurality of captured images obtained by capturing images of an object present in a predetermined imaging space from a plurality of viewpoints, and camera parameters corresponding to each of the plurality of viewpoints when capturing the images; an outside-area color data acquisition means for acquiring color information regarding an outside-area object, which is an object that exists outside a learning area set in a three-dimensional space corresponding to the predetermined imaging space; a learning means for learning spatial information, which is information about color and volume density at each position within the learning area, based on data of the plurality of captured images, camera parameters corresponding to each of the plurality of viewpoints, and color information about objects outside the area, the learning means learning the spatial information based on a difference between a rendering color, which includes a cumulative color obtained by accumulating the spatial information and a color about the objects outside the area, and the color of the captured image; An information processing device comprising:

[0079] <Configuration 2> further comprising area setting means for setting the learning area within the three-dimensional space; 2. The information processing device according to configuration 1,

[0080] <Configuration 3> the learning means learns the spatial information so that a difference between the rendering color and the color of the captured image becomes smaller; 3. The information processing device according to configuration 1 or 2, characterized in that:

[0081] <Configuration 4> the learning means calculates the cumulative color and the opacity of the cumulative color by accumulating the spatial information along a ray in the three-dimensional space determined based on the position of a pixel in the captured image and the camera parameters, and calculates the rendering color by superimposing the cumulative color and an out-of-area color determined based on color information regarding the out-of-area object in accordance with a blending ratio based on the opacity; 4. The information processing device according to any one of configurations 1 to 3, characterized in that:

[0082] <Configuration 5> the out-of-area color data acquisition means acquires image data showing only the out-of-area object as color information relating to the out-of-area object; 5. The information processing device according to any one of configurations 1 to 4, characterized in that:

[0083] <Configuration 6> a shape acquisition means for acquiring shape information indicating a shape of at least a part of the out-of-area object; and the outside-area color data acquisition means calculates color information regarding the outside-area object based on the captured image and the shape information; 6. The information processing device according to any one of configurations 1 to 5,

[0084] <Configuration 7> the spatial information is configured by network parameters in a multilayer perceptron; the learning means learns the spatial information by a backpropagation algorithm using a signal related to the rendering color as a predicted signal and a signal related to the color of the captured image as a teacher signal; 7. The information processing device according to any one of configurations 1 to 6,

[0085] <Configuration 8> the learning means learns the spatial information by excluding from the learning area an area in which an in-area object, which is the object within the learning area, is occluded by the out-area object; 7. The information processing device according to any one of configurations 1 to 6,

[0086] <Configuration 9> the spatial information is configured by network parameters in a multilayer perceptron; the learning means learns the spatial information by a backpropagation algorithm using a signal related to the rendering color as a predicted signal and a signal related to the color of the captured image as a teacher signal, and the learning means learns the spatial information by excluding the predicted signal and the teacher signal corresponding to an area where the intra-area object is occluded by the extra-area object; 9. The information processing device according to configuration 8,

[0087] <Configuration 10> the spatial information is information about a radiance field; 10. The information processing device according to any one of configurations 1 to 9, characterized in that:

[0088] <Method> an imaging data acquisition step of acquiring data of a plurality of captured images obtained by capturing images of an object present in a predetermined imaging space from a plurality of viewpoints, and camera parameters corresponding to each of the plurality of viewpoints when capturing the images; an outside-area color data acquisition step of acquiring color information regarding an outside-area object, which is an object existing outside a learning area set in a three-dimensional space corresponding to the predetermined imaging space; a learning process for learning spatial information, which is information about color and volume density at each position within the learning area, based on data of the plurality of captured images, camera parameters corresponding to each of the plurality of viewpoints, and color information about objects outside the area, the learning process for learning the spatial information based on a difference between a rendering color, which includes a cumulative color obtained by accumulating the spatial information and a color about the objects outside the area, and the color of the captured image; An information processing method comprising:

[0089] <Program> 11. A program for causing a computer to function as the information processing device according to any one of configurations 1 to 10. [Explanation of symbols]

[0090] 200 Information processing device 301 Imaging data acquisition unit 303 Out-of-area color data acquisition unit 304 Learning Department

Claims

1. an imaging data acquisition means for acquiring data of a plurality of captured images obtained by capturing images of an object present in a predetermined imaging space from a plurality of viewpoints, and camera parameters corresponding to each of the plurality of viewpoints when capturing the images; an outside-area color data acquisition means for acquiring color information regarding an outside-area object, which is an object that exists outside a learning area set in a three-dimensional space corresponding to the predetermined imaging space; a learning means for learning spatial information, which is information about color and volume density at each position within the learning area, based on data of the plurality of captured images, camera parameters corresponding to each of the plurality of viewpoints, and color information about objects outside the area, the learning means learning the spatial information based on a difference between a rendering color, which includes a cumulative color obtained by accumulating the spatial information and a color about the objects outside the area, and the color of the captured image; An information processing device comprising:

2. further comprising area setting means for setting the learning area within the three-dimensional space; 2. The information processing device according to claim 1,

3. the learning means learns the spatial information so that a difference between the rendering color and the color of the captured image becomes smaller; 2. The information processing device according to claim 1,

4. the learning means calculates the cumulative color and the opacity of the cumulative color by accumulating the spatial information along a ray in the three-dimensional space determined based on the position of a pixel in the captured image and the camera parameters, and calculates the rendering color by superimposing the cumulative color and an out-of-area color determined based on color information regarding the out-of-area object in accordance with a blending ratio based on the opacity; 2. The information processing device according to claim 1,

5. the out-of-area color data acquisition means acquires image data showing only the out-of-area object as color information relating to the out-of-area object; 2. The information processing device according to claim 1,

6. a shape acquisition means for acquiring shape information indicating a shape of at least a part of the out-of-area object; and the outside-area color data acquisition means calculates color information regarding the outside-area object based on the captured image and the shape information; 2. The information processing device according to claim 1,

7. the spatial information is configured by network parameters in a multilayer perceptron; the learning means learns the spatial information by a backpropagation algorithm using a signal related to the rendering color as a predicted signal and a signal related to the color of the captured image as a teacher signal; 2. The information processing device according to claim 1,

8. the learning means learns the spatial information by excluding from the learning area an area in which an in-area object, which is the object within the learning area, is occluded by the out-area object; 2. The information processing device according to claim 1,

9. the spatial information is configured by network parameters in a multilayer perceptron; the learning means learns the spatial information by a backpropagation algorithm using a signal related to the rendering color as a predicted signal and a signal related to the color of the captured image as a teacher signal, and the learning means learns the spatial information by excluding the predicted signal and the teacher signal corresponding to an area where the intra-area object is occluded by the extra-area object; 9. The information processing device according to claim 8,

10. the spatial information is information about a radiance field; 2. The information processing device according to claim 1,

11. an imaging data acquisition step of acquiring data of a plurality of captured images obtained by capturing images of an object present in a predetermined imaging space from a plurality of viewpoints, and camera parameters corresponding to each of the plurality of viewpoints when capturing the images; an outside-area color data acquisition step of acquiring color information regarding an outside-area object, which is an object existing outside a learning area set in a three-dimensional space corresponding to the predetermined imaging space; a learning process for learning spatial information, which is information about color and volume density at each position within the learning area, based on data of the plurality of captured images, camera parameters corresponding to each of the plurality of viewpoints, and color information about objects outside the area, the learning process for learning the spatial information based on a difference between a rendering color, which includes a cumulative color obtained by accumulating the spatial information and a color about the objects outside the area, and the color of the captured image; An information processing method comprising:

12. A program for causing a computer to function as the information processing device according to any one of claims 1 to 10.

Images