Image processing apparatus, image processing method, and storage medium

Abstract

To acquire a three dimensional field capable of suppressing deterioration of reproducibility of an image of an object included in a virtual viewpoint image even when a position or a direction of a viewpoint used in learning is greatly different from a position of a virtual viewpoint or a direction of a line of sight at the virtual viewpoint.SOLUTION: The image processing apparatus 102 acquires a plurality of captured images obtained by image capturing from a plurality of positions and a plurality of camera parameters of a plurality of viewpoints corresponding to the plurality of positions. A camera parameter of an auxiliary viewpoint different from the plurality of viewpoints is generated, shape data of an object estimated based on the plurality of acquired camera parameters and the plurality of captured images is acquired, an auxiliary viewpoint image is generated based on the shape data and the generated camera parameter, and information regarding a three dimensional field corresponding to at least a partial space in an image capturing space captured from a plurality of positions is generated based on the plurality of acquired camera parameters, the plurality of captured images, the generated camera parameter, and the auxiliary viewpoint image.SELECTED DRAWING: Figure 10A

Description

[Technical Field]

[0001] The present disclosure relates to image processing techniques for generating virtual viewpoint images. [Background technology]

[0002] There is a technology for generating an image (hereinafter referred to as a "virtual viewpoint image") corresponding to a view from an arbitrary viewpoint (hereinafter referred to as a "virtual viewpoint") using multiple captured images (hereinafter referred to as "multi-viewpoint images") obtained by capturing images from multiple different viewpoints. Patent Document 1 discloses a technology called NeRF (Neural Radiance Fields) as a method for generating a virtual viewpoint image. NeRF consists of a neural network that responds to density and color with respect to an arbitrary position and direction, and volume rendering that calculates a pixel value by accumulating colors obtained at multiple sampling points on a light ray corresponding to a pixel according to density. Training of the neural network in NeRF is achieved by using the pixel values ​​of each captured image that constitutes the multi-viewpoint image as training data and taking the squared error between the pixel value and the pixel value calculated by volume rendering as loss. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2023-066705 Summary of the Invention [Problem to be solved by the invention]

[0004] However, depending on the imaging conditions, there may be positions or directions that make it difficult for an imaging device to capture an object (hereinafter simply referred to as "object"), such as a field of view that looks up from below. In such cases, it is not possible to obtain an image corresponding to such a position or direction. Therefore, the position or direction of the viewpoint used when training the neural network in NeRF may differ significantly from the position or direction of the virtual viewpoint or the line of sight at the virtual viewpoint. In such cases, conventional NeRFs have the problem of significantly impairing the reproducibility of the image of the object included in the virtual viewpoint image. [Means for solving the problem]

[0005] The image processing device 102 according to the present disclosure includes an image acquisition means for acquiring data on a plurality of captured images obtained by imaging from a plurality of positions; a camera parameter acquisition means for acquiring a plurality of camera parameters for a plurality of viewpoints corresponding to the plurality of positions; a camera parameter generation means for generating camera parameters for an auxiliary viewpoint different from the plurality of viewpoints; a shape acquisition means for acquiring shape data indicating a three-dimensional shape of an object estimated based on the plurality of camera parameters acquired by the camera parameter acquisition means and data on the plurality of captured images acquired by the image acquisition means; an image generation means for generating an auxiliary viewpoint image corresponding to an appearance from the auxiliary viewpoint based on the shape data and the camera parameters generated by the camera parameter generation means; and an information generation means for generating three-dimensional field information regarding a three-dimensional field corresponding to at least a portion of a space in an imaging space imaged from the plurality of positions based on the plurality of camera parameters acquired by the camera parameter acquisition means, data on the plurality of captured images, camera parameters generated by the camera parameter generation means, and data on the auxiliary viewpoint image. [Effects of the Invention]

[0006] According to the present disclosure, even if the position or direction of the viewpoint used during learning is significantly different from the position of the virtual viewpoint or the direction of the line of sight at the virtual viewpoint, a three-dimensional field can be obtained that can suppress a decrease in the reproducibility of the image of an object contained in a virtual viewpoint image. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a diagram illustrating an example of the configuration of an image processing system according to a first embodiment. [Figure 2] 1 is a block diagram showing an example of a hardware configuration of an image processing apparatus and an information processing apparatus according to a first embodiment. [Figure 3] FIG. 1 is a diagram for explaining problems with conventional NeRF. [Figure 4] FIG. 10 is a diagram for explaining an example of a virtual viewpoint image generated by a conventional NeRF. [Figure 5] FIG. 1 is a diagram showing an example of a radiance field obtained as a result of training using conventional NeRF. [Figure 6] FIG. 1 is a diagram illustrating an example of a virtual viewpoint image obtained by a conventional NeRF. [Figure 7] 1A and 1B are diagrams showing an example of the position and direction of a virtual viewpoint that may reduce the reproducibility of a virtual viewpoint image obtained by a conventional NeRF. [Figure 8] FIG. 10 is a diagram illustrating an example of an auxiliary viewpoint according to the present disclosure. [Figure 9] 1A and 1B are diagrams illustrating an example of a virtual viewpoint image and an auxiliary viewpoint image according to the present disclosure. [Figure 10A] 1 is a block diagram showing an example of a functional configuration of an image processing apparatus according to a first embodiment. [Figure 10B] 1 is a block diagram showing an example of a functional configuration of an information processing device according to a first embodiment. [Figure 11A] 4 is a flowchart showing an example of a processing flow in the image processing device according to the first embodiment. [Figure 11B] 4 is a flowchart showing an example of a processing flow in the information processing device according to the first embodiment. [Figure 12] FIG. 3 is a diagram for explaining an example of a method for generating auxiliary camera parameters according to the first embodiment. [Figure 13] FIG. 2 is a diagram illustrating an example of a GUI according to the first embodiment. [Figure 14] FIG. 10 is a block diagram showing an example of the functional configuration of an image processing apparatus according to a second embodiment. [Figure 15] 10 is a flowchart showing an example of a processing flow of an image processing apparatus according to a second embodiment. [Figure 16] 10 is a flowchart showing an example of the flow of a generation process of auxiliary camera parameters according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that the following embodiments do not limit the solution of the present disclosure, and not all of the combinations of features described in the present embodiments are necessarily essential to the solution of the present disclosure. Note that the same components will be described with the same reference numerals. Furthermore, each process (step) in the flowchart described below will be indicated with a reference numeral beginning with "S."

[0009] In the following description, a two-dimensional region in an image will be simply referred to as a "region," and a three-dimensional region in an imaging space or a virtual space will be referred to as a "space." In the following embodiments, a method for generating a trained model will be described, assuming that the trained model is generated by training a learning model that models a three-dimensional field (hereinafter referred to as a "three-dimensional field") in the imaging space to be learned. In the following embodiments, the trained model that models the three-dimensional field (hereinafter referred to as a "three-dimensional field model") will be described as a radiance field based on NeRF, which is configured using a multilayer perceptron; however, the three-dimensional field model is not limited to this.

[0010] The method of representing a three-dimensional field varies depending on the learning content. Specifically, for example, the three-dimensional field model may be constructed using InstantNGP, a high-speed method similar to NeRF. Furthermore, the three-dimensional field model is not limited to one constructed using a multilayer perceptron, but may be constructed using Plenoxels or TensoRF (Tensorial Radiance Fields), which explicitly represent three-dimensional fields. Furthermore, the three-dimensional field model may be constructed using NeuS, which improves the accuracy of shape estimation by representing three-dimensional fields using SDF (Signed Distance Field). Furthermore, the three-dimensional field model may be constructed using various methods, such as 3D Gaussian Splatting, which represents three-dimensional fields using a set of points with a spread.

[0011] [First embodiment] <Image processing system configuration> 1 is a diagram showing an example of the configuration of an image processing system according to the first embodiment. The image processing system includes a plurality of image capturing devices 101, an image processing device 102, a user interface (hereinafter referred to as "UI") panel 103, a storage device 104, a display device 105, an information processing device 108, a display device 109, and an input device 110.

[0012] The multiple imaging devices 101 are composed of digital still cameras, digital video cameras, etc., and each imaging device 101 is located at a different position. Each imaging device 101 captures an object 107 present in an imaging space 106 from different viewpoints in a synchronized manner according to predetermined imaging conditions, thereby acquiring data of multiple captured images (multi-viewpoint images) corresponding to each viewpoint. Note that "synchronized imaging" does not mean simultaneous imaging, but rather imaging after synchronization processing. In other words, synchronized imaging does not need to be performed at exactly the same time, and may also be performed at approximately the same time. Captured image data obtained by imaging devices 101 (hereinafter referred to as "captured image data") may be still image data, moving image data, or both still image and moving image data. Hereinafter, the term "image" will be explained as including both "still image" and "moving image" unless otherwise specified. The captured image data acquired by each imaging device 101 is transmitted to the image processing device 102.

[0013] The image processing device 102 acquires data of multiple captured images (multi-viewpoint images) transmitted from multiple imaging devices 101, and uses the acquired multi-viewpoint images to learn a three-dimensional field of a space including an object 107 present in an imaging space 106. Information indicating the trained three-dimensional field obtained as a result of learning by the image processing device 102 is output to an information processing device 108 via a network 111 such as the Internet. Information or a signal indicating the trained three-dimensional field may be output to a storage device 104, a display device 105, or the like. The image processing device 102 may also generate a virtual viewpoint image based on a three-dimensional field in the middle of learning or a trained three-dimensional field obtained as a result of learning. In this case, data or a signal of the virtual viewpoint image generated by the image processing device 102 is output to, for example, the storage device 104, the display device 105, or the like.

[0014] 1, the description will be given assuming that each of the multiple imaging devices 101 and the image processing device 102 are connected to one another, but the method of connection between the imaging devices 101 and the image processing device 102 is not limited to this. Specifically, for example, the multiple imaging devices 101 may be cascade-connected by connecting adjacent imaging devices 101 to one another, and at least one of the multiple imaging devices 101 may be connected to the image processing device 102.

[0015] 1 as an example, the present embodiment will be described assuming that a plurality of imaging devices 101 are arranged at different positions, but the number and arrangement of the imaging devices 101 are not limited to this. For example, if the position, shape, and color of an object 107 present in the imaging space 106, as well as the intensity or hue of ambient light, do not change over time, at least one imaging device 101 whose position and orientation can be changed may be arranged. In this case, the imaging device 101 may be caused to capture images at a plurality of different positions while changing the position and orientation of the imaging device 101, and the image processing device 102 may acquire a plurality of pieces of captured image data obtained by the imaging as data of a multi-viewpoint image.

[0016] The UI panel 103 includes a display device such as a liquid crystal panel, and displays a GUI (Graphical User Interface) on the display device to present information such as the image capture conditions of the image capture device 101 and the processing settings of the image processing device 102 to the user. The UI panel 103 may also include an input device such as a touch panel or buttons, in which case the UI panel 103 accepts instructions from the user regarding changes to the above-mentioned image capture conditions or processing settings. In this case, information indicating the user's instructions accepted by the UI panel 103 is transmitted to the image processing device 102. The input device may be provided separately from the UI panel 103, such as a mouse or a keyboard.

[0017] The storage device 104 is configured with a hard disk drive or the like, and acquires data of the virtual viewpoint image output from the image processing device 102 and stores the acquired data. The storage device 104 also acquires information indicating the three-dimensional field output from the image processing device 102 and stores the acquired information.

[0018] The display device 105 is configured with a liquid crystal display or the like, acquires a display image signal including a virtual viewpoint image output from the image processing device 102, and displays a virtual viewpoint image corresponding to the signal. The display device 105 also acquires a display image signal including an image showing a three-dimensional field output from the image processing device 102, and displays an image showing a three-dimensional field corresponding to the signal.

[0019] The imaging space 106 is a three-dimensional space surrounded by a plurality of imaging devices 101 installed in a studio or the like, and the solid-line frame in FIG. 1 indicates the contour of the imaging space 106 on the floor. Hereinafter, as an example, a mode will be described in which eight imaging devices 101 installed in a studio are used to capture images of one or more objects from around the objects. While the camera parameters of each imaging device 101 will be described as being pre-stored in the storage device 204, they may also be estimated by the image processing device 102 using captured image data. In this case, for example, the image processing device 102 estimates the camera parameters of each imaging device 101 using an algorithm known as COLMAP, which is well known in the technical field such as NeRF, and which estimates the shape of an object while estimating the imaging position based on a captured image.

[0020] The camera parameters include internal parameters, external parameters, distortion parameters, etc. Here, the internal parameters are parameters that represent the coordinates of the center of a captured image obtained by capturing an image using an imaging device and the focal length of the lens. The external parameters are parameters that represent the position and orientation of the imaging device, and the distortion parameters are parameters that indicate lens distortion. Note that the camera parameters of the multiple imaging devices 101, particularly the internal parameters and distortion parameters, may be common to each other. Note that distortion parameters other than the internal parameters and external parameters, etc., are data that are included in the camera parameters as needed, but do not necessarily have to be included in the camera parameters.

[0021] The information processing device 108 generates a virtual viewpoint image based on the information indicating the learned three-dimensional field output from the image processing device 102. Data or a signal of the virtual viewpoint image generated by the information processing device 108 is output to, for example, a display device 109. The display device 109 has the same configuration as the display device 105, and therefore a description thereof will be omitted. The input device 110 is configured with a mouse, a keyboard, or the like, and receives an input operation from a user of the information processing device 108 and transmits an input signal corresponding to the input operation to the information processing device 108.

[0022] <Hardware configuration of image processing device and information processing device> Fig. 2 is a block diagram showing an example of the hardware configuration of the image processing device 102 and the information processing device 108 according to the first embodiment. Specifically, Fig. 2(a) shows an example of the hardware configuration of the image processing device 102. The image processing device 102 has, as its hardware configuration, a CPU 201, a RAM 202, a ROM 203, a storage device 204, a control interface (hereinafter referred to as "I / F") 205, an input I / F 206, an output I / F 207, and a main bus 208.

[0023] The CPU 201 is a processor that comprehensively controls each unit of the image processing device 102. The CPU 201 uses the RAM 202 as a work memory and executes an operating system (OS) and various programs stored in the ROM 203, the storage device 204, etc. The CPU 201 executes the various programs to control the entire image processing device 102 via the main bus 208. Note that the processing of each step shown in a flowchart related to the image processing device 102, which will be described later, is realized by the CPU 201 executing program code stored in the ROM 203, the storage device 204, etc., which is loaded into the RAM 202. The RAM 202 functions as the main memory and work area of ​​the CPU 201. The ROM 203 stores a group of programs executed by the CPU 201. The storage device 204 is configured by a hard disk drive, etc., and stores application programs executed by the CPU 201 and various data used in the processing of the CPU 201.

[0024] The control I / F 205 is connected to each of the multiple image capture devices 101 and is a communication interface for controlling the setting of image capture conditions for each image capture device 101, the start and stop of image capture, etc. The input I / F 206 is a communication interface using a serial bus such as SDI (Serial Digital Interface) or HDMI (High-Definition Multimedia Interface (registered trademark)). Captured image data output from each image capture device 101 is acquired via the input I / F 206. The output I / F 207 is a communication interface using a serial bus such as USB (Universal Serial Bus) or IEEE1394. Data or signals such as virtual viewpoint images and three-dimensional fields are output to the storage device 104 or the display device 105 via the output I / F 207. The main bus 208 is a transmission path that communicably connects the above-mentioned hardware components of the image processing device 102 to each other.

[0025] FIG. 2(b) shows an example of the hardware configuration of the information processing apparatus 108. The information processing apparatus 108 includes, as its hardware configuration, a CPU 251, a RAM 252, a ROM 253, a storage device 254, an output I / F 257, and a main bus 258. The CPU 251 is a processor that comprehensively controls each part of the information processing apparatus 108. The CPU 251 uses the RAM 252 as a work memory and executes an OS (operating system) and various programs stored in the ROM 253 or the storage device 254 or the like. By executing various programs, the CPU 251 controls the entire information processing apparatus 108 via the main bus 258. Note that the processing of each step shown in the flowchart regarding the information processing apparatus 108 described later is realized by the program code stored in the ROM 253 or the storage device 254 or the like being expanded into the RAM 252 and the CPU 251 executing this. The RAM 252 functions as the main memory and work area of the CPU 251.

[0026] The ROM 253 stores a group of programs to be executed by the CPU 251. The storage device 254 is constituted by a hard disk drive or the like and stores application programs to be executed by the CPU 251 and various data used for the processing of the CPU 251. The output I / F 257 is a communication interface using a serial bus such as USB or IEEE 1394. A signal indicating a virtual viewpoint image is output to the display device 109 via the output I / F 257. The main bus 258 is a transmission path that communicably connects the above-described hardware configurations of the information processing apparatus 108 to each other.

[0027] <Regarding the learning of NeRF> In this embodiment, as an example, a three-dimensional field is represented by a radiance field using NeRF, and therefore, an overview of NeRF learning will be first described. NeRF is configured by a neural network that outputs volume density σ and color (r, g, b) in response to five-dimensional input variables of three-dimensional coordinates (x, y, z) and direction (θ, φ) indicating an arbitrary spatial position. Here, each element of color (r, g, b) is a value corresponding to red (R), green (G), and blue (B), respectively. To obtain pixel values ​​(r, g, b), first, multiple (N (N is a positive integer of 2 or more)) sampling points P are located on the light ray corresponding to the pixel. i (i is a positive integer less than or equal to N) is prepared. Next, the sampling point P i By inputting the position (x, y, z) and the direction of the ray (θ, φ) into the neural network, each sampling point P i The volume density σ i and color c i The neural network outputs the volume density σ i The color weight sum c based on i By calculating the pixel value C v is determined.

[0028] In volume rendering, first, each sampling point P is calculated based on the volume density and the distance between the sampling points. i Cumulative transmittance T i is obtained. The cumulative transmittance T i is the sampling point P i Color in c i represents the ratio of light reaching the imaging position. Specifically, the cumulative transmittance T i is calculated, for example, by using the formula (1).

[0029]

number

[0030] where δ jis the current sampling point P j to the next sampling point P j+1 As shown in equation (1), the cumulative transmittance T i is the volume density σ during the calculation process j The larger the value of , the smaller the value. In volume rendering, the cumulative transmittance T i , volume density σ i , and the distance δ j Based on this, each sampling point P i Color in c i The weight of i Furthermore, the color c i and weight w i and based on the pixel value C v Specifically, the weight w i is calculated using, for example, equation (2), and the pixel value C v For example, color c can be calculated using equation (3). i is calculated by weighted addition.

[0031]

number

[0032]

number

[0033] In NeRF, the neural network is trained by first using the pixel value C obtained by volume rendering. v and the squared error between the value of the corresponding pixel (pixel value C) in the captured image data, which is the training data, is obtained as the loss L. Next, the weight parameters of the neural network are changed by any method, such as the backpropagation method, using the obtained loss L. The loss L is calculated using, for example, Equation (4).

[0034]

number

[0035] In addition, even when a virtual viewpoint image corresponding to a desired virtual viewpoint is generated using a trained neural network, processing similar to the volume rendering executed during training is executed.

[0036] <Problems with conventional NeRF> Before describing the specific embodiments of the present disclosure, problems with conventional NeRF will be described. FIG. 3 is a diagram illustrating the problems with conventional NeRF. FIG. 3(a) shows an example of an object 301 being imaged by an imaging device 101 (imaging devices 302 to 307 in FIG. 3) installed on a wall, ceiling, or support. Specifically, FIG. 3(a) shows an image capture space 106 viewed from the horizontal direction. When capturing an image of a moving object such as a person, it is not possible to install the imaging device in a space that allows the object to move. Therefore, as shown as an example in FIG. 3(a), it can be difficult to install the imaging device in a position that allows the object to be imaged while looking up.

[0037] FIG. 3(b) shows an example of a radiance field (three-dimensional field) 308 of an object 301 obtained by NeRF learning. Specifically, FIG. 3(a) shows a cross section parallel to the vertical direction when the radiance field of the object 301 is viewed from the horizontal direction. Note that the image capture devices 302 to 307 shown in FIG. 3 indicate the positions and orientations of the image capture devices estimated by calibration. In FIG. 3(b), the color of positions where the density is equal to or greater than a certain value is shown using the original color of the object 301. When the radiance field is learned using NeRF, the density at positions corresponding to the surface of the object 301 becomes high in the density field indicated by the radiance field.

[0038] However, under imaging conditions such as those described above where it is difficult to install the imaging device in a certain direction, the density may be low at some positions 309 corresponding to the surface of the object 301. This can occur when the colors at position 309, where a ray corresponding to a pixel in the captured image corresponding to the low density intersects with a curved surface corresponding to the surface of the object 301 in the radiance field, are similar to those at positions 310 and 311. In this case, even if the density at position 309 close to the positions corresponding to the imaging devices 302 and 303 is low, the pixel values ​​obtained by volume rendering are close to the pixel values ​​of the captured image serving as training data. As a result, the loss L calculated based on the pixel values ​​obtained by volume rendering and the pixel values ​​of the captured image serving as training data is reduced. This reduction in loss L causes the learning of the three-dimensional field model to converge without the loss at position 309 increasing.

[0039] FIG. 4 is a diagram illustrating an example of a virtual viewpoint image generated by a conventional NeRF. FIG. 4(a) shows an example of an image region corresponding to position 309 and its surroundings in a captured image obtained by capturing an image using an image capturing device 303. FIG. 4(b) shows an example of the image region in a virtual viewpoint image generated based on the position and gaze direction of a virtual viewpoint corresponding to the position and orientation of the image capturing device 303 and the radiance field 308. In FIGS. 4(a) and 4(b), region 409 is the region of pixels corresponding to position 309. The color of region 409, which is the region of pixels corresponding to position 309 in the captured image shown in FIG. 4(a), is the same color as the color of the position on the surface of object 301 (107) corresponding to position 309 shown in FIG. 3(b). The color of region 409, which is the region of pixels corresponding to position 309 in the virtual viewpoint image shown in FIG. 4(b), is the same color as the color of position 311 shown in FIG. 3(b). The color of position 311 shown in FIG. 3(b) is close to the color of the position on the surface of object 301 (107) corresponding to position 309 shown in FIG. 3(b), so the learning of the three-dimensional field model converges while the density at position 309 remains low.

[0040] The deterioration of the reproducibility of the virtual viewpoint image will be described with reference to Fig. 5 and Fig. 6. Fig. 5 is a diagram showing an example of a radiance field 308 obtained as a result of learning using conventional NeRF. As shown in Fig. 5, consider a ray that penetrates the radiance field 308 and passes through a position 309 where density is low at a position corresponding to the surface of an object in the radiance field 308, and a position 502 corresponding to the surface of the object. It is assumed that the surface color of the object corresponding to the position 309 in the radiance field 308 and the surface color of the object corresponding to the position 502 are different colors from each other, as shown in Fig. 5.

[0041] FIG. 6 is a diagram illustrating an example of a virtual viewpoint image obtained by a conventional NeRF. Specifically, FIG. 6(a) shows an example of an image in which the view from the virtual viewpoint is faithfully reproduced, and FIG. 6(b) shows an example of a virtual viewpoint image obtained using a radiance field 308 obtained as a result of learning by the conventional NeRF. Ideally, the color of region 409 in the virtual viewpoint image shown in FIG. 6(b) should be the same color as region 409 shown in FIG. 6(a), which is the surface color of the object corresponding to position 309. However, in the case of the radiance field 308 shown in FIG. 5, the color of region 409 in the virtual viewpoint image shown in FIG. 6(b) becomes the surface color of the object corresponding to position 402. In this way, the difference between the actually generated virtual viewpoint image and the expected virtual viewpoint image is an example of a decrease in the reproducibility of the virtual viewpoint image.

[0042] The reproducibility of a virtual viewpoint image may be reduced in cases other than those described with reference to FIG. 5. FIG. 7 illustrates an example of the position and direction of a virtual viewpoint that may reduce the reproducibility of a virtual viewpoint image obtained by a conventional NeRF. Even if the density of all positions corresponding to the surface of an object in the density field indicated by the radiance field 308 is high, a virtual viewpoint image may be generated in which the color of the object's image differs from its actual appearance. For example, as shown in FIG. 7, this occurs when the position and direction of the virtual viewpoint 703 are significantly different from the position or imaging direction (orientation) of the image capture device 101 (302 to 307 in FIG. 7). In the case of FIG. 7, there is no image capture device 101 that captures an object from below at an angle looking up at it, as in the case of the virtual viewpoint 703. Therefore, it is not possible to use data of captured images captured at such an angle as training data to train the radiance field 308. Furthermore, the NeRF model is designed to allow color changes with direction. Therefore, if there is no training data of captured image data as described above, the solution may become unstable because no loss occurs during training.

[0043] <Basic Concept of This Disclosure> In order to solve the problems of conventional NeRF described above, in the present disclosure, an auxiliary image corresponding to the view from an auxiliary viewpoint is generated using a method different from a three-dimensional field model such as NeRF, and data of the generated image is used as training data in learning the three-dimensional field model. This improves the reproducibility of a virtual viewpoint image corresponding to the view from a virtual viewpoint in a direction not captured by the imaging device 101. Hereinafter, the auxiliary viewpoint will be referred to as an "auxiliary viewpoint," and the auxiliary image will be referred to as an "auxiliary viewpoint image."

[0044] FIG. 8 is a diagram illustrating an example of auxiliary viewpoints 801 to 803 according to the present disclosure. FIG. 8 also illustrates a radiance field 808 obtained when additional learning is performed using data from the auxiliary viewpoint image described above as training data. FIG. 9 is a diagram illustrating an example of a virtual viewpoint image and an auxiliary viewpoint image corresponding to the view from the auxiliary viewpoint 801 according to the present disclosure. Specifically, FIG. 9(a) is an example of a virtual viewpoint image corresponding to the view from the auxiliary viewpoint 801, and FIG. 9(b) is an example of an auxiliary viewpoint image corresponding to the view from the auxiliary viewpoint 801. Additional learning using data from auxiliary viewpoint images corresponding to the views from the auxiliary viewpoints 801 to 803 as training data can reduce the occurrence of a phenomenon in which learning of a three-dimensional field model converges while the density at position 309 remains low, as shown in FIG. 3(b). Specifically, additional learning using data from the auxiliary viewpoint image shown as an example in FIG. 9(b) can increase the density at position 309 in the density field indicated by the radiance field 808. If the density at position 309 is low in radiance field 808 during training, a ray passing between the position of auxiliary viewpoint 801 and position 309 will reach position 705. That is, the color of region 409 corresponding to position 309 in the virtual viewpoint image shown as an example in FIG. 9( a) , which is generated using radiance field 808 during training, becomes the color of the position on the surface of the object corresponding to position 705, as shown in FIG. 6( b) . Therefore, a difference, that is, a loss, occurs between the pixel values ​​in region 409 corresponding to position 309 in the virtual viewpoint image and the pixel values ​​in region 409 corresponding to position 309 in the auxiliary viewpoint image shown as an example in FIG. 9 . Training of radiance field 808 is performed to reduce this loss, and such training can increase the density at position 309 in the density field represented by radiance field 808.

[0045] According to the results of such learning and the obtained learned radiance field 808, no matter what direction the virtual viewpoint is set in, the color of the pixel corresponding to position 309 in the virtual viewpoint image will be the same color as the color of the position on the surface of the object corresponding to position 309. Furthermore, by using, as training data, image data (auxiliary viewpoint image) corresponding to the view from a direction in which the imaging device 101 is not placed as viewed from the imaging space, it is possible to prevent the learning of the radiance field 808 from converging on a color solution that is far removed from reality.

[0046] <Functional configuration of image processing device and information processing device> The functional configurations of the image processing device 102 and the information processing device 108 will be described with reference to FIGS. 10A and 10B. FIG. 10A is a block diagram showing an example of the functional configuration of the image processing device 102 according to the first embodiment. The image processing device 102 has, as its functional configuration, a camera parameter acquisition unit 1001, a camera parameter generation unit 1002, an image acquisition unit 1003, a shape acquisition unit 1004, a second image generation unit 1005, a first image generation unit 1006, a learning unit 1007, an information acquisition unit 1008, and an output unit 1009. Each unit included in the functional configuration of the image processing device 102 is realized by the CPU 201 executing a program stored in the ROM 203 or the like, using the RAM 202 as a work memory. Note that not all of the processes described below necessarily need to be realized by the CPU 201 executing a program, and the image processing device 102 may be configured so that part or all of the processes are executed by one or more processing circuits other than the CPU 201.

[0047] The camera parameter acquisition unit 1001 acquires camera parameters (hereinafter referred to as "imaging camera parameters") of each imaging device 101. The imaging camera parameters acquired by the camera parameter acquisition unit 1001 are transmitted to the camera parameter generation unit 1002, the shape acquisition unit 1004, the first image generation unit 1006, and the learning unit 1007.

[0048] The camera parameter generation unit 1002 generates camera parameters (hereinafter referred to as "auxiliary camera parameters") different from each of the imaging camera parameters based on information about the learning space and the imaging camera parameters transmitted from the camera parameter acquisition unit 1001. Hereinafter, the information about the learning space will be referred to as "learning space information." In this embodiment, the camera parameter generation unit 1002 will be described as having previously stored therein the learning space information. However, the camera parameter generation unit 1002 may acquire the learning space information by reading it from the storage device 204. The auxiliary camera parameters are camera parameters corresponding to an imaging device (also called a "virtual camera") that performs virtual imaging from an auxiliary viewpoint (auxiliary viewpoint) different from the viewpoint corresponding to each imaging device 101. Here, the imaging camera parameters and the auxiliary camera parameters have the same data configuration (data format). That is, the auxiliary camera parameters have extrinsic parameters, internal parameters, distortion parameters, etc., similar to the imaging camera parameters. Details of the auxiliary camera parameter generation process in the camera parameter generation unit 1002 will be described later. The auxiliary camera parameters generated by the camera parameter generation unit 1002 are transmitted to the first image generation unit 1006 , the second image generation unit 1005 , and the learning unit 1007 .

[0049] The image acquisition unit 1003 acquires captured image data obtained by capturing images with each of the multiple imaging devices 101. The source of the captured image data is not limited to the imaging device 101, and the image acquisition unit 1003 may acquire the captured image data by reading it from the storage device 104 or the like. The captured image data acquired by the image acquisition unit 1003 is transmitted to the shape acquisition unit 1004 and the second image generation unit 1005. In addition, the captured image data acquired by the image acquisition unit 1003 is transmitted to the learning unit 1007 as training data for learning a three-dimensional field model in the middle of learning.

[0050] The shape acquisition unit 1004 acquires shape data indicating the three-dimensional shape of an object existing in the learning space. For example, the shape acquisition unit 1004 acquires shape data indicating the three-dimensional shape of the object existing in the learning space by inferring the three-dimensional shape of the object using imaging camera parameters and captured image data. Details of the three-dimensional shape inference process in the shape acquisition unit 1004 will be described later. The shape data acquired by the shape acquisition unit 1004 is transmitted to the second image generation unit 1005.

[0051] The second image generation unit 1005 generates an image (auxiliary viewpoint image) corresponding to the view from the auxiliary viewpoint using the shape data, captured image data, and auxiliary camera parameters. Details of the auxiliary viewpoint image generation process in the second image generation unit 1005 will be described later. The auxiliary viewpoint image data generated by the second image generation unit 1005 (hereinafter referred to as "auxiliary viewpoint image data") is transmitted to the learning unit 1007 as training data for learning a three-dimensional field model during learning.

[0052] The first image generation unit 1006 generates a virtual viewpoint image using the three-dimensional field model in the middle of learning and virtual viewpoint information acquired from the information acquisition unit 1008, which will be described later. This virtual viewpoint image is an image that allows the user to check the virtual viewpoint image generated by the three-dimensional field model in the middle of learning. The virtual viewpoint image in the middle of learning generated by the first image generation unit 1006 is transmitted to the output unit 1009. The first image generation unit 1006 also generates a virtual viewpoint image that corresponds to the view from the position of the virtual viewpoint indicated by the virtual viewpoint information, using the trained three-dimensional field model and the virtual viewpoint information. The virtual viewpoint image generated by the first image generation unit 1006 using the trained three-dimensional field model is transmitted to the output unit 1009.

[0053] The learning unit 1007 learns the three-dimensional field model. Specifically, the learning unit 1007 learns the three-dimensional field model using the captured image data and the auxiliary viewpoint image data as training data. After all learning is completed, the learning unit 1007 transmits data of the learned three-dimensional field model to the output unit 1009.

[0054] The information acquisition unit 1008 acquires virtual viewpoint information including at least information on the position of the virtual viewpoint and information on the line of sight direction at the virtual viewpoint. The virtual viewpoint information acquired by the information acquisition unit 1008 is transmitted to the first image generation unit 1006.

[0055] The image processing device 102 may be configured so that the user can check the auxiliary viewpoint image generated by the second image generation unit 1005. In this case, the auxiliary viewpoint image is transmitted from the second image generation unit 1005 to the output unit 1009. Thereafter, the output unit 1009 outputs the auxiliary viewpoint image to, for example, the UI panel 103 or the display device 105.

[0056] The output unit 1009 outputs the trained three-dimensional field model. For example, the output unit 1009 outputs data of the trained three-dimensional field model to the storage device 104 and stores the data in the storage device 104. The output destination of the output unit 1009 is not limited to the storage device 104. For example, the output unit 1009 may output a display image signal including an image showing the trained three-dimensional field model to a display device such as the UI panel 103 or the display device 105 and display the image on the display device. Furthermore, the output unit 1009 may output a virtual viewpoint image generated using the trained or trained three-dimensional field model in addition to the trained three-dimensional field model. In this case, the output unit 1009, for example, outputs data of the virtual viewpoint image to the storage device 104 and stores the data in the storage device 104. Furthermore, for example, the output unit 1009 may output the virtual viewpoint image to a display device such as the UI panel 103 or the display device 105 and display the virtual viewpoint image on the display device.

[0057] FIG. 10B is a block diagram showing an example of the functional configuration of the information processing device 108 according to the first embodiment. The information processing device 108 has, as its functional configuration, a model acquisition unit 1051, an information acquisition unit 1052, an image generation unit 1053, and an output unit 1054. Each unit included in the functional configuration of the information processing device 108 is realized by the CPU 251 executing a program stored in the ROM 253 or the like, using the RAM 252 as a work memory. Note that not all of the processes described below necessarily need to be realized by the CPU 251 executing a program. The information processing device 108 may be configured so that some or all of the processes are executed by one or more processing circuits other than the CPU 251. The model acquisition unit 1051 acquires a trained three-dimensional field model output from the image processing device 102. The trained three-dimensional field model acquired by the model acquisition unit 1051 is transmitted to the image generation unit 1053.

[0058] The information acquisition unit 1052 acquires virtual viewpoint information including at least information on the position of the virtual viewpoint and information on the direction of the line of sight at the virtual viewpoint. For example, the virtual viewpoint information is input based on an input operation from the user of the information processing device 108 via the input device 110. The virtual viewpoint information acquired by the information acquisition unit 1052 is transmitted to the image generation unit 1053. Note that the virtual viewpoint information acquired by the information acquisition unit 1052 may be output to the image processing device 102 via the network 111. In this case, the virtual viewpoint information output to the image processing device 102 is acquired by the information acquisition unit 1008. The image generation unit 1053 generates a virtual viewpoint image corresponding to the view from the position of the virtual viewpoint indicated by the virtual viewpoint information, using the trained three-dimensional field model transmitted from the model acquisition unit 1051 and the virtual viewpoint information transmitted from the information acquisition unit 1052. Data of the virtual viewpoint image generated by the image generation unit 1053 is transmitted to the output unit 1054. The output unit 1054 outputs the virtual viewpoint image transmitted from the image generation unit 1053 to the display device 109. The display device 109 displays the virtual viewpoint image output from the output unit 1054.

[0059] <Operation of image processing device> The operations of the image processing device 102 and the information processing device 108 will be described with reference to Fig. 11A and Fig. 11A is a flowchart showing an example of a processing flow in the image processing device 102 according to the first embodiment. First, in S1101, the camera parameter acquisition unit 1001 acquires image capturing camera parameters of each image capturing device 101. Next, in S1102, the camera parameter generation unit 1002 generates auxiliary camera parameters.

[0060] Fig. 12 is a diagram for explaining an example of a method for generating auxiliary camera parameters in the camera parameter acquisition unit 1001 according to the first embodiment. Fig. 12 shows, as an example, a scene in which a plurality of image capture devices 101 are installed on a wall surface 1201, such as a wall or ceiling of a studio, so as to surround a learning space 1203 in the studio, and a radiance field (three-dimensional field) corresponding to the learning space 1203 is learned. Note that Fig. 12 shows, as an example, a cross-sectional view of a certain vertical plane. In Fig. 12, the positions of the image capture devices 101 are indicated by solid-line circles, and the position of the auxiliary viewpoint is indicated by a dash-dotted circle.

[0061] First, the camera parameter generation unit 1002 generates a group of direction vectors connecting the center 1204 of the learning space 1203 and each of the image capture devices 101. In the following description, each direction vector included in the group of direction vectors will be referred to as a direction vector Nr. Next, the camera parameter generation unit 1002 arranges a plurality of points on the surface of a sphere (hereinafter referred to as a "sphere") centered at the center 1204, and generates a group of direction vectors connecting the center 1204 and each of the arranged points. In the following description, each direction vector included in the group of direction vectors will be referred to as a direction vector Nv. For example, the camera parameter generation unit 1002 sets a sphere whose radius is a length equivalent to the distance from the center 1204 to the farthest image capture device 101, and arranges a plurality of points on the spherical surface so that, for example, the plurality of points mutually satisfy Fibonacci lattice points. Arranging points so as to satisfy Fibonacci lattice points is one method of evenly distributing a predetermined number of points on a spherical surface.

[0062] Next, the camera parameter generation unit 1002 identifies a direction vector Nr indicating the direction closest to each direction vector Nv. Subsequently, the camera parameter generation unit 1002 sets, as the position of the auxiliary viewpoint 1205, a point corresponding to the direction vector Nv, where the angle formed by the direction vector Nv and the direction vector Nr is equal to or greater than a predetermined threshold. Furthermore, the camera parameter generation unit 1002 sets the orientation of the virtual camera at the auxiliary viewpoint 1205 based on the direction vector Nv. Subsequently, the camera parameter generation unit 1002 sets internal parameters of the auxiliary viewpoint 1205, for example, so that the entire learning space 1203 is included in the angle of view of the virtual camera. Note that in this embodiment, the description is given assuming that the captured image and the auxiliary viewpoint image have the same size. In this manner, the camera parameter generation unit 1002 generates auxiliary camera parameters by setting external parameters and internal parameters for the auxiliary viewpoint. Here, the camera parameter generation unit 1002 also generates distortion parameters and other auxiliary camera parameters. For example, when generating an undistorted virtual viewpoint image, the camera parameter generation unit 1002 sets the distortion parameter of the auxiliary camera parameter to a parameter (such as zero) that indicates no distortion. In this way, the camera parameter generation unit 1002 generates auxiliary camera parameters in accordance with the camera parameter format.

[0063] After S1102, in S1103, the image acquisition unit 1003 acquires captured image data obtained by capturing images with each imaging device 101. Next, in S1104, the shape acquisition unit 1004 estimates the three-dimensional shape of the object based on the capturing camera parameters and the captured image data, and generates shape data indicating the three-dimensional shape of the object. Specifically, for example, the shape acquisition unit 1004 extracts an area corresponding to the image of the object from each captured image to generate a silhouette image, and estimates the three-dimensional shape of the object by a volume intersection method using the silhouette image.

[0064] Hereinafter, the shape data will be described as point cloud data that represents positions corresponding to the surface of the object using a point cloud, but the shape data may also be data expressed using a polygon mesh, voxels, or the like. In addition, although the above description has been given assuming that the three-dimensional shape of the object is estimated using a volume intersection method, the method for estimating the three-dimensional shape of the object is not limited to the volume intersection method. For example, the shape acquisition unit 1004 may estimate the three-dimensional shape of the object by using images captured by two adjacent image capture devices 101 as stereo cameras and estimating the distance to the surface of the object using the captured images captured by these image capture devices 101.

[0065] In S1105, the second image generation unit 1005 determines the color of each point on the surface of the three-dimensional shape of the object represented by the shape data based on the imaging camera parameters and the captured image data. Specifically, the second image generation unit 1005 first projects the three-dimensional shape onto the positions of the imaging devices 101 to generate a depth map, and determines from which imaging devices 101 each point on the surface of the three-dimensional shape is visible. Next, the second image generation unit 1005 samples pixel values ​​from the captured images of one or more imaging devices 101 determined to be visible, and determines the color of each point by, for example, calculating the average value of the sampled pixel values. The second image generation unit 1005 assigns color information indicating the determined color to the shape data in association with information on the corresponding point. Note that in the present embodiment, the second image generation unit 1005 has been described as executing the processing of S1105, but the processing of S1105 may also be executed by the shape acquisition unit 1004.

[0066] Next, in S1106, the second image generation unit 1005 generates an image corresponding to the view from the auxiliary viewpoint (auxiliary viewpoint image) based on the shape data to which color information has been added and the auxiliary camera parameters. Next, in S1108, the learning unit 1007 learns a three-dimensional field model using the captured image data and the auxiliary viewpoint image data as training data. That is, in this embodiment, the learning unit 1007 learns a radiance field using the above-mentioned NeRF using the captured image data and the auxiliary viewpoint image data as training data. During the learning process in S1108, the first image generation unit 1006 generates a virtual viewpoint image for the user to check. The output unit 1009 uses the virtual viewpoint image generated by the first image generation unit 1006 to generate a GUI, which will be described later with reference to FIG. 13, and outputs the GUI to a display device such as the UI panel 103 or the display device 105. Note that if the user does not need to check the virtual viewpoint image, the process of generating the virtual viewpoint image and the GUI and the process of outputting the GUI in S1108 may not be performed.

[0067] Next, in S1109, the learning unit 1007 determines whether to terminate learning of the three-dimensional field model. For example, the learning unit 1007 determines whether to terminate learning of the three-dimensional field model by determining whether learning of the three-dimensional field model has been performed for a predetermined period or number of times. The conditions for determining whether learning of the three-dimensional field model is terminated are not limited to those described above. For example, the learning unit 1007 may determine whether to terminate learning of the three-dimensional field model by determining whether the loss L is equal to or less than a predetermined threshold and determining whether learning of the three-dimensional field model has converged.

[0068] If it is determined in S1109 that the learning of the three-dimensional field model should not be terminated, the image processing device 102 terminates the processing of the flowchart shown in Fig. 11. Thereafter, the image processing device 102 repeatedly executes the processing of the flowchart shown in Fig. 11 until it is determined in S1109 that the learning of the three-dimensional field model should be terminated. In this case, the processing of S1101 and S1103 to S1105 can be omitted.

[0069] If it is determined in S1109 that the learning of the three-dimensional field model is to be ended, in S1110, the output unit 1009 outputs the trained three-dimensional field model obtained as a result of learning by the training unit 1007. For example, when the trained three-dimensional field model is to be output as data, the output unit 1009 outputs the network parameters of the trained three-dimensional field model as a file. In this case, in this embodiment, the network parameters of the NeRF model representing the trained radiance field are output as a file.

[0070] After S1110, in S1111, the information acquisition unit 1008 acquires virtual viewpoint information. Next, in S1112, the first image generation unit 1006 uses the trained three-dimensional field model to generate a virtual viewpoint image based on the virtual viewpoint information (hereinafter referred to as a "trained virtual viewpoint image"). Next, in S1113, the output unit 1009 outputs the trained virtual viewpoint image generated in S1112. After S1113, the image processing device 102 ends the processing of the flowchart shown in FIG. 11.

[0071] Note that, if the captured images are moving images, the image processing device 102 repeatedly executes the processing of the flowchart shown in FIG. 11 each time frame data at a new time is acquired in S1103. In this case, the processing of S1101 and S1102 can be omitted. Furthermore, the image processing device 102 may generate trained virtual viewpoint images corresponding to a plurality of virtual viewpoints using a trained three-dimensional field model. In this case, for example, the image processing device 102 repeatedly executes the processing from S1111 to S1113 each time new virtual viewpoint information is acquired in S1111. Furthermore, if the virtual viewpoint information acquired in S1111 includes information indicating the position and line of sight of the virtual viewpoint in time series, the image processing device 102 may repeatedly execute the processing of S1112 and S1113 to generate and output trained virtual viewpoint images in time series.

[0072] FIG. 13 is a diagram showing an example of a GUI 1300 displayed on a display device such as the UI panel 103 or the display device 105 according to the first embodiment. The GUI 1300 includes display images corresponding to the captured image, the auxiliary viewpoint image, and the virtual viewpoint image. The GUI 1300 also enables setting of options for learning the three-dimensional field model, such as the process of acquiring the three-dimensional shape, the process of generating the auxiliary viewpoint image, the process of adding color information, the mode of the learning model, and the mode of the learning step. The GUI 1300 also enables the user to check the captured image, the auxiliary viewpoint image generated by the second image generation unit 1005, and the virtual viewpoint image generated using the three-dimensional field model during learning and after learning.

[0073] FIG. 11B is a flowchart showing an example of a processing flow in the information processing device 108 according to the first embodiment. First, in S1151, the model acquisition unit 1051 acquires a trained three-dimensional field model output from the image processing device 102. Next, in S1152, the information acquisition unit 1052 acquires virtual viewpoint information. Next, in S1153, the image generation unit 1053 uses the trained three-dimensional field model acquired in S1051 and the virtual viewpoint information acquired in S1153 to generate a virtual viewpoint image (trained virtual viewpoint image) corresponding to the appearance from the virtual viewpoint indicated by the virtual viewpoint information. Next, in S1154, the output unit 1054 outputs the trained virtual viewpoint image generated in S1153. After S1154, the information processing device 108 ends the processing of the flowchart shown in FIG. 11B.

[0074] The image processing device 102 configured as described above can suppress a decrease in the reproducibility of the image of an object included in a virtual viewpoint image even when the image capturing position of the image capturing device 101 or the attitude of the image capturing device 101 differs greatly from the position or direction of the virtual viewpoint. Furthermore, the image processing device 102 can generate a three-dimensional field model that can generate a virtual viewpoint image in which a decrease in the reproducibility of the image of an object is suppressed, even in the above-mentioned case.

[0075] [Second embodiment] In the first embodiment, a method was described in which, in learning a three-dimensional field model, an auxiliary viewpoint is set in a direction where the imaging device 101 is not present and auxiliary viewpoint image data is added to the training data to suppress deterioration in the reproducibility of a virtual viewpoint image corresponding to the view from that direction. However, because the auxiliary viewpoint image is a virtual image generated by a simple method, when optimizing learning of the three-dimensional field model, it may be better to train the three-dimensional field model with emphasis on the color of the captured image. In this embodiment, a method is described in which learning using the auxiliary viewpoint image (hereinafter referred to as "initial learning") is performed, followed by learning using the captured image (hereinafter referred to as "main learning") to obtain a more accurate trained three-dimensional field model.

[0076] <Configuration of image processing device> FIG. 14 is a block diagram showing an example of the functional configuration of an image processing device 102 (hereinafter simply referred to as "image processing device 102") according to the second embodiment. Note that the configuration of the image processing system according to the second embodiment is the same as the configuration of the image processing system according to the first embodiment shown as an example in FIG. 1, and therefore a description of the configuration of the image processing system will be omitted. The image processing device 102 has, as its functional configuration, a camera parameter acquisition unit 1001, a camera parameter generation unit 1402, an image acquisition unit 1003, a shape acquisition unit 1004, a second image generation unit 1405, a first image generation unit 1006, a learning unit 1407, and an information acquisition unit 1008. In addition to the functional configuration, the image processing device 102 also has an output unit 1409, a space determination unit 1410, and a shape generation unit 1411. Below, functional configurations different from those of the image processing device 102 according to the first embodiment will be described.

[0077] Each unit that the image processing device 102 has as a functional configuration is realized by the CPU 201 executing a program stored in the ROM 203, etc., using the RAM 202 as a work memory. Note that not all of the processes described below necessarily need to be realized by the CPU 201 executing a program, and the image processing device 102 may be configured so that part or all of the processes are executed by one or more processing circuits other than the CPU 201.

[0078] The space determination unit 1410 determines a learning space based on the shape data transmitted from the shape acquisition unit 1004. Specifically, for example, the space determination unit 1410 determines, as the learning space, a space corresponding to a rectangular parallelepiped that contains the three-dimensional shape of the object indicated by the shape data in the imaging space. Learning space information indicating the learning space determined by the space determination unit 1410 is output to the camera parameter generation unit 1402.

[0079] The camera parameter generation unit 1402 generates auxiliary camera parameters based on the learning space information transmitted from the space determination unit 1410 and the imaging camera parameters transmitted from the camera parameter acquisition unit 1001. Details of the auxiliary camera parameter generation process in the camera parameter generation unit 1402 will be described later. The auxiliary camera parameters generated by the camera parameter generation unit 1402 are transmitted to the first image generation unit 1006, the second image generation unit 1405, and the learning unit 1407.

[0080] The second image generation unit 1405 generates an auxiliary viewpoint image using the auxiliary camera parameters, captured image data, and shape data received from the camera parameter generation unit 1402, the image acquisition unit 1003, or the shape acquisition unit 1004. Details of the auxiliary viewpoint image generation process in the second image generation unit 1405 will be described later. The data of the auxiliary viewpoint image generated by the second image generation unit 1405 is transmitted to the learning unit 1407 as teacher data for performing initial learning of a three-dimensional field model during learning.

[0081] The learning unit 1407 performs initial learning of the three-dimensional field model using auxiliary viewpoint image data as training data, and main learning of the three-dimensional field model using captured image data as training data. Details of the learning process of the three-dimensional field model in the learning unit 1407 will be described later. After all learning is completed, the learning unit 1407 transmits data of the learned three-dimensional field model to the output unit 1409 and the shape generation unit 1411.

[0082] The shape generation unit 1411 generates three-dimensional shape data based on the trained three-dimensional field model. Specifically, the shape generation unit 1411 extracts information about density from the trained three-dimensional field model and generates three-dimensional shape data by converting the extracted information into data indicating the three-dimensional shape of the surface of the object using a polygon mesh or the like. The three-dimensional shape data generated by the shape generation unit 1411 is transmitted to the output unit 1409.

[0083] The output unit 1409 outputs the three-dimensional shape data transmitted from the shape generation unit 1411. For example, the output unit 1409 outputs the three-dimensional shape data to the storage device 104 and stores it therein. The output unit 1409 also outputs a trained three-dimensional field model. For example, the output unit 1409 outputs data of the trained three-dimensional field model to the storage device 104 and stores the data therein. The output destination of the output unit 1409 is not limited to the storage device 104. For example, the output unit 1409 may output a display image signal including an image showing the trained three-dimensional field model to a display device such as the UI panel 103 or the display device 105, and display the image on the display device. The output unit 1409 may also output a virtual viewpoint image that is in the middle of training or has already been trained, in addition to the trained three-dimensional field model. In this case, for example, the output unit 1409 outputs data of the virtual viewpoint image to the storage device 104 and stores the data in the storage device 104. Furthermore, for example, the output unit 1409 may output the virtual viewpoint image to a display device such as the UI panel 103 or the display device 105, and cause the learned virtual viewpoint image to be displayed on the display device.

[0084] <Operation of image processing device> FIG. 15 is a flowchart showing an example of a processing flow of the image processing device 102 according to the second embodiment. First, the image processing device 102 sequentially executes the processes of S1101, S1103, and S1104. After S1104, in S1501, the space determination unit 1410 determines, for example, a space that contains a space corresponding to the three-dimensional shape of the object as the learning space based on the shape data generated in S1104. Specifically, for example, the space determination unit 1410 determines, as the learning space, a rectangular parallelepiped space that circumscribes the space corresponding to the three-dimensional shape, and generates information indicating the learning space (learning space information). Next, in S1502, the camera parameter generation unit 1402 generates auxiliary camera parameters based on the learning space information generated in S1501 and the imaging camera parameters acquired in S1101. Details of the auxiliary camera parameter generation process in S1502 will be described later with reference to FIG. 16.

[0085] Next, in S1503, the second image generation unit 1405 generates an auxiliary viewpoint image based on the captured image data acquired in S1103, the shape data generated in S1104, and the auxiliary camera parameters generated in S1501. Specifically, in S1503, the second image generation unit 1405 first generates a depth map corresponding to each auxiliary viewpoint by projecting the three-dimensional shape of the object indicated by the shape data onto the position of the auxiliary viewpoint indicated by each auxiliary camera parameter. Next, the second image generation unit 1405 acquires three-dimensional coordinates corresponding to the surface of the object based on the depth values ​​of each depth map. Next, the second image generation unit 1405 determines the visibility of each acquired three-dimensional coordinate from each image capture device 101, and samples pixel values ​​of the captured image of one or more image capture devices 101 corresponding to the three-dimensional coordinates determined to be visible.

[0086] Next, the color of each pixel in the auxiliary viewpoint image is determined based on one or more sampled pixel values. Specifically, when it is determined that the three-dimensional coordinates are visible from multiple image capture devices 101, the second image generation unit 1405 determines the color of each pixel in the auxiliary viewpoint image by calculating statistical values ​​such as the average value of pixel values ​​sampled from images captured by each image capture device 101. For example, the second image generation unit 1405 calculates an arithmetic average by weighting pixel values ​​of images captured by an image capture device 101 whose position and orientation are the same as or similar to the position and line of sight of the auxiliary viewpoint. Calculating an arithmetic average in this manner makes it possible to obtain an auxiliary viewpoint image with higher reproducibility.

[0087] After S1503, in S1504, the learning unit 1407 performs initial learning of a three-dimensional field in the learning space using the data of the auxiliary viewpoint image generated in S1503 as training data. During the initial learning process in S1504, the first image generation unit 1006 generates a virtual viewpoint image for the user to confirm. The output unit 1009 uses the virtual viewpoint image generated by the first image generation unit 1006 to generate a GUI, an example of which is shown in FIG. 13, and outputs the GUI to a display device such as the UI panel 103 or the display device 105. Note that if the user does not need to confirm the virtual viewpoint image, the virtual viewpoint image and GUI generation process and the GUI output process in S1504 may not be performed. Next, in S1505, the learning unit 1407 performs main learning of a three-dimensional field in the learning space using the captured image data acquired in S1103 as training data. Specifically, the main learning performed in S1505 is relearning, using the three-dimensional field after the initial learning in step S1504 as the initial value. During this learning process in S1505, the first image generation unit 1006 generates a virtual viewpoint image for the user to check. The output unit 1009 uses the virtual viewpoint image generated by the first image generation unit 1006 to generate a GUI, an example of which is shown in Fig. 13, and outputs the GUI to a display device such as the UI panel 103 or the display device 105. Note that if the user does not need to check the virtual viewpoint image, the process of generating the virtual viewpoint image and GUI and the process of outputting the GUI in S1505 may not be performed.

[0088] Next, the image processing device 102 executes the process of S1109. If it is determined in S1109 not to end the learning of the three-dimensional field model, the image processing device 102 ends the process of the flowchart shown in Fig. 15. Thereafter, the image processing device 102 repeatedly executes the process of the flowchart shown in Fig. 15 until it is determined in S1109 to end the learning of the three-dimensional field model. In this case, the processes of S1101 and S1103 can be omitted.

[0089] If it is determined in S1109 that learning of the three-dimensional field model is to be ended, in S1506, the shape generation unit 1411 generates three-dimensional shape data based on the learned three-dimensional field model. After S1506, in S1507, the output unit 1409 outputs the three-dimensional shape data generated in S1506. After S1507, the image processing device 102 executes the processes from S1110 to S1113. After S1113, the image processing device 102 ends the process of the flowchart shown in FIG. 15.

[0090] Note that, if the captured images are moving images, the image processing device 102 repeatedly executes the processing of the flowchart shown in FIG. 15 each time frame data at a new time is acquired in S1103. In this case, the processing of S1101 can be omitted. Also, the image processing device 102 may generate trained virtual viewpoint images corresponding to a plurality of virtual viewpoints using a trained three-dimensional field model. In this case, for example, the image processing device 102 repeatedly executes the processing from S1111 to S1113 each time new virtual viewpoint information is acquired in S1111. Also, if the virtual viewpoint information acquired in S1111 includes information indicating the position and line of sight of the virtual viewpoint in time series, the image processing device 102 may repeatedly execute the processing of S1112 and S1113 to generate and output trained virtual viewpoint images in time series.

[0091] <Generation of auxiliary camera parameters> FIG. 16 is a flowchart showing an example of the flow of the auxiliary camera parameter generation process in the camera parameter generation unit 1402 according to the second embodiment, and is a flowchart showing an example of the flow of the camera parameter generation process of S1502 shown in FIG.

[0092] First, in S1601, the camera parameter generation unit 1402 sets a curved surface on which an auxiliary viewpoint is set based on the distance from the center of the learning space to each image capture device 101. Specifically, for example, the camera parameter generation unit 1402 defines a spherical surface with a radius R that is 10 times the maximum distance from the center of the learning space to each image capture device 101 and a center at the center of the learning space, and sets the spherical surface as the curved surface on which the auxiliary viewpoint is set. By setting the distance from the auxiliary viewpoint to the learning space longer than the distance from the image capture device 101 to the learning space, it is possible to reduce the fluctuation in local resolution within the learning space. As a result, it is possible to learn the entire learning space with a constant resolution. Note that the length of the radius is arbitrary. For example, the image processing device 102 may be configured so that the user can set the length of the radius R or a multiple value of the distance from the center of the learning space to each image capture device 101. Furthermore, in this embodiment, the curved surface on which the auxiliary viewpoint is set is described as a spherical surface, but the curved surface is not limited to a spherical surface.

[0093] Next, in S1602, the camera parameter generation unit 1402 determines the position of the auxiliary viewpoint on the curved surface set in S1601. Specifically, for example, the camera parameter generation unit 1402 determines the position of a Fibonacci lattice point that is a distance from the center of the learning space that corresponds to the above-mentioned radius R as the position of the auxiliary viewpoint. The number of Fibonacci lattice points is a predetermined number, such as 120, and the camera parameter generation unit 1402 determines the positions of all of the Fibonacci lattice points as the positions of the auxiliary viewpoint. The number of Fibonacci lattice points may be input by the user via the UI panel 103. Furthermore, if there is a purpose such as simplifying the initial learning of a three-dimensional field, reducing the number of Fibonacci lattice points to a predetermined number, such as 25, does not significantly adversely affect the initial learning of a three-dimensional field.

[0094] Next, in S1603, the camera parameter generation unit 1402 determines the line of sight direction from each auxiliary viewpoint determined in S1602. Specifically, for example, the camera parameter generation unit 1402 determines the line of sight direction from the auxiliary viewpoint to be a direction from the auxiliary viewpoint toward the center of the learning space. In other words, the line of sight direction from the auxiliary viewpoint may be represented by a direction vector indicating the direction from the position of the auxiliary viewpoint toward the center of the learning space.

[0095] Next, in S1604, the camera parameter generation unit 1402 acquires the resolution of the captured image in the learning space (hereinafter referred to as "capture resolution"). Specifically, first, the camera parameter generation unit 1402 calculates the capture resolution K [mm / pix] at the center of the learning space when viewed from the respective positions of all the image capture devices 101. The capture resolution K is calculated, for example, by dividing the depth z [mm] corresponding to the distance from the image capture device 101 to the learning space by the focal length f [pix] expressed in pixel values. Next, the camera parameter generation unit 1402 specifies the maximum value Kmax of the computed capture resolution K corresponding to all the image capture devices 101.

[0096] Next, in S1605, the camera parameter generation unit 1402 sets the resolution of the auxiliary viewpoint image (hereinafter referred to as "auxiliary viewpoint resolution") to twice the maximum value Kmax, which is the worst imaging resolution. Here, the auxiliary viewpoint is set on a sphere with a radius R such that the distance from the auxiliary viewpoint to the center of the learning space is the distance from each imaging device 101 to the center of the learning space, that is, 10 times the maximum value of the depth z. Therefore, the auxiliary viewpoint resolution is 2Kmax=10z / fi. Here, fi is the focal length of the auxiliary viewpoint, which is a value calculated by fi=5z / Kmax.

[0097] The initial learning using auxiliary-viewpoint image data as training data is not intended to precisely reproduce an object. In other words, the purpose of the initial learning is to provide initial values ​​for a three-dimensional field model prior to main learning using captured image data as training data so that the actual object does not differ significantly from the actual object when viewed from any direction. Therefore, detailed learning is not necessary in the initial learning using auxiliary-viewpoint image data as training data. Therefore, as described above, the auxiliary-viewpoint resolution is set to a resolution lower than the imaging resolution. This setting reduces the amount of computation required for initial learning using auxiliary-viewpoint image data as training data. In this embodiment, the auxiliary-viewpoint resolution is set to twice the maximum value Kmax of the imaging resolution K. However, this ratio is arbitrary, and the image processing device 102 may be configured so that it can be set by the user via a user interface, for example.

[0098] After S1605, in S1606, the camera parameter generation unit 1402 determines the image size of the auxiliary viewpoint image. Specifically, first, the camera parameter generation unit 1402 acquires the positions of eight corners of the learning space as horizontal and vertical values ​​[mm] in each camera coordinate system at the positions of all the image capture devices 101. Next, the camera parameter generation unit 1402 identifies the minimum values ​​[mm] and maximum values ​​[mm] in the horizontal and vertical directions in all the acquired camera coordinate systems. Next, the camera parameter generation unit 1402 divides each of the identified minimum and maximum values ​​by fi to acquire the minimum number of pixels [pix] and the maximum number of pixels [pix] in the horizontal and vertical directions in the image coordinate system.

[0099] Next, the camera parameter generation unit 1402 determines the image size of the auxiliary viewpoint image based on the minimum number of pixels [pix] and the maximum number of pixels in the horizontal and vertical directions in the acquired image coordinate system. For example, the camera parameter generation unit 1402 determines the image size of the auxiliary viewpoint image by setting the image size in the horizontal and vertical directions in the image coordinate system to the maximum number of pixels - the minimum number of pixels + 1. Note that the principal point of the auxiliary viewpoint image is set to the determined image size / 2. After S1606, the camera parameter generation unit 1402 ends the processing of the flowchart shown in FIG. 14, i.e., the processing of S1502.

[0100] In this embodiment, the auxiliary viewpoint resolution is determined based on the imaging resolution K. However, the method for determining the auxiliary viewpoint resolution is not limited to this. For example, if the three-dimensional field model to be learned is expressed by voxels such as Plenoxels or TensoRF, the auxiliary viewpoint resolution may be determined based on the size [mm] of the voxels.

[0101] As described above, the image processing device 102 is configured so that the auxiliary viewpoint resolution is lower than the imaging resolution. Furthermore, in learning a three-dimensional field model, the image processing device 102 is configured so that initial learning is performed using the auxiliary viewpoint image data as training data before main learning is performed using the captured image data as training data. The image processing device 102 configured as described above can reduce the amount of calculation in initial learning using the auxiliary viewpoint image data as training data while suppressing a decrease in the reproducibility of the three-dimensional field in directions in which the imaging device 101 does not exist as viewed from the learning space.

[0102] [Other embodiments] In the above-described embodiment, the auxiliary-viewpoint image is generated based on the captured image. However, the method for generating the auxiliary-viewpoint image is not necessarily limited to this. For example, the auxiliary-viewpoint image may be generated based on a random texture pattern that is artificially generated in advance. Generating the auxiliary-viewpoint image based on a random texture pattern allows for clearer initial learning of the three-dimensional field related to the density of objects in a direction where fewer imaging devices 101 are installed. However, the color information in the auxiliary-viewpoint image data generated based on a random texture pattern significantly deviates from the three-dimensional field that should be learned. Therefore, in this case, it is preferable to initialize color-related parameters in the three-dimensional field model obtained by initial learning using the auxiliary-viewpoint image data as training data before actual learning using the captured image data as training data. The image processing device 102 configured in this manner can improve the reproducibility of the three-dimensional field of objects in a direction where fewer imaging devices 101 are installed.

[0103] The present disclosure can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0104] It should be noted that within the scope of the present disclosure, the embodiments may be freely combined, any component of each embodiment may be modified, or any component of each embodiment may be omitted.

[0105] [Configuration of the present disclosure] The present disclosure includes the following configurations, methods, and programs.

[0106] <Configuration 1> an image acquisition means for acquiring data of a plurality of captured images obtained by capturing images from a plurality of positions; a camera parameter acquisition means for acquiring a plurality of camera parameters for a plurality of viewpoints corresponding to the plurality of positions; a camera parameter generating means for generating camera parameters for an auxiliary viewpoint different from the plurality of viewpoints; a shape acquisition means for acquiring shape data indicating a three-dimensional shape of an object estimated based on the plurality of camera parameters acquired by the camera parameter acquisition means and the data of the plurality of captured images acquired by the image acquisition means; an image generating means for generating an auxiliary viewpoint image corresponding to an appearance from the auxiliary viewpoint based on the shape data and the camera parameters generated by the camera parameter generating means; an information generating means for generating three-dimensional field information regarding a three-dimensional field corresponding to at least a part of a space in an imaging space imaged from the plurality of positions, based on the plurality of camera parameters acquired by the camera parameter acquiring means, data of the plurality of captured images, camera parameters generated by the camera parameter generating means, and data of the auxiliary viewpoint image; An image processing device comprising:

[0107] <Configuration 2> the shape acquisition means acquires the shape data by estimating a three-dimensional shape of the object based on the plurality of camera parameters acquired by the camera parameter acquisition means and data of the plurality of captured images, and generating the shape data; 2. The image processing device according to configuration 1,

[0108] <Configuration 3> the shape data includes color information indicating a color of a surface of the object that is determined based on data of the plurality of captured images; 3. The image processing device according to configuration 1 or 2, characterized in that:

[0109] <Configuration 4> the shape data includes color information indicating a color of a surface of the object that is determined without using data of the plurality of captured images; 3. The image processing device according to configuration 1 or 2, characterized in that:

[0110] <Configuration 5> the shape data includes color information indicating a surface color of the object, the color information being determined based on image data having a predetermined texture pattern; 5. The image processing device according to configuration 4,

[0111] <Configuration 6> the camera parameter generating means places the auxiliary viewpoint outside the imaging space and generates camera parameters corresponding to the placed auxiliary viewpoint; 6. The image processing device according to any one of configurations 1 to 5, characterized in that:

[0112] <Configuration 7> the camera parameter generating means sets the position of the auxiliary viewpoint farther from the imaging space than the plurality of positions; 7. The image processing device according to configuration 6,

[0113] <Configuration 8> the image generation means generates the auxiliary viewpoint image having a resolution equal to or lower than a resolution of the plurality of captured images; 8. The image processing device according to any one of configurations 1 to 7, characterized in that:

[0114] <Configuration 9> the image generation means determines a resolution of the auxiliary viewpoint image to be generated in accordance with a resolution of the three-dimensional field; 9. The image processing device according to any one of configurations 1 to 8,

[0115] <Configuration 10> the information generating means determines a space in which the three-dimensional field information is to be generated based on the shape data; 10. The image processing device according to any one of configurations 1 to 9, characterized in that:

[0116] <Configuration 11> the three-dimensional field information is a trained model relating to the three-dimensional field; 11. The image processing device according to any one of configurations 1 to 10, characterized in that:

[0117] <Configuration 12> the information generation means generates the trained model by performing training of a training model related to the three-dimensional field using data of the plurality of captured images and data of the auxiliary viewpoint image as teacher data; 12. The image processing device according to configuration 11,

[0118] <Configuration 13> the information generating means sets a weight for learning using data of the plurality of captured images to be greater than a weight for learning using data of the auxiliary viewpoint image; 13. The image processing device according to configuration 12,

[0119] <Configuration 14> the information generating means performs learning using data of the plurality of captured images after learning using data of the auxiliary viewpoint image; 14. The image processing device according to configuration 12 or 13,

[0120] <Configuration 15> the information generating means initializes information about colors in the learning model during learning after learning using data of the auxiliary viewpoint image, and learns the learning model after the initialization using data of the plurality of captured images; 15. The image processing device according to configuration 14,

[0121] <Method> an image acquisition step of acquiring data of a plurality of captured images obtained by capturing images from a plurality of positions; a camera parameter acquisition step of acquiring a plurality of camera parameters for a plurality of viewpoints corresponding to the plurality of positions; a camera parameter generating step of generating camera parameters for an auxiliary viewpoint different from the plurality of viewpoints; a shape acquisition step of acquiring shape data indicating a three-dimensional shape of an object estimated based on the plurality of camera parameters acquired in the camera parameter acquisition step and the data of the plurality of captured images acquired in the image acquisition step; an image generating step of generating an auxiliary viewpoint image corresponding to an appearance from the auxiliary viewpoint based on the shape data and the camera parameters generated in the camera parameter generating step; an information generating step of generating three-dimensional field information regarding a three-dimensional field corresponding to at least a part of a space in an imaging space imaged from the plurality of positions, based on the plurality of camera parameters acquired in the camera parameter acquiring step, data of the plurality of captured images, camera parameters generated in the camera parameter generating step, and data of the auxiliary viewpoint image; An image processing method comprising:

[0122] <Program> A program for causing a computer to function as the image processing device according to any one of configurations 1 to 15. [Explanation of symbols]

[0123] 102 Image processing device 1003 Image acquisition unit 1001 Camera parameter acquisition unit 1002 camera parameter generation unit 1004 Shape acquisition section 1005 Second image generation unit 1007 Learning Department

Claims

1. an image acquisition means for acquiring data of a plurality of captured images obtained by capturing images from a plurality of positions; a camera parameter acquisition means for acquiring a plurality of camera parameters for a plurality of viewpoints corresponding to the plurality of positions; a camera parameter generating means for generating camera parameters for an auxiliary viewpoint different from the plurality of viewpoints; a shape acquisition means for acquiring shape data indicating a three-dimensional shape of an object estimated based on the plurality of camera parameters acquired by the camera parameter acquisition means and the data of the plurality of captured images acquired by the image acquisition means; an image generating means for generating an auxiliary viewpoint image corresponding to an appearance from the auxiliary viewpoint based on the shape data and the camera parameters generated by the camera parameter generating means; an information generating means for generating three-dimensional field information regarding a three-dimensional field corresponding to at least a part of a space in an imaging space imaged from the plurality of positions, based on the plurality of camera parameters acquired by the camera parameter acquiring means, data of the plurality of captured images, camera parameters generated by the camera parameter generating means, and data of the auxiliary viewpoint image; An image processing device comprising:

2. the shape acquisition means acquires the shape data by estimating a three-dimensional shape of the object based on the plurality of camera parameters acquired by the camera parameter acquisition means and data of the plurality of captured images, and generating the shape data; 2. The image processing device according to claim 1, wherein:

3. the shape data includes color information indicating a color of a surface of the object that is determined based on data of the plurality of captured images; 2. The image processing device according to claim 1, wherein:

4. the shape data includes color information indicating a color of a surface of the object that is determined without using data of the plurality of captured images; 2. The image processing device according to claim 1, wherein:

5. the shape data includes color information indicating a surface color of the object that is determined based on image data having a predetermined texture pattern; 5. The image processing device according to claim 4, wherein:

6. the camera parameter generating means places the auxiliary viewpoint outside the imaging space and generates camera parameters corresponding to the placed auxiliary viewpoint; 2. The image processing device according to claim 1, wherein:

7. the camera parameter generating means sets the position of the auxiliary viewpoint farther from the imaging space than the plurality of positions; 7. The image processing device according to claim 6,

8. the image generation means generates the auxiliary viewpoint image having a resolution equal to or lower than a resolution of the plurality of captured images; 2. The image processing device according to claim 1, wherein:

9. the image generation means determines a resolution of the auxiliary viewpoint image to be generated in accordance with a resolution of the three-dimensional field; 2. The image processing device according to claim 1, wherein:

10. the information generating means determines a space in which the three-dimensional field information is to be generated based on the shape data; 2. The image processing device according to claim 1, wherein:

11. the three-dimensional field information is a trained model relating to the three-dimensional field; 2. The image processing device according to claim 1, wherein:

12. the information generation means generates the trained model by performing training of a training model related to the three-dimensional field using data of the plurality of captured images and data of the auxiliary viewpoint image as teacher data; The image processing device according to claim 11 ,

13. the information generating means sets a weight for learning using data of the plurality of captured images to be greater than a weight for learning using data of the auxiliary viewpoint image; The image processing device according to claim 12 ,

14. the information generating means performs learning using data of the plurality of captured images after learning using data of the auxiliary viewpoint image; The image processing device according to claim 12 ,

15. the information generating means initializes information about colors in the learning model during learning after learning using data of the auxiliary viewpoint image, and learns the learning model after the initialization using data of the plurality of captured images; The image processing device according to claim 14,

16. an image acquisition step of acquiring data of a plurality of captured images obtained by capturing images from a plurality of positions; a camera parameter acquisition step of acquiring a plurality of camera parameters for a plurality of viewpoints corresponding to the plurality of positions; a camera parameter generating step of generating camera parameters for an auxiliary viewpoint different from the plurality of viewpoints; a shape acquisition step of acquiring shape data indicating a three-dimensional shape of an object estimated based on the plurality of camera parameters acquired in the camera parameter acquisition step and the data of the plurality of captured images acquired in the image acquisition step; an image generating step of generating an auxiliary viewpoint image corresponding to an appearance from the auxiliary viewpoint based on the shape data and the camera parameters generated in the camera parameter generating step; an information generating step of generating three-dimensional field information regarding a three-dimensional field corresponding to at least a part of a space in an imaging space imaged from the plurality of positions, based on the plurality of camera parameters acquired in the camera parameter acquiring step, data of the plurality of captured images, camera parameters generated in the camera parameter generating step, and data of the auxiliary viewpoint image; An image processing method comprising:

17. A program for causing a computer to function as the image processing device according to any one of claims 1 to 15.

Images