Image processing device, image processing method, and program

Abstract

We estimate a three-dimensional field capable of generating virtual viewpoint images that do not exhibit any noticeable inconsistencies in image quality when switching virtual viewpoints. [Solution] The image processing device 102 acquires multiple captured images obtained from multiple viewpoints and camera parameters corresponding to each viewpoint, acquires the shooting resolution which is the resolution of each captured image, generates multiple training images corresponding to each captured image by converting the resolution of each captured image based on the resolution of each of the multiple training images corresponding to each captured image which is determined based on the shooting resolution of each captured image, generates camera parameters corresponding to each training image by converting the camera parameters corresponding to each viewpoint based on the resolution of each training image, and learns a learning model concerning the three-dimensional field of the target space based on the multiple training images and the camera parameters corresponding to each training image.

Description

Technical Field

[0001] The present disclosure relates to a technique for estimating a three-dimensional field based on a plurality of captured images obtained by capturing from a plurality of directions.

Background Art

[0002] There is a technique for estimating a three-dimensional field related to an object by using a plurality of captured images (hereinafter referred to as "multi-viewpoint images") obtained by capturing an object existing in a target space from various positions and directions. Non-Patent Document 1 discloses a technique for estimating a radiance field related to an object called "NeRF (Neural Radiance Fields)" as a technique for estimating a three-dimensional field related to an object using multi-viewpoint images. In NeRF, learning of a neural network that outputs information regarding color and density for any position and direction in a space viewed from various positions and directions is performed, and a learned neural network indicating a radiance field related to an object in the space is obtained. Further, in NeRF, an image (hereinafter referred to as a "virtual viewpoint image") corresponding to the appearance when viewing the target space from an arbitrary virtual viewpoint (hereinafter referred to as a "virtual viewpoint") is generated by volume rendering using the learned neural network.

[0003] The image quality of virtual viewpoint images generated by NeRF depends on the resolution of the images used as training data in the training of the neural network described above (hereinafter referred to as "training images"). For example, virtual viewpoint images generated at the same resolution as the training images will be of high quality, but virtual viewpoint images generated at a lower resolution than the training images will have jagged edges and be of low quality. Conversely, virtual viewpoint images generated at a higher resolution than the training images will have blur and be of similarly low quality. This means that the technology disclosed in Non-Patent Document 1 has a problem in that the image quality deteriorates in virtual viewpoint images corresponding to virtual viewpoints that are farther away from the position where the training images were taken (hereinafter referred to as "shooting viewpoint") and virtual viewpoints that are closer to the shooting viewpoint.

[0004] Non-patent document 2 discloses a technique called "Mip-NeRF" as an improved NeRF technique that can solve the aforementioned problems. Mip-NeRF is characterized by training a neural network that outputs information about color and density, taking into account the size of the pixels in the training images. In Mip-NeRF, the neural network is trained using training images of multiple resolutions. Specifically, in Mip-NeRF, a common resolution range is set for all shooting viewpoints, and the training is performed using training images of multiple resolutions prepared so that they fall within that resolution range. With a trained neural network obtained as a result of such training, high-quality virtual viewpoint images can be generated within that resolution range.

[0005] More specifically, in the learning of the neural network in Mip-NeRF, any position in space is sampled from the ray corresponding to each pixel of the captured image, and a cone is set such that the ray passes through the center, adjusted according to the pixel size. Information regarding color and density is output for each frustum-shaped region (hereinafter referred to as the "frustum region") obtained by dividing the above-described cone so that each sampled position (hereinafter referred to as the "sampling point") is included one by one. Specifically, the neural network outputs information regarding color and density at the sampling point based on the frequency components of the pixel values corresponding to the set cone. Then, the parameters of the neural network are updated by the error backpropagation method using, as the loss, the difference between the pixel value calculated by accumulating the colors of the sampling points on the ray according to the density and the pixel value of the captured image.

Prior Art Documents

Non-Patent Documents

[0006]

Non-Patent Document 1

Non-Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0007] When multiple imaging devices used to photograph an object have different focal lengths, or when the object is moving and the distance between the imaging device and the object fluctuates, the resolution of the object's image in the captured image varies at each shooting viewpoint. When applying the technology disclosed in Non-Patent Document 2 in such cases, it is difficult to set an appropriate resolution range common to all shooting viewpoints, resulting in the problem that the range of resolutions from which high-quality virtual viewpoint images can be obtained becomes narrower. [Means for solving the problem]

[0008] The image processing apparatus according to this disclosure includes: an image acquisition means for acquiring data of multiple captured images obtained by shooting from multiple viewpoints and camera parameters corresponding to each of the multiple viewpoints; a resolution acquisition means for acquiring the shooting resolution, which is the resolution of each of the multiple captured images; a generation means for generating the multiple training images corresponding to each captured image by converting the resolution of each of the multiple captured images based on the resolution of each of the multiple training images corresponding to each captured image, and for generating camera parameters corresponding to each of the multiple training images by converting the camera parameters corresponding to each of the multiple viewpoints based on the resolution of each of the multiple training images; and a learning means for learning a learning model relating to the three-dimensional field of a target space based on the multiple training images and the camera parameters corresponding to each of the multiple training images. [Effects of the Invention]

[0009] According to this disclosure, it is possible to estimate a three-dimensional field capable of generating virtual viewpoint images that do not cause any unnaturalness in image quality when switching virtual viewpoints, regardless of variations in the resolution of object images in captured images. [Brief explanation of the drawing]

[0010] [Figure 1] This figure shows an example of the configuration of the imaging system according to Embodiment 1. [Figure 2] This is a block diagram showing an example of the hardware configuration of an image processing device according to Embodiment 1. [Figure 3] This is a block diagram showing an example of the functional configuration of an image processing apparatus according to Embodiment 1. [Figure 4] This flowchart shows an example of the processing flow in the image processing device according to Embodiment 1. [Figure 5] This flowchart shows an example of the resolution setting process in the setting unit according to Embodiment 1. [Figure 6] This figure shows an example of a screen for setting the drawing resolution range according to Embodiment 1. [Figure 7] This flowchart shows an example of the process flow for acquiring the resolution of a captured image in the setting unit according to Embodiment 1. [Figure 8] This figure shows an example of a captured image, background image, and object region map according to Embodiment 1. [Figure 9] This flowchart shows an example of the process for setting the resolution of the learning image in the setting unit according to Embodiment 1. [Figure 10] This flowchart shows an example of the flow of the learning image generation process in the first generation unit according to Embodiment 1. [Figure 11] This flowchart shows an example of the learning process flow in the learning unit according to Embodiment 1. [Figure 12] This figure shows an example of a GUI according to Embodiment 1. [Figure 13] This flowchart shows an example of the process flow for acquiring the resolution of a captured image in the setting unit according to a modified example 1 of Embodiment 1. [Figure 14] This figure illustrates an example of a method for calculating the amount of movement according to a modified example 1 of Embodiment 1. [Figure 15] This flowchart shows an example of the process flow for setting the resolution of the learning image in the setting unit according to Embodiment 2.

Best Mode for Carrying Out the Invention

[0011] Hereinafter, embodiments of the technology of the present disclosure will be described with reference to the drawings. Note that the following embodiments do not limit the technology of the present disclosure, and not all combinations of the features described in this embodiment are essential for the solution means of the present disclosure. The same components will be denoted by the same reference numerals and described. Also, each step (step) in the flowchart described later will be described using a symbol starting with "S".

[0012] [Embodiment 1] <Configuration of the Imaging System> FIG. 1 is a diagram showing an example of the configuration of an imaging system according to Embodiment 1. The imaging system includes a plurality of imaging devices 101, an image processing device 102, a user interface (hereinafter referred to as "UI") panel 103, a storage device 104, and a display device 105. Each imaging device 101 is composed of a digital still camera, a digital video camera, or the like, and is arranged at different positions. Each imaging device 101 performs synchronized imaging of the object 107 existing in the imaging space 106 from different viewpoints (imaging viewpoints) according to the imaging conditions, and generates data of an imaging image corresponding to each imaging viewpoint (hereinafter referred to as "imaging image data").

[0013] Note that synchronized imaging with each other does not mean simultaneous, but means imaging after performing synchronization processing. That is, synchronized imaging with each other does not need to be performed exactly at the same time, and includes cases where it is performed at approximately the same time. The imaging image data obtained by imaging the imaging device 101 may be data of a still image, data of a moving image, or data of both a still image and a moving image. Hereinafter, the term "image" will be described as including both "still image" and "moving image" unless otherwise specified. The imaging image data generated by each imaging device 101 is transmitted to the image processing device 102.

[0014] The image processing device 102 acquires data from multiple captured images (multi-view images) transmitted from multiple imaging devices 101, and uses the acquired multi-view images to learn a three-dimensional field relating to the space containing the object 107 present in the imaging space 106. The information or signal indicating the learned three-dimensional field obtained as a result of the learning by the image processing device 102 is output to the storage device 104, UI panel 103, or display device 105, etc. The image processing device 102 may also generate a virtual viewpoint image based on the information indicating the learned three-dimensional field obtained as a result of the learning. In this case, for example, the data or signal of the virtual viewpoint image generated by the image processing device 102 is output to the storage device 104, UI panel 103, or display device 105, etc.

[0015] In this embodiment, as shown in Figure 1, it is assumed that each of the multiple imaging devices 101 is connected to the image processing device 102. However, the method of connecting the imaging devices 101 and the image processing device 102 is not limited to this. Specifically, for example, multiple imaging devices 101 may be cascaded by connecting adjacent imaging devices 101 to each other, and at least one of the multiple imaging devices 101 may be connected to the image processing device 102.

[0016] Furthermore, in this embodiment, as shown in Figure 1 as an example, multiple imaging devices 101 are described as being arranged at different positions from one another, but the number and arrangement of imaging devices 101 are not limited to this. For example, if the position, shape, and color of the object 107 present in the imaging space 106, as well as the intensity or hue of the ambient light, etc., do not change over time, at least one imaging device 101 whose position and orientation can be changed may be arranged. In this case, the imaging device 101 may be made to take images at each of several different positions while changing its position and orientation, and the image processing device 102 may acquire the multiple image data obtained from the imaging as data for a multi-view image.

[0017] The UI panel 103 includes a display device such as a liquid crystal display and displays a GUI (Graphical User Interface) on the display device to present information to the user, such as the shooting conditions in each shooting device 101 and the processing settings of the image processing device 102. The UI panel 103 may also include an input device such as a touch panel or buttons. In this case, the UI panel 103 accepts user operations (hereinafter referred to as "user operations") related to changes in the aforementioned shooting conditions or processing settings via the input device. The input device may be provided separately from the UI panel 103, such as a mouse or keyboard.

[0018] The storage device 104 is composed of a hard disk drive or the like, and acquires information about the three-dimensional field related to the object 107 obtained from the image processing device 102, and stores the acquired information. In addition, when the image processing device 102 generates a virtual viewpoint image, the storage device 104 may acquire the data of the virtual viewpoint image output from the image processing device 102 and store the acquired data.

[0019] The display device 105 is composed of a liquid crystal display or the like, and acquires a signal of a display image including an image showing a three-dimensional field related to an object output from the image processing device 102, and displays the display image. In addition, if the image processing device 102 generates a virtual viewpoint image, the display device 105 may acquire a signal of a display image including the virtual viewpoint image output from the image processing device 102, and display the display image corresponding to that signal.

[0020] The shooting space 106 is a three-dimensional space surrounded by multiple shooting devices 101 installed in a studio or the like. In Figure 1, the solid line frame indicates the outline of the shooting space 106 on the floor. Below, as an example, a method of photographing one or more objects 107 from around them using 12 shooting devices 101 installed in a studio will be described.

[0021] Figure 2 is a block diagram showing an example of the hardware configuration of the image processing device 102 according to Embodiment 1. The image processing device 102 has a hardware configuration including a CPU 201, RAM 202, ROM 203, storage device 204, control interface (hereinafter referred to as "I / F") 205, input I / F 206, output I / F 207, and main bus 208.

[0022] The CPU 201 is a processor that comprehensively controls each part of the image processing device 102. The RAM 202 functions as the main memory and work area of ​​the CPU 201. The ROM 203 stores the programs executed by the CPU 201. The storage device 204 is composed of a hard disk drive or the like and stores application programs executed by the CPU 201, as well as various data used in the processing of the CPU 201. The CPU 201 uses the RAM 202 as work memory and executes the OS (operating system) and various programs stored in the ROM 203 or storage device 204, etc. The CPU 201 controls the entire image processing device 102 via the main bus 208 by executing various programs. The processing of each step shown in the flowchart described later is realized by the program code stored in the ROM 203 or storage device 204, etc. being loaded into the RAM 202, and then executed by the CPU 201.

[0023] The control I / F 205 is connected to each of the multiple imaging devices 101 and is a communication interface for controlling each imaging device 101, such as setting imaging conditions, starting imaging, and stopping imaging. The input I / F 206 is a communication interface for communication via serial buses such as SDI (Serial Digital Interface) or HDMI (High-Definition Multimedia Interface). Image data is acquired from each imaging device 101 via the input I / F 206. The output I / F 207 is a communication interface for communication via serial buses such as USB (Universal Serial Bus) or DisplayPort (registered trademark). Data or signals of the three-dimensional field and virtual viewpoint images are output to the storage device 104 or display device 105 via the output I / F 207. The main bus 208 is a transmission path that connects the above-mentioned hardware configurations of the image processing device 102 so that they can communicate with each other.

[0024] <Image Processing Device Functional Configuration> Figure 3 is a block diagram showing an example of the functional configuration of an image processing device 102 according to Embodiment 1. The image processing device 102 has an image acquisition unit 301, a setting unit 302, a first generation unit 303, a learning unit 304, a viewpoint acquisition unit 305, a second generation unit 306, and an output unit 307 as its functional configuration. Each unit of the image processing device 102 is realized by the CPU 201 executing a program stored in ROM 203 or the like, using RAM 202 as work memory. Note that not all of the processes shown below necessarily need to be realized by the execution of a program by the CPU 201, and the image processing device 102 may be configured so that some or all of the processes are executed by one or more processing circuits other than the CPU 201.

[0025] The image acquisition unit 301 acquires captured image data obtained by each shooting device 101. The source of the captured image data is not limited to the shooting device 101; the image acquisition unit 301 may also acquire captured image data by reading it from the storage device 204 or the storage device 104, etc. The image acquisition unit 301 also acquires background image data corresponding to the captured image of each shooting device 101. The background image is an image taken using the camera parameters used to take the corresponding captured image, and is, for example, an image obtained by the following type of shooting. For example, the background image is an image obtained by taking a picture in advance of objects other than object 107, such as the ground or structures, that are expected to be captured when object 107 is photographed, in a state where object 107 does not exist. For example, the background image data is stored in advance in the storage device 204 or the storage device 104, etc., and the image acquisition unit 301 acquires the background image data by reading it from the storage device 204, etc.

[0026] Furthermore, the image acquisition unit 301 acquires the camera parameters of each imaging device 101. The camera parameters of each imaging device 101 are described as being pre-stored in the memory device 204, but they may also be estimated by the image processing device 102 using the captured image data. In this case, for example, the image processing device 102 estimates the camera parameters of each imaging device 101 using a method called COLMAP, which is well known in the field of NeRF and other technologies, and estimates the shape of the object 107 while estimating the shooting position based on the captured image.

[0027] Camera parameters include intrinsic parameters, extrinsic parameters, and distortion parameters. Intrinsic parameters represent the coordinates of the center of the image obtained by the imaging device and the focal length of the lens of the imaging device. Extrinsic parameters represent the position and orientation of the imaging device, and distortion parameters represent the distortion of the lens. The camera parameters of multiple imaging devices 101, especially the intrinsic parameters and distortion parameters, do not have to be common to each other. For example, the field of view of some imaging devices 101 may differ from the field of view of other imaging devices 101.

[0028] The setting unit 302 sets multiple resolutions for images to be used as training images for each shooting viewpoint. The first generation unit 303 generates multiple training images corresponding to each captured image by converting the resolution of the captured image to the resolution set by the setting unit 302. Furthermore, for each of the generated training images, the first generation unit 303 generates internal parameters corresponding to each training image by converting the internal parameters corresponding to the captured image to become the internal parameters corresponding to the training image.

[0029] The learning unit 304 performs a learning process using captured image data acquired by the image acquisition unit 301, with the data of the learning images generated by the first generation unit 303 as the ground truth data, thereby estimating a three-dimensional field relating to the object 107. In this embodiment, the learning unit 304 is described as estimating a three-dimensional field relating to the object 107, which is represented by the color and density in the learning space set within the shooting space 106. The radiance field can be expressed, for example, using the following formula (1).

[0030]

number

[0031] The viewpoint acquisition unit 305 acquires virtual viewpoint information that includes at least information indicating the position of the virtual viewpoint and information indicating the direction of the line of sight at the virtual viewpoint (hereinafter referred to as "direction of the virtual viewpoint"). For example, information indicating the position and direction of the virtual viewpoint is provided by the user by inputting it using a GUI (not shown) displayed on the UI panel 103. The second generation unit 306 generates a virtual viewpoint image by volume rendering using the trained three-dimensional field model obtained as a result of training by the learning unit 304 and the virtual viewpoint information acquired by the viewpoint acquisition unit 305.

[0032] The output unit 307 outputs a trained three-dimensional field model obtained as a result of training by the learning unit 304. Specifically, for example, the output unit 307 outputs the data of the trained three-dimensional field model to the storage device 104, causing the storage device 104 to store the data of the trained three-dimensional field model. The output unit 307 may also convert the three-dimensional field represented by the trained three-dimensional field model into a display image signal and output it to the display device 105, causing the three-dimensional field to be displayed as a display image on the display device 105. In addition, the output unit 307 outputs a virtual viewpoint image generated by the second generation unit 306. For example, the output unit 307 outputs the data of the virtual viewpoint image to the storage device 104, causing the storage device 104 to store the data of the virtual viewpoint image. The output unit 307 may also convert the virtual viewpoint image into a display image signal and output it to the display device 105, causing the display device 105 to display the virtual viewpoint image.

[0033] <Operation of the image processing device> Figure 4 is a flowchart showing an example of the processing flow in the image processing device 102 according to Embodiment 1. Referring to Figure 4, the operation of the image processing device 102 during the learning phase of the three-dimensional field model will be explained. First, in S401, the image acquisition unit 301 acquires the camera parameters of each shooting device 101, the data of the captured images (multi-view images) obtained by shooting with each shooting device 101, and the data of the background image corresponding to each captured image.

[0034] Next, in S402, the setting unit 302 sets multiple resolutions for the training images to be generated for each captured image, based on the range of resolutions for the virtual viewpoint images that can be generated (hereinafter referred to as the "drawing resolution range") and the resolution of each captured image, as specified by user operation, etc. Details of the resolution setting process in S402 will be described later. Next, in S403, the first generation unit 303 generates multiple training images for each captured image by converting the resolution of each captured image to the resolution of the training image set in S402. Details of the training image generation process in S403, that is, the resolution conversion process for the captured image, will be described later. Next, in S404, the learning unit 304 performs a training process for the three-dimensional field model using the captured image data acquired in S401 and the training image data generated in S403 as ground truth data. Details of the three-dimensional field model training process in S404 will be described later. Next, in S405, the output unit 307 outputs the trained three-dimensional field model obtained as a result of the training process in S404. After S405, the image processing device 102 terminates the processing of the flowchart related to the learning phase shown in Figure 4.

[0035] <Resolution setting process> Referring to Figure 5, the resolution setting process in S402, executed by the setting unit 302, will be described. Figure 5 is a flowchart showing an example of the resolution setting process in the setting unit 302 according to Embodiment 1, and is a flowchart showing a detailed example of the processing flow of the resolution setting process in S402. The processing in the flowchart shown in Figure 5 is executed after the processing in S401. After S401, first, in S501, the setting unit 302 obtains information regarding the range of resolution (drawing resolution range) of the virtual viewpoint image that can be generated in the generation phase described later, which is specified by user operation via the UI panel 103.

[0036] Referring to Figure 6, the method for specifying the drawing resolution range by user operation will be explained. Figure 6 is a diagram showing an example of a drawing resolution range setting screen 600 displayed on the display device of the UI panel 103 according to Embodiment 1. The setting screen 600 includes a scaling ratio setting area 601 and an object preview area 602. In the object preview area 602, any captured image from among the multiple captured images constituting the multi-view image acquired in S401 is displayed at a predetermined magnification such as equal magnification. In the scaling ratio setting area 601, the user selects whether to set the maximum or minimum number of pixels as the setting mode by pressing the "maximum" button 611 or the "minimum" button 612 using an operating device such as a mouse 603. Subsequently, the user specifies the scaling ratio by adjusting the maximum or minimum value of the number of pixels by moving the knob 614 of the scaling ratio slider bar 613 using an operating device such as a mouse 603.

[0037] Here, the maximum number of pixels refers to the number of pixels in the region of the image containing the image of object 107 (hereinafter referred to as the "object region") corresponding to the scaling ratio when the virtual viewpoint is closest to object 107. The minimum value refers to the number of pixels in the object region of the image corresponding to the scaling ratio when the virtual viewpoint is furthest from the object. As the user moves the handle 614, the object preview region 602 displays the captured image scaled according to the scaling ratio corresponding to the position of the handle 614. By checking the scaled captured image displayed in the object preview region 602, the user can get a sense of the resolution of the image of object 107 in the scaled captured image. After adjusting the scaling ratio to the desired value, the user presses the "OK" button 615. When the user presses the "OK" button, the setting unit 302 calculates the number of pixels in the object region of the image corresponding to the scaling ratio determined by the user. The setting unit 302 acquires the calculated number of pixels as the lower or upper limit of the drawing resolution range according to the selected setting mode, and stores information regarding that resolution.

[0038] Following S501, in S502, the setting unit 302 obtains the resolution of each captured image that makes up the multi-view image acquired in S401. Details of the process for obtaining the resolution of the captured images in S502 will be described later. Next, in S503, based on the drawing resolution range acquired in S501 and the resolution of the captured images acquired in S502, the setting unit 302 sets multiple parameters related to the resolution of the training image to be generated by the first generation unit 303 in S403 for each captured image. Details of the process for setting the resolution of the training image in S503 will be described later. After S503, the setting unit 302 completes the process shown in the flowchart in Figure 5, that is, the process shown in S401 in Figure 4.

[0039] <Processing for obtaining the resolution of captured images> Referring to Figures 7 and 8, the process of acquiring the resolution of the captured image in S502, which is executed by the setting unit 302, will be described. Figure 7 is a flowchart showing an example of the flow of the process of acquiring the resolution of the captured image in the setting unit 302 according to Embodiment 1, and is a flowchart showing an example of a detailed processing flow of the process of acquiring the resolution of the captured image in S502. In this embodiment, the setting unit 302 is described as acquiring the number of pixels included in the region (object region) containing the image of object 107 in the captured image as the resolution of the captured image. The process of the flowchart shown in Figure 7 is executed after the process of S501.

[0040] Following S501, in S701, the setting unit 302 first calculates the difference between each captured image and the background image corresponding to that captured image for each captured image that makes up the multi-view image acquired in S401, and generates an object region map based on the calculated difference. Here, an object region map is an image that shows the object region in the captured image. Note that this method of generating an object region map is generally called the background subtraction method, and since the background subtraction method is a well-known technique, its explanation will be omitted. In this embodiment, the method of generating the object region map is described as being generated by the background subtraction method, but the method of generating the object region map is not limited to this. For example, the setting unit 302 may generate an object region map using a trained model that has been trained to output an object region map corresponding to an image when an image is input.

[0041] Figure 8 shows an example of a captured image 801, a background image 802, and an object region map 803 according to Embodiment 1. In this embodiment, the object region map 803 is described as a binary image. Specifically, in this embodiment, as an example, the object region map 803 shown in Figure 8(c) is assumed to represent the pixel values ​​(pixel values) of pixels included in the region corresponding to the object region as "255". Furthermore, the pixel values ​​(pixel values) of pixels included in regions other than the object region (hereinafter referred to as "non-object region") are assumed to be represented as "0".

[0042] Following S701, in S703, the setting unit 302 selects an arbitrary captured image from among the multiple captured images that make up the multi-view image acquired in S401. Next, in S704, the setting unit 302 sets the initial value of the number of pixels included in the object region of the captured image selected in S703 to "0". Next, in S705, the setting unit 302 selects an arbitrary pixel in the object region map acquired in S701 that corresponds to the selected captured image. Next, in S706, the setting unit 302 determines whether the value of the pixel selected in S705 (pixel value) is "255". If it is determined in S706 that the pixel value is not "255", the setting unit 302 executes the process in S708. The process in S708 will be described later.

[0043] If the pixel value is determined to be "255" in S706, the setting unit 302 increments the value of the number of pixels included in the object area in S707. If there are multiple objects in the target space and the captured image contains images of multiple objects, the setting unit 302 performs the following process, for example. In this case, the setting unit 302 clusters pixels that are adjacent to each other and classifies them into clusters, then increments the value of the number of pixels for each cluster. After S707, the setting unit 302 performs the process in S708.

[0044] In S708, the setting unit 302 determines whether all pixels in the object region map have been selected in S705. If it is determined in S708 that at least some pixels have not been selected, the setting unit 302 returns to the process in S705 and repeatedly executes the processes from S705 to S707 until it is determined in S708 that all pixels have been selected. During this repeated process, the setting unit 302 selects the pixels that have not yet been selected in S705. If it is determined in S708 that all pixels have been selected, in S709, the setting unit 302 obtains the value of the number of pixels included in the object region, which has been counted by the repeated process described above, as the resolution of the captured image selected in S703.

[0045] Next, in S710, the setting unit 302 determines whether all captured images have been selected in S703. If it is determined in S710 that at least some of the captured images have not been selected, the setting unit 302 returns to the process in S703 and repeatedly executes the process from S703 to S710 until it is determined in S710 that all captured images have been selected. During this repeated process, the setting unit 302 selects the captured images that have not yet been selected in S703. If it is determined in S710 that all captured images have been selected, the setting unit 302 terminates the process shown in the flowchart in Figure 7, that is, the process in S502 shown in Figure 5.

[0046] <Setting the resolution of training images> Referring to Figure 9, the process of setting the resolution of the training image in S503, which is executed by the setting unit 302, will be described. Figure 9 is a flowchart showing an example of the flow of the process of setting the resolution of the training image in the setting unit 302 according to Embodiment 1, and is a flowchart showing a detailed example of the processing flow of the process of setting the resolution of the training image in S503. The process shown in the flowchart in Figure 9 is executed after the process of S502.

[0047] Following S502, in S902, the setting unit 302 first selects an arbitrary captured image from among the multiple captured images that make up the multi-view image acquired in S401. Next, in S904, the setting unit 302 sets "0" as the initial value for the number of resolution conversion processes for the captured image selected in S902 (hereinafter referred to as the "selected captured image"). Next, in S905, the setting unit 302 determines whether the resolution after the conversion process, if the resolution conversion process is performed on the selected captured image acquired in S502, is within the drawing resolution range set in S501. If it is determined in S905 that it is not within the drawing resolution range, the setting unit 302 executes the process in S908. The process in S908 will be described later.

[0048] If it is determined in S905 that the image is within the drawing resolution range, in S906 the setting unit 302 sets parameters related to the resolution conversion process for the selected captured image (hereinafter referred to as "resolution conversion parameters"). In the resolution conversion process according to this embodiment, downsampling is performed on each captured image and object area map so that the resolution of the captured image after the conversion process falls within the drawing resolution range. The setting unit 302 sets the downsampling ratio value as the resolution conversion parameter according to the number of times the resolution conversion process is performed. The downsampling ratio r can be calculated, for example, using the following formula (2) when the number of times the resolution conversion process is performed is n.

[0049]

number

[0050] If it is determined in S908 that the resolution conversion parameter is not unset, then in S909 the setting unit 302 determines whether all captured images constituting the multi-view image have been selected. If it is determined in S909 that at least some of the captured images constituting the multi-view image have not been selected, the setting unit 302 returns to the process in S902. Subsequently, the setting unit 302 repeatedly executes the processes from S902 to S909 until it is determined in S909 that all captured images have been selected. During this repeated process, the setting unit 302 selects the captured images that have not yet been selected in S902. If it is determined in S909 that all captured images have been selected, the setting unit 302 terminates the process shown in the flowchart in Figure 9, that is, the process shown in S503 in Figure 5.

[0051] In this embodiment, the training image generation process in S504 is described as being performed only as downsampling of the captured image, but the training image generation process is not limited to this. For example, the training image generation process may be performed only as upsampling, or a combination of downsampling and upsampling may be performed. In this case, the upsampling ratio r can be calculated, for example, using the following formula (3), where n is the number of resolution conversion processes.

[0052]

number

[0053] <Processing for generating training images> Referring to Figure 10, the training image generation process in S403, which is performed by the first generation unit 303, that is, the resolution conversion process for the captured image, will be described. Figure 10 is a flowchart showing an example of the flow of the training image generation process in the first generation unit 303 according to Embodiment 1. Specifically, Figure 10 is a flowchart showing a detailed example of the processing flow of the training image generation process in S403, that is, the resolution conversion process for the captured image. The processing shown in the flowchart in Figure 10 is performed after the processing in S402.

[0054] Following S402, in S1001, the first generation unit 303 selects an arbitrary captured image from among multiple captured images that constitute a multi-view image. Next, in S1003, the first generation unit 303 converts the resolution of the selected captured image (selected captured image) using each resolution conversion parameter, based on the multiple resolution conversion parameters set in S906 in S402. Also in S1003, the first generation unit 303 converts the resolution of the object region map corresponding to the selected captured image using each resolution conversion parameter. Furthermore, in S1003, the first generation unit 303 obtains camera parameters corresponding to the image after the conversion process of the selected captured image by converting the values ​​of the camera parameters of the imaging device 101 that captured the selected captured image using each resolution conversion parameter. Specifically, the first generation unit 303 obtains internal parameters corresponding to the image after the conversion process of the selected captured image by converting the values ​​of the internal parameters of the imaging device 101 that captured the selected captured image using each resolution conversion parameter.

[0055] In this embodiment, the well-known bicubic method is used as the pixel interpolation method in the downsampling process, but the pixel interpolation method is not limited to the bicubic method. Furthermore, in this embodiment, the formula (2) described above is used as the method for calculating the downsampling ratio r. Internal parameter cp of the imaging device 101. in and the internal parameter cp after the conversion process inThe terms can be expressed, for example, using the following equations (4) and (5). Here, (cx,cy) are coordinates indicating the central position in the captured image, and (fx,fy) is the focal length of the imaging device 101 that captured the image.

[0056]

number

[0057] <Learning process for three-dimensional field models> Referring to Figure 11, the learning process in S404 executed by the learning unit 304 will be described. Figure 11 is a flowchart showing an example of the flow of the learning process in the learning unit 304 according to Embodiment 1, and is a flowchart showing a detailed example of the processing flow of the learning process in S404. The processing shown in the flowchart in Figure 11 is executed after the processing in S403.

[0058] Following S403, in S1101, the learning unit 304 first sets the initial values ​​of the parameters and hyperparameters of the three-dimensional field model. Next, in S1102, the learning unit 304 selects an arbitrary learning image from among the multiple learning images generated for each captured image in S403.

[0059] In S1103, the learning unit 304 acquires information about the light rays corresponding to each pixel of the learning image (hereinafter referred to as "light ray information") based on the learning image selected in S1102 and the camera parameters corresponding to the learning image acquired in S1003. In this embodiment, as an example, the learning unit 304 acquires information indicating the position of the imaging device 101 that captured the image corresponding to the learning image which is the starting point of the light ray, information indicating the direction of the light ray, and the pixel value of the pixel corresponding to the light ray as the light ray information. The direction of the light ray can be calculated based on the camera parameters corresponding to the learning image and the coordinates of the pixels in the learning image, for example, using the following formula (6).

[0060]

number

[0061] Following S1103, in S1104, the learning unit 304 calculates the pixel values ​​(hereinafter referred to as "drawing values") when volume rendering is performed using the ray information and three-dimensional field model acquired in S1103. Specifically, first, the learning unit 304 samples a predetermined number of learning rays based on the ray information. Next, the learning unit 304 calculates the pixel values ​​(hereinafter referred to as "drawing values") when volume rendering is performed on the three-dimensional field model based on the sampled rays. Specifically, in volume rendering, the learning unit 304 first sets a cone through which the rays pass through the center, and acquires information on the color and density of each frustum region obtained by dividing the cone based on multiple sampling points set in the learning space on each ray. Next, the learning unit 304 calculates the drawing values ​​corresponding to the rays using, for example, the following formulas (7) and (8).

[0062]

number

[0063]

number

[0064] After S1104, in S1105, the learning unit 304 calculates the difference between the calculated drawing value and the pixel value corresponding to the light ray (pixel value), and updates the three-dimensional field model to minimize the difference. For example, the learning unit 304 calculates this difference using the squared Euclidean distance and a loss function, and updates the parameters of the three-dimensional field model using backpropagation. Next, in S1106, the learning unit 304 determines whether all training images were selected in S1102. If it is determined in S1106 that all training images have been selected, the learning unit 304 executes the process in S1107. If it is determined in S1106 that at least some of the training images have not been selected, the learning unit 304 returns to the process in S1102 and repeatedly executes the processes from S1102 to S1106 until it is determined in S1106 that all training images have been selected. During this repeated process, the learning unit 304 selects the training images that were not yet selected in S1102.

[0065] In S1107, the learning unit 304 determines whether the learning has converged. For example, the learning unit 304 determines whether the learning has converged by checking whether the value of the loss calculated by the loss function is less than or equal to a given threshold. If it is determined in S1107 that the learning has not converged, the learning unit 304 returns to the process in S1102 and repeatedly executes the processes from S1102 to S1107 until it is determined in S1107 that the learning has converged. If it is determined in S1107 that the learning has converged, the learning unit 304 terminates the process in the flowchart shown in Figure 11, that is, the process in S404 shown in Figure 4. After S404, in S405, the output unit 307 outputs the three-dimensional field model in which the learning has converged as the learned three-dimensional field model. For example, the output unit 307 outputs the parameters of the learned three-dimensional field model to the storage device 204 and stores the parameters in the storage device 204.

[0066] As described above, the image processing device 102 was configured to set the resolution of the training images used for training the three-dimensional field model based on a specified drawing resolution range. With the image processing device 102 configured in this way, it is possible to estimate a three-dimensional field capable of generating virtual viewpoint images that do not cause any unnaturalness in image quality when switching virtual viewpoints, regardless of the variation in the resolution of the image of object 107 in the captured image.

[0067] As described above, the processing flow in the learning phase of the image processing device 102 according to Embodiment 1 was explained with reference to Figure 4. After S405 shown in Figure 4, the image processing device 102 may perform the process of generating and outputting a virtual viewpoint image using the outputted trained three-dimensional field model in the inference phase.

[0068] Figure 12 shows an example of a GUI 1200 displayed on a display device 105 or UI panel 103 according to Embodiment 1. Specifically, Figure 12 shows an example of how a virtual viewpoint image, including an image of object 107, is drawn on the GUI 1200 based on virtual viewpoint information input by user operation. The GUI 1200 includes a range display area 1201 and an image display area 1202.

[0069] The range display area 1201 shows, for example, that 60% of the settable range of 10% to 90% for the scaling ratio of the virtual viewpoint image has been selected, based on the image captured by the camera device 101 with camera number "1". In this case, the image display area 1202 displays the virtual viewpoint image corresponding to the virtual viewpoint when the virtual viewpoint is moved along the optical axis of the camera device 101 so that the scaling ratio of the image captured by the camera device 101 is 60%. The joystick 1203 is used to set the scaling ratio of the virtual image and to operate the virtual viewpoint. The image display area 1202 displays the virtual viewpoint image corresponding to the virtual viewpoint moved based on the user operation by the joystick 1203.

[0070] [Modification 1 of Embodiment 1] In Embodiment 1, with reference to Figure 7, a configuration in which the setting unit 302 calculates the number of pixels included in the object area and obtains the calculated number of pixels as the resolution of the captured image was described. In Modification 1 of Embodiment 1, a configuration in which the setting unit 302 obtains the length of each pixel of the object 107 in the captured image as the resolution of the captured image will be described.

[0071] Referring to Figures 13 and 14, the process of setting the resolution of the training image, which is performed by the setting unit 302 (hereinafter simply referred to as "setting unit 302") according to Modification 1 of Embodiment 1, will be described. Figure 13 is a flowchart showing an example of the flow of the process of acquiring the resolution of the captured image in the setting unit 302 according to Modification 1 of Embodiment 1, and is a flowchart showing a detailed example of the processing flow of the process of acquiring the resolution of the captured image in S502 shown in Figure 5. In the following description of Figure 13, the same reference numerals are used for the same processes as in the processing steps shown in Figure 7, and their explanation is omitted. The process in the flowchart shown in Figure 13 is performed after the process in S501.

[0072] After S501, the setting unit 302 first executes the process in S702. Next, in S1301, the setting unit 302 acquires three-dimensional shape data by estimating the three-dimensional shape of the object based on the object region map acquired in S702. For example, the setting unit 302 estimates the three-dimensional shape of the object using the visual volume cross-section method with the object region map. The three-dimensional shape data is point cloud data, or mesh data in which the surface shape of the object is represented by multiple polygons. The representation of the three-dimensional shape data is not limited to these; for example, the three-dimensional shape data may be voxel data in which voxels corresponding to the object are represented by ON voxels and voxels that do not correspond are represented by OFF voxels. Hereinafter, the three-dimensional shape data acquired by the setting unit 302 will be described as being represented by mesh data.

[0073] After S702, in S1302, the setting unit 302 generates a depth map corresponding to each imaging device 101 by projecting the three-dimensional shape estimated in S1301 onto the viewpoint of each imaging device 101. Next, the setting unit 302 executes the process in S703. After S703, in S1303, the setting unit 302 obtains the minimum depth value (hereinafter referred to as the "minimum depth value") in the depth map generated in S1302 that corresponds to the imaging device 101 that captured the image selected in S703 (selected image).

[0074] Next, in S1304, the setting unit 302 calculates the length of the object 107 per pixel of the captured image selected in S703, based on the minimum depth value acquired in S1303. Specifically, for example, the setting unit 302 calculates the amount of movement r of the object 107 on the depth plane when it is shifted by one pixel on the image plane. Here, the image plane is a plane that is perpendicular to the optical axis of the imaging device 101 and is located at a distance equal to the focal length from the position of the imaging device 101. The depth plane is a plane that is perpendicular to the optical axis of the imaging device 101 and is located at a distance equal to the minimum depth value from the position of the imaging device 101.

[0075] Figure 14 is a diagram illustrating an example of a method for calculating the amount of movement according to a modification 1 of Embodiment 1. Specifically, Figure 14 shows the amount of movement 1405 of object 107 on the depth plane 1404 when it moves by only one pixel on the image plane 1402. The amount of movement Δx in the x direction and the amount of movement Δy in the y direction can be calculated, for example, using the following formula (11).

[0076]

number

[0077]

number

[0078] With the image processing device 102 configured as described above, it is possible to estimate a three-dimensional field capable of generating virtual viewpoint images that do not cause any unnaturalness in image quality when switching virtual viewpoints, regardless of variations in the resolution of the image of object 107 in the captured image.

[0079] [Embodiment 2] In Embodiment 1, with reference to Figure 7, a configuration in which the setting unit 302 sets the resolution of the training image based on a drawing resolution range specified by user operation was described. In Embodiment 2, a configuration in which the setting unit 302 sets the resolution of the training image based on the lower limit of the resolution of multiple captured images will be described.

[0080] Referring to Figure 15, the process of setting the resolution of the training image, which is performed by the setting unit 302 according to Embodiment 2 (hereinafter simply referred to as "setting unit 302"), will be described. Figure 15 is a flowchart showing an example of the flow of the process of setting the resolution of the training image in the setting unit 302 according to Embodiment 2, and is a flowchart showing a detailed example of the processing flow of the process of setting the resolution of the training image in S402 shown in Figure 4. In the following description of Figure 15, the same reference numerals are used for the same processes as in Figures 5 and 9, and their explanation is omitted. The process in the flowchart shown in Figure 15 is performed after the process in S401 shown in Figure 4.

[0081] After S401, the setting unit 302 first executes the process in S502. Next, in S1502, the setting unit 302 obtains the minimum resolution (lower limit of resolution) among the resolutions of the multiple captured images acquired in S502. Next, the setting unit 302 executes the process in S902. Next, in S1504, the setting unit 302 sets the lower limit of resolution acquired in S1502 as the initial value of the resolution to be downsampled for the selected captured image (selected captured image) in S902. Next, the setting unit 302 executes the process in S904.

[0082] Next, in S1506, the setting unit 302 determines whether the number of resolution conversion processes is less than or equal to a predetermined value. If it is determined in S1506 that it is less than or equal to the predetermined value, the setting unit 302 sequentially executes processes S906 and S907. After that, the setting unit 302 returns to process S1506 and repeatedly executes processes S1505 to S907 until it is determined in S1506 that it is not less than or equal to the predetermined value. If it is determined in S1506 that it is not less than or equal to the predetermined value, the setting unit 302 executes process S909. If it is determined in S909 that at least some of the captured images constituting the multiview image have not been selected, the setting unit 302 returns to process S902 and repeatedly executes processes S902 to S909 until all captured images have been selected in S909. In this repeated process, the setting unit 302 selects the captured images that have not yet been selected in S902. If it is determined in S909 that all captured images have been selected, the setting unit 302 terminates the process shown in the flowchart in Figure 15, that is, the process shown in S402 in Figure 4.

[0083] As described above, the image processing device 102 is configured to set the resolution of the training images used for training the three-dimensional field model based on the resolution of the captured images. With the image processing device 102 configured in this way, it is possible to estimate a three-dimensional field capable of generating virtual viewpoint images that do not cause any unnaturalness in image quality when switching virtual viewpoints, regardless of the variation in the resolution of the image of object 107 in the captured images. Furthermore, with the image processing device 102 configured in this way, the resolution of the training images corresponding to each captured viewpoint is standardized, which can improve the image quality of the virtual viewpoint images.

[0084] [Other embodiments] The technology of this disclosure can also be implemented by supplying a program that implements one or more of the functions of the embodiments described above to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. Furthermore, the technology of this disclosure can also be implemented by a processing circuit such as an ASIC that implements one or more functions.

[0085] Furthermore, within the scope of this disclosure, the technologies described herein allow for free combination of each embodiment, modification of any component of each embodiment, or omission of any component in each embodiment.

[0086] [Technical Features of This Disclosure] This disclosure includes the following configurations, methods, and programs.

[0087] <Configuration 1> Image acquisition means for acquiring data of multiple captured images obtained from multiple viewpoints, and camera parameters corresponding to each of the multiple viewpoints, A resolution acquisition means for acquiring the shooting resolution, which is the resolution of each of the aforementioned multiple captured images, A generation means that generates a plurality of training images corresponding to each of the multiple captured images by converting the resolution of each of the plurality of captured images based on the resolution of each of the plurality of training images corresponding to each of the captured images, and generates camera parameters corresponding to each of the plurality of training images by converting the camera parameters corresponding to each of the plurality of viewpoints based on the resolution of each of the plurality of training images, A learning means for learning a learning model relating to the three-dimensional field of a target space based on the plurality of learning images and the camera parameters corresponding to each of the plurality of learning images, An image processing apparatus characterized by having

[0088] <Configuration 2> The resolution acquisition means acquires the shooting resolution of each of the multiple captured images based on the number of pixels included in the region containing the image of the object present in the target space in each of the multiple captured images. An image processing apparatus according to configuration 1, characterized by the above.

[0089] <Structure 3> The resolution acquisition means acquires the shooting resolution of each of the multiple captured images based on the length of an object present in the target space per pixel of each of the multiple captured images. An image processing apparatus according to configuration 1 or 2, characterized by the above.

[0090] <Structure 4> The generation means determines the resolution of each of the plurality of training images based on the drawing resolution range specified by user operation and the respective shooting resolutions of the plurality of captured images, such that the drawing resolution range falls within the resolution range of the plurality of training images. An image processing apparatus according to any one of configurations 1 to 3 characterized by the above.

[0091] <Composition 5> The drawing resolution range is the range of resolutions of virtual viewpoint images that can be generated using the trained model obtained as a result of the training, and which correspond to the view of the object's space from a virtual viewpoint. The image processing apparatus described in configuration 4, characterized by the above.

[0092] <Composition 6> The generation means determines the resolution of each of the plurality of training images based on at least one of the lower limit and upper limit of the resolution of the plurality of captured images. An image processing apparatus according to any one of configurations 1 to 5 characterized by the above.

[0093] <Composition 7> The generation means determines the resolution of each of the multiple training images such that the lower limit of the resolution of each of the multiple training images corresponding to each of the multiple captured images matches the lower limit of the resolution of the multiple captured images. An image processing apparatus according to configuration 6, characterized by the above.

[0094] <Structure 8> The generation means determines the resolution of each of the multiple training images such that the upper limit of the resolution of each of the multiple training images corresponding to each of the multiple captured images matches the upper limit of the resolution of the multiple captured images. An image processing apparatus according to configuration 6 or 7, characterized by the above.

[0095] <Composition 9> The generation means generates the plurality of training images for each of the plurality of captured images by performing a resolution conversion process on each of the plurality of captured images by at least one of a downsampling process and an upsampling process. An image processing apparatus according to any one of configurations 1 to 8 characterized by the above.

[0096] <Composition 10> Image generation means that generates a virtual viewpoint image corresponding to how the object's space appears from the virtual viewpoint, based on the learned model obtained as a result of the learning and information about the virtual viewpoint. Having further, An image processing apparatus according to any one of configurations 1 to 9 characterized by the above.

[0097] <Method> An image acquisition process that acquires data from multiple captured images obtained by shooting from multiple viewpoints, and camera parameters corresponding to each of the multiple viewpoints, A resolution acquisition step to acquire the shooting resolution, which is the resolution of each of the aforementioned multiple captured images, A generation step of generating a plurality of training images corresponding to each of the multiple captured images by converting the resolution of each of the plurality of captured images based on the resolution of each of the plurality of training images corresponding to each of the captured images, and a generation step of generating camera parameters corresponding to each of the plurality of training images by converting the camera parameters corresponding to each of the plurality of viewpoints based on the resolution of each of the plurality of training images, A learning process in which a learning model relating to the three-dimensional field of the target space is trained based on the plurality of training images and the camera parameters corresponding to each of the plurality of training images, An image processing method characterized by including

[0098] <Program> A program for causing a computer to function as an image processing device described in any one of configurations 1 through 10. [Explanation of Symbols]

[0099] 102 Image Processing Device 301 Image acquisition unit 302 Settings Section 303 1st generation part 304 Learning Department

Claims

1. Image acquisition means for acquiring data of multiple captured images obtained from multiple viewpoints, and camera parameters corresponding to each of the multiple viewpoints, A resolution acquisition means for acquiring the shooting resolution, which is the resolution of each of the aforementioned multiple captured images, A generation means that generates a plurality of training images corresponding to each of the multiple captured images by converting the resolution of each of the plurality of captured images based on the resolution of each of the plurality of training images corresponding to each of the captured images, and generates camera parameters corresponding to each of the plurality of training images by converting the camera parameters corresponding to each of the plurality of viewpoints based on the resolution of each of the plurality of training images, A learning means for learning a learning model relating to the three-dimensional field of a target space based on the plurality of learning images and the camera parameters corresponding to each of the plurality of learning images, An image processing apparatus characterized by having

2. The resolution acquisition means acquires the shooting resolution of each of the multiple captured images based on the number of pixels included in the region containing the image of the object present in the target space in each of the multiple captured images. The image processing apparatus according to claim 1, characterized in that

3. The resolution acquisition means acquires the shooting resolution of each of the multiple captured images based on the length of an object present in the target space per pixel of each of the multiple captured images. The image processing apparatus according to claim 1, characterized in that

4. The generation means determines the resolution of each of the plurality of training images based on the drawing resolution range specified by user operation and the respective shooting resolutions of the plurality of captured images, such that the drawing resolution range falls within the resolution range of the plurality of training images. The image processing apparatus according to claim 1, characterized in that

5. The drawing resolution range is the range of resolutions of virtual viewpoint images that can be generated using the trained model obtained as a result of the training, and which correspond to the view of the object's space from a virtual viewpoint. The image processing apparatus according to claim 4, characterized by the above.

6. The generation means determines the resolution of each of the plurality of training images based on at least one of the lower limit and upper limit of the resolution of the plurality of captured images. The image processing apparatus according to claim 1, characterized in that

7. The generation means determines the resolution of each of the multiple training images such that the lower limit of the resolution of each of the multiple training images corresponding to each of the multiple captured images matches the lower limit of the resolution of the multiple captured images. The image processing apparatus according to claim 6, characterized by the above.

8. The generation means determines the resolution of each of the multiple training images such that the upper limit of the resolution of each of the multiple training images corresponding to each of the multiple captured images matches the upper limit of the resolution of the multiple captured images. The image processing apparatus according to claim 6, characterized by the above.

9. The generation means generates the plurality of training images for each of the plurality of captured images by performing a resolution conversion process on each of the plurality of captured images by at least one of a downsampling process and an upsampling process. The image processing apparatus according to claim 1, characterized in that

10. Image generation means that generates a virtual viewpoint image corresponding to how the object's space appears from the virtual viewpoint, based on the learned model obtained as a result of the learning and information about the virtual viewpoint. Having further, The image processing apparatus according to claim 1, characterized in that

11. An image acquisition process that acquires data from multiple captured images obtained by shooting from multiple viewpoints, and camera parameters corresponding to each of the multiple viewpoints, A resolution acquisition step to acquire the shooting resolution, which is the resolution of each of the aforementioned multiple captured images, A generation step of generating a plurality of training images corresponding to each of the multiple captured images by converting the resolution of each of the plurality of captured images based on the resolution of each of the plurality of training images corresponding to each of the captured images, and a generation step of generating camera parameters corresponding to each of the plurality of training images by converting the camera parameters corresponding to each of the plurality of viewpoints based on the resolution of each of the plurality of training images, A learning process in which a learning model relating to the three-dimensional field of the target space is trained based on the plurality of training images and the camera parameters corresponding to each of the plurality of training images, An image processing method characterized by including

12. A program for causing a computer to function as an image processing device according to any one of claims 1 to 10.

Images