Image processing device, image processing method, and program

Abstract

Set the parameters of the learning model that represents spatial information appropriately. [Solution] The image processing device 100 according to this disclosure obtains multiple images by capturing a three-dimensional space including an object from multiple directions, analyzes the images to obtain image features related to the images and position information indicating the position of the object, and sets parameters for a learning model that estimates spatial information related to the learning region included in the three-dimensional space based on the position information and the image features.

Description

Technical Field

[0001] The present disclosure relates to a technique for estimating spatial information from captured images obtained by imaging from a plurality of viewpoints.

Background Art

[0002] There is a technique for estimating spatial information that can generate a virtual viewpoint image corresponding to the appearance from an arbitrary virtual viewpoint (hereinafter referred to as "virtual viewpoint") based on captured images obtained by imaging from a plurality of viewpoints and camera parameters related to the imaging. Patent Document 1 discloses the following technique for estimating spatial information. First, by giving information on the position and direction of the imaging viewpoint to a learning model, the color information of each pixel is estimated, and the estimated color information of the pixel is compared with the color information of the pixel of the captured image corresponding to the pixel. Subsequently, by feeding back the error between the color information to the spatial information for learning the learning model, spatial information adapted to the captured image is generated. In the estimation method disclosed in Patent Document 1, the larger the number of parameters (hereinafter referred to as "spatial information parameter number") representing the color or density in the space of the learning model for estimating spatial information, the more complex spatial information can be represented. Specifically, for example, when the learning model represents spatial information by a DNN (Deep Neural Network), the number of layers and nodes of the DNN corresponds to the spatial information parameter number.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, if the number of spatial information parameters is excessively large, the number of variables to optimize increases, and the amount of computation required for training becomes enormous. Conversely, if the number of spatial information parameters is too small for the object represented by the spatial information, sufficient training accuracy cannot be obtained, and the image of that object in the virtual viewpoint image may become blurred. Therefore, in order to achieve training with sufficient accuracy with less computation, it is necessary to appropriately set the number of spatial information parameters of the training model according to the object represented by the spatial information.

[0005] This disclosure is made to solve the aforementioned problems and aims to provide a technology that enables appropriate setting of parameters for a learning model that represents spatial information. [Means for solving the problem]

[0006] The image processing apparatus according to this disclosure includes: an image acquisition means for acquiring a plurality of images obtained by capturing a three-dimensional space including an object from a plurality of directions; a feature acquisition means for acquiring image features related to the image by analyzing the image; a position acquisition means for acquiring position information indicating the position of the object; and a setting means for setting parameters of a learning model that estimates spatial information related to a learning region included in the three-dimensional space based on the position information and the image features. [Effects of the Invention]

[0007] According to this disclosure, the parameters of a learning model that represents spatial information can be appropriately set. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing an example of the logical configuration of an image processing device according to the first embodiment. [Figure 2] This is a block diagram showing an example of the hardware configuration of an image processing device according to the first embodiment. [Figure 3]This flowchart shows an example of the processing flow of the image processing apparatus according to the first embodiment. [Figure 4] This figure illustrates an example of the acquisition process in the feature acquisition unit according to the first embodiment. [Figure 5] This figure illustrates an example of the acquisition process in the position acquisition unit according to the first embodiment. [Figure 6] This figure illustrates an example of the spatial resolution estimation process in the setting unit according to the first embodiment. [Figure 7] This figure illustrates an example of volume rendering processing in the learning unit according to the first embodiment. [Figure 8] This figure illustrates an example of the acquisition process in the position acquisition unit according to a modified example 1 of the first embodiment. [Figure 9] This figure illustrates an example of the acquisition process in the feature acquisition unit according to a modified example 2 of the first embodiment. [Figure 10] This flowchart shows an example of the processing flow of the image processing apparatus according to the second embodiment. [Figure 11] This figure illustrates an example of the acquisition process in the feature acquisition unit according to the second embodiment. [Figure 12] This figure illustrates an example of the acquisition process in the position acquisition unit according to the second embodiment. [Figure 13] This figure illustrates an example of the spatial resolution estimation process in the setting unit according to the second embodiment. [Figure 14] This figure illustrates an example of the setting process in the setting unit according to the second embodiment. [Figure 15] This figure illustrates an example of the setting process in the setting unit according to Modification 1 of the second embodiment. [Modes for carrying out the invention]

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples, and the technical scope of the present disclosure is not limited only to those configurations.

[0010] [First Embodiment] In the first embodiment, an aspect of setting parameters of a learning model based on image features obtained from each captured image obtained by capturing an object from a plurality of viewpoints and the position of the object will be described. Specifically, an aspect of obtaining the spatial frequency included in the captured image as an image feature and setting parameters related to the spatial information of the learning model using the obtained spatial frequency as an image feature will be described.

[0011] [Logical Configuration of Image Processing Apparatus] Referring to FIGS. 1 to 7, the image processing apparatus 100 according to the first embodiment will be described. First, referring to FIG. 1, the logical configuration of the image processing apparatus 100 according to the first embodiment will be described. FIG. 1 is a block diagram showing an example of the logical configuration of the image processing apparatus 100 according to the first embodiment. The image processing apparatus 100 includes, as a logical configuration, an image acquisition unit 101, a feature acquisition unit 102, a position acquisition unit 103, a setting unit 104, and a learning unit 105.

[0012] The image acquisition unit 101 acquires a plurality of captured images obtained by imaging at least one object from a plurality of directions, and camera parameters corresponding to each captured image. Here, the camera parameters are files or information describing imaging conditions. Specifically, the camera parameters include at least the position of the imaging viewpoint, the direction of the line of sight at the imaging viewpoint (hereinafter referred to as "the orientation of the imaging viewpoint"), and information on the focal length. The camera parameters may include additional information on the settings of the imaging device as necessary. The feature acquisition unit 102 analyzes the spatial frequency of each captured image acquired by the image acquisition unit 101, and acquires the highest frequency having a signal intensity equal to or higher than a given threshold value included in the captured image as an image feature. The position acquisition unit 103 acquires the positional relationship between the imaging viewpoint corresponding to each captured image acquired by the image acquisition unit 101 and the region for estimating spatial information.

[0013] The setting unit 104 estimates the spatial resolution of the learning region necessary for expressing the image feature based on the image feature acquired by the feature acquisition unit 102 and the positional relationship acquired by the position acquisition unit 103, and sets the parameters of the learning model based on the estimated spatial resolution. The learning unit 105 estimates the spatial information by performing learning of the learning model whose parameters are set by the setting unit 104 based on the plurality of captured images acquired by the image acquisition unit 101 and the camera parameters corresponding to each captured image. The learned spatial information is expressed by the parameters of the learned model obtained as a result of the learning.

[0014] <Hardware Configuration of Image Processing Apparatus> The processing of each unit that the image processing apparatus 100 has as a logical configuration is performed by hardware such as a CPU (Central Processor Unit) built in the image processing apparatus 100. The processing of each unit that the image processing apparatus 100 has as a logical configuration may be performed by software using a CPU or GPU (Graphics Processor Unit) and a memory built in the image processing apparatus 100.

[0015] Referring to Figure 2, the hardware configuration of the image processing device 100 will be described when each part of the logical configuration of the image processing device 100 is realized by software execution. Figure 2 is a block diagram showing an example of the hardware configuration of the image processing device 100 according to the first embodiment. The image processing device 100 is composed of a computer. As shown as an example in Figure 2, the computer has a CPU 201, a GPU 202, a ROM 203, a RAM 204, a VRAM 205, an auxiliary storage device 206, a display unit 207, an operation unit 208, a communication unit 209, and a bus 210.

[0016] The CPU 201 controls the computer using programs and data stored in the ROM 203 or RAM 204, thereby realizing the processing of each part of the image processing device 100 shown in Figure 1. The CPU 201 may also work in conjunction with the GPU 202 and VRAM 205 to realize the processing of each part of the image processing device 100. Furthermore, the image processing device 100 may have one or more dedicated processing hardware components distinct from the CPU 201, and these dedicated processing hardware components may perform at least a portion of the processing performed by the CPU 201. Examples of dedicated processing hardware include ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), and DSPs (Digital Signal Processors).

[0017] ROM 203 stores programs and other data that do not require modification. RAM 204 temporarily stores programs or data supplied from the auxiliary storage device 206, or data supplied from an external source via the communication unit 209. VRAM 205 temporarily stores data supplied from ROM 203, RAM 204, or the auxiliary storage device 206. Data stored in VRAM 205 is used for processing by the GPU 202. The auxiliary storage device 206 is configured, for example, as a hard disk drive, and stores various types of data such as image data or audio data. The display unit 207 is configured as a liquid crystal display or LED, and displays a GUI (Graphical User Interface) for the user to operate the image processing device 100 or to view the status or results of processing by the image processing device 100. The operation unit 208 is configured as a keyboard, mouse, or touch panel, and receives user input and inputs various instructions corresponding to the operation to the CPU 201. The CPU 201 also operates as a display control unit that controls the display unit 207, and an operation control unit that controls the operation unit 208.

[0018] The communication unit 209 is used for communication between the image processing device 100 and external devices. For example, if the image processing device 100 is connected to an external device by a wired connection, a communication cable is connected to the communication unit 209. If the image processing device 100 has a function for wireless communication with an external device, the communication unit 209 has an antenna. The bus 210 transmits information by connecting the various parts that the image processing device 100 has as a hardware configuration. In the first embodiment, the display unit 207 and the operation unit 208 are described as being located inside the image processing device 100, but at least one of the display unit 207 and the operation unit 208 may be located outside the image processing device 100 as a separate device.

[0019] <Operation of the image processing device> The operation of the image processing device 100 will be described with reference to Figures 3 to 7. Figure 3 is a flowchart showing an example of the processing flow of the image processing device 100 according to the first embodiment. Figure 4 is a diagram illustrating an example of the image feature acquisition process in the feature acquisition unit 102 according to the first embodiment. Figure 5 is a diagram illustrating an example of the acquisition process of the positional relationship between the imaging viewpoint corresponding to each captured image and the region for estimating spatial information in the position acquisition unit 103 according to the first embodiment. Figure 6 is a diagram illustrating an example of the spatial resolution estimation process in the setting unit 104 according to the first embodiment. Figure 7 is a diagram illustrating an example of the volume rendering process in the learning unit 105 according to the first embodiment. The processing of the flowchart shown in Figure 3 is realized by the CPU 201 loading a program stored in ROM 203 or the like into RAM 204 and executing it. In the following description, the symbol "S" means step.

[0020] First, in S301, the image acquisition unit 101 acquires multiple captured images obtained by capturing at least one object from multiple directions, along with camera parameters corresponding to each captured image. Hereinafter, the captured images acquired by the image acquisition unit 101 will be described as RGB images, but the captured images may be of other forms, such as monochrome images, monochrome images with transparency, or RGB images with transparency.

[0021] Next, in S302, the feature acquisition unit 102 acquires image features of each captured image by analyzing the spatial frequency of each captured image acquired in S301. An example of image feature acquisition by the feature acquisition unit 102 will be explained using Figure 4. Figure 4(a) shows an example of an captured image 401. First, the feature acquisition unit 102 performs a two-dimensional discrete Fourier transform on the captured image 401. Figure 4(b) shows an example of a spatial frequency domain image 402 obtained as a result of the two-dimensional discrete Fourier transform on the captured image 401. Subsequently, the feature acquisition unit 102 integrates the signal intensity for each frequency band in the spatial frequency domain image 402. Figure 4(c) shows an example of a power spectrum 403 obtained as a result of the signal intensity integration.

[0022] Next, the feature acquisition unit 102 performs thresholding on the power spectrum 403 and acquires the highest spatial frequency among the spatial frequencies with signal intensity above the threshold as an image feature 404. If the captured image is an RGB image, the feature acquisition unit 102 acquires, for example, the spatial frequency for each color in the RGB image and acquires image features based on the power spectrum obtained by summing the power spectra for each color. Alternatively, in this case, the feature acquisition unit 102 may acquire image features by, for example, analyzing a luminance image obtained by extracting luminance information from the RGB image.

[0023] Following S302, in S303, the position acquisition unit 103 estimates the imaging area where the angles of view from each imaging viewpoint (hereinafter referred to as the "imaging range") overlap, based on the camera parameters acquired in S5301, and acquires the positional relationship between the estimated imaging area and the imaging viewpoint as positional information. An example of acquiring the positional relationship in the position acquisition unit 103 will be explained using Figure 5. First, the position acquisition unit 103 estimates the area where the imaging ranges of all imaging viewpoints 501 overlap, and designates the estimated area as the imaging area 502. Next, the position acquisition unit 103 acquires the distance 503 from each imaging viewpoint 501 to the imaging area as information indicating the positional relationship between the imaging area and the imaging viewpoint.

[0024] Next, in S304, the setting unit 104 calculates a spatial resolution capable of representing the spatial frequency corresponding to the image features in the learning domain acquired in S302. An example of spatial resolution calculation in the setting unit 104 will be explained using Figure 6. First, the setting unit 104 calculates the length on the sensor 600 used for imaging from the imaging viewpoint 501, based on the number of pixels corresponding to the length of one wavelength of the spatial frequency corresponding to the image feature 404 acquired in S302. Hereinafter, the length on the sensor 600 corresponding to this number of pixels will be referred to as the "sensor size 610".

[0025] Next, the setting unit 104 sets the imaging area 502 estimated in S303 as the learning area 602. Then, the setting unit 104 calculates a width equivalent to one wavelength of the spatial frequency corresponding to the image features in the learning area 602 based on the distance 603 from the imaging viewpoint 601 to the learning area 602 and the focal length 604 of the optical system of the imaging device located at the imaging viewpoint 601. That is, the setting unit 104 calculates the width when the sensor size 610 is projected onto the learning area 602 (hereinafter referred to as "projection width 605"). Next, the setting unit 104 calculates a spatial resolution 606 capable of representing the spatial frequency corresponding to the image features in the learning area for all imaging viewpoints 501 based on the projection width 605 corresponding to each imaging viewpoint 501. For example, if a pinhole model is assumed as the imaging system, the projection width 605 can be calculated using formula (1).

[0026]

number

[0027]

number

[0028]

number

[0029] Next, in S306, the learning unit 105 estimates spatial information by training a learning model whose parameters were set in S305, using the captured image and camera parameters acquired in S301. The trained model obtained as a result of this training is output to an external device, for example, via the communication unit 209. The training of the learning model is performed so that the corresponding color and density are estimated when an arbitrary position (x, y, z) in three-dimensional space and the direction (θ, φ) from the imaging viewpoint to that position are specified.

[0030] Specifically, the learning process of the learning model consists of the following four main steps: The first step is to determine the pixel to be processed and multiple sample points on the light ray based on that pixel and the imaging viewpoint within the learning area. The second step is to calculate density information and color information corresponding to each sample point determined in the first step. The third step is to estimate the color value (pixel value) of the pixel to be processed by integrating the density information and color information calculated in the second step. The fourth step is to update the parameters of the learning model based on the error between the color value (pixel value) estimated in the third step and the color value (pixel value) of the pixel to be processed.

[0031] For example, in the first step, the learning unit 105 determines a sample point at which a ray of light directed from the position of the imaging viewpoint 501 toward each pixel of the captured image obtained from imaging viewpoint 501 intersects with each grid set in the setting unit 104 in the learning area. An example of sample point determination in the learning unit 105 will be explained using Figure 7(a). First, the learning unit 105 sets a sample point 706 at which a ray of light 705 corresponding to an arbitrary pixel 703 in the captured image 700 obtained from imaging viewpoint 701 intersects with a grid in the learning area 702. In Figure 7, the distance 704 is the focal length of the optical system, such as a lens, of the imaging device located at the imaging viewpoint 701.

[0032] Next, in the second step, the learning unit 105 calculates the density information and color information of sample point 706 by interpolating using the density information and color information of the grid points surrounding the grid containing sample point 706. An example of the calculation of the density information and color information of sample point 706 by the learning unit 105 will be explained using Figure 7(b). The learning model holds density information at the coordinates corresponding to each grid point 707 of the grid, and color information corresponding to the direction (θ, φ) from the imaging viewpoint. The density information and color information of sample point 706 are calculated by interpolating based on the density information and color information corresponding to multiple grid points 707 located in the vicinity of sample point 706.

[0033] Next, in the third step, the learning unit 105 integrates the color information and density information of each sample point 706 set on the light ray 705 that travels from the position of the imaging viewpoint 701 to each pixel 703 of the captured image 700, in order of proximity to the imaging viewpoint 701. Through this integration, the value (pixel value) of the pixel 703 corresponding to each captured image 700 is estimated. This method of estimating pixel values ​​is generally called the volume rendering method. Specifically, the pixel value for a light ray r obtained by the volume rendering method can be calculated using the following formulas (4) to (6).

[0034]

number

[0035] Next, in the fourth step, the learning unit 105 trains the learning model by updating the parameters of the learning model so that the difference (error) between the image generated by volume rendering and the captured image as ground truth data becomes smaller. The learning unit 105 repeats the first to fourth steps for each captured image until the learning termination condition is met. Here, the learning termination condition is, for example, when the number of updates of the learning model reaches a predetermined number. Note that the learning termination condition is not limited to the number of updates of the learning model, but may also be, for example, when a predetermined learning time is reached, or when the error or the rate of error reduction falls below a given threshold.

[0036] As described above, in this embodiment, the image processing device 100 is configured to change the spatial resolution of the learning model according to the image characteristics of the captured image when estimating spatial information. With the image processing device 100 configured in this way, the parameters of the learning model that represent spatial information can be set in accordance with the high-frequency components contained in the captured image. As a result, highly accurate spatial information can be estimated while appropriately controlling the amount of computation required to train the learning model.

[0037] In this embodiment, the learning model to be processed in the setting unit 104 and the learning unit 105 was described as having grid-like color information and density information. However, the form of the learning model is not limited to such a three-dimensional grid-like information format. For example, the learning model to be processed may be a DNN (Deep Neural Network) that takes position information and rendering direction information as input and outputs color information and density information. When using such a DNN as the learning model, the number of spatial information parameters of the DNN per volume is changed based on the spatial resolution calculated in S305. By changing the number of spatial information parameters in this way, a learning model with a model size capable of learning representations corresponding to high-frequency components contained in the captured image can be automatically set. Alternatively, for example, the learning model to be processed may be a tetrahedron group format in which the space containing the target object is divided by a plurality of tetrahedra, and each vertex of each tetrahedron has color information and density information. When using such a tetrahedron group format as the learning model, the number of tetrahedra per volume can be increased or decreased using tessellation or tetrahedron integration based on the spatial resolution calculated in S305. This allows for the automatic setting of a learning model with a model size capable of learning representations corresponding to the high-frequency components contained in the captured image, even for learning tetrahedron group forms.

[0038] Furthermore, for example, the learning model to be processed may be 3D Gaussian Splatting (3DGS), which represents a three-dimensional scene by three-dimensionally scattering data points that have information about spatial extent, color, and density. When using 3DGS as the learning model, the number of data points scattered per unit volume is changed based on the spatial resolution calculated in S305. By changing the number of data points scattered per unit volume in this way, a learning model with a model size capable of learning more complex representations corresponding to the high-frequency components contained in the captured image can be automatically set.

[0039] Furthermore, although this embodiment has been described as performing color information and density information learning by the learning unit 105, the learning targets of the learning unit 105 are not limited to these. For example, the learning unit 105 may be configured to learn information such as density at a certain location, signed distance from the object surface, color, or different colors in each direction, or information combining these.

[0040] [Modification 1 of the First Embodiment] In the first embodiment, the position acquisition unit 103 sets the region where the imaging ranges of all imaging viewpoints 501 overlap (imaging region) as a learning region based on camera parameters, and acquires the distance between the imaging viewpoint and the learning region as position information. The setting unit 104 sets the parameters of the learning model based on the position information acquired by the position acquisition unit 103. However, the position information acquired by the position acquisition unit 103 is not limited to information indicating the distance between the imaging viewpoint and the learning region. For example, the position acquisition unit 103 may acquire depth information from each imaging viewpoint as position information based on multiple imaging images acquired by the image acquisition unit 101 and the camera parameters corresponding to each imaging image. In this case, the setting unit 104 sets the parameters of the learning model based on the depth information acquired by the position acquisition unit 103. A modified example of the above will be explained using Figure 8.

[0041] Figure 8 is a diagram illustrating an example of the position information acquisition process in the position acquisition unit 103 according to Modification 1 of the First Embodiment. Figures 8(a) and (b) show examples of captured images obtained by imaging from different imaging viewpoints. First, the position acquisition unit 103 calculates the distance from each imaging viewpoint to the object by performing stereo matching between the captured images shown in Figures 8(a) and (b). Next, the position acquisition unit 103 generates depth information based on the distance obtained by the calculation and acquires the generated depth information as position information. Figure 8(c) shows an example of a depth image shown by the depth information obtained by the position acquisition unit 103 performing stereo matching between the captured images shown in Figures 8(a) and (b).

[0042] In the above description, depth information was explained as being obtained by the position acquisition unit 103 calculating the distance from each imaging viewpoint to an object based on the captured image acquired by the image acquisition unit 101 and camera parameters, but it is not limited to this. For example, the position acquisition unit 103 may acquire depth information by acquiring depth image data calculated and output by an external device of the image processing device 100 via the communication unit 209.

[0043] The setting unit 104 estimates the spatial resolution of the learning region necessary to represent image features based on the distance from the imaging viewpoint to the object indicated by the depth information acquired as position information by the position acquisition unit 103, and sets the parameters of the learning model based on that spatial resolution.

[0044] If an object exists within the imaging area 502, the depth value, which corresponds to the distance from the imaging viewpoint 501 to the object, will be larger than the distance from the imaging viewpoint 501 to the imaging area 502. Therefore, when calculating the spatial resolution r using formula (3), the spatial resolution r will be lower when depth information is used compared to the spatial resolution r according to the first embodiment. As a result, in the learning process in the learning unit 105, the number of points where the light ray intersects with the grid of the learning area is reduced, and the amount of computation required for the learning process can be reduced.

[0045] The learning unit 105 may set initial values ​​for the parameters of the learning model based on the depth information, which is the position information acquired by the position acquisition unit 103. For example, the learning unit 105 can start learning the learning model from a state close to the convergence of learning by setting a high initial value for the density information for the coordinates corresponding to the object surface that can be estimated based on the depth value indicated by the acquired depth information.

[0046] In the above description, as an example, depth information calculated by stereo matching using two captured images was used as position information, but the method of acquiring depth information is not limited to this. For example, the position acquisition unit 103 may calculate the distance from each imaging viewpoint to the object by using a multi-view stereo method that uses three or more captured images obtained by imaging from three or more imaging viewpoints. Alternatively, for example, instead of stereo matching using two captured images or a multi-view stereo method using three or more captured images, the position acquisition unit 103 may estimate the distance from each imaging viewpoint to the object using a trained model obtained as a result of deep learning or the like.

[0047] [Modification 2 of the First Embodiment] In the first embodiment, the feature acquisition unit 102 analyzed the spatial frequencies of each captured image and acquired the highest spatial frequency among the spatial frequencies having components greater than a threshold as an image feature. However, the method of acquiring image features in the feature acquisition unit 102 is not limited to this. For example, the feature acquisition unit 102 may set the parameters of the learning model based on the shape of the object in the captured image. Specifically, for example, the feature acquisition unit 102 first extracts from each captured image a region containing the image of the object (hereinafter referred to as the "foreground region") and a region related to the background (hereinafter referred to as the "background region"). Subsequently, the feature acquisition unit 102 estimates the shape of the object in the captured image based on the foreground region in each extracted captured image. Subsequently, the feature acquisition unit 102 sets the parameters of the learning model based on the estimated information regarding the shape of the object in the captured image.

[0048] Figure 9 is a diagram illustrating an example of the image feature acquisition process in the feature acquisition unit 102 according to a modified example 2 of the first embodiment. Figure 9(a) shows an example of the captured image 901. The feature acquisition unit 102 generates a silhouette image showing the foreground region by extracting the region containing the image of an object (foreground region) from the captured image 901. The method for extracting the foreground region may be a method based on the difference with a previously captured background image, a method based on chroma key processing using a green screen, or a method using a trained model that has been trained to separate the foreground region and the background region of the input image. Figure 9(b) shows an example of the silhouette image 902. In the silhouette image 902 shown in Figure 9(b), the region corresponding to the foreground region in the captured image 901 is represented in black, and the region corresponding to the region other than the foreground region (background region) is represented in white.

[0049] Next, the feature acquisition unit 102 generates a contour image showing the contour of an object by extracting the contour of the region corresponding to the foreground region in the silhouette image 902. Figure 9(c) shows an example of the contour image 910. Figure 9(d) is an enlarged view of a subregion 903 in the contour image 910. In the contour image 910, the feature acquisition unit 102 acquires the normal vector 904 pointing inward from each pixel constituting the contour of the object (hereinafter referred to as the "inward normal vector"). Next, the feature acquisition unit 102 calculates the minimum distance from each pixel constituting the contour to other pixels constituting the contour that are located in the direction of the inward normal vector 904 corresponding to each pixel constituting the contour as the object width 905. Next, the feature acquisition unit 102 acquires the smallest object width 905 among the calculated object widths 905 corresponding to all pixels constituting the contour as an image feature of the captured image 901.

[0050] The setting unit 104 calculates the sensor size based on the number of pixels corresponding to the object width 905 acquired as image features by the feature acquisition unit 102. Next, the setting unit 104 calculates the spatial resolution in the learning domain using formula (2), etc., based on the calculated sensor size. Subsequently, the setting unit 104 sets the parameters of the learning model based on the calculated spatial resolution in the learning domain.

[0051] With the image processing device 100 configured as described above, it is possible to set parameters for a learning model that can reproduce the foreground region in the captured image 901.

[0052] [Second Embodiment] The image processing device 100 according to the first embodiment acquired image features for each captured image and set parameters with the same number of spatial information parameters throughout the entire learning domain. However, if an object does not have a uniform texture or shape from all imaging viewpoints, the number of spatial information parameters in a learning model suitable for representing the spatial information corresponding to the object will differ depending on the part of the object. Therefore, in the second embodiment, an embodiment will be described in which parameters with a suitable number of spatial information parameters are set for each sub-domain of the learning domain in the learning model.

[0053] <Configuration of the image processing device> The image processing apparatus according to the second embodiment has, as a logical configuration, an image acquisition unit, a feature acquisition unit, a position acquisition unit, a setting unit, and a learning unit. Hereinafter, unless otherwise specified, the image processing apparatus according to the second embodiment will be referred to as "image processing apparatus 100". Also, hereafter, unless otherwise specified, each part of the logical configuration of the image processing apparatus 100 will be referred to as the image acquisition unit 101, the feature acquisition unit 102, the position acquisition unit 103, the setting unit 104, and the learning unit 105. The processing of each part of the logical configuration of the image processing apparatus 100 is performed by hardware such as a CPU built into the image processing apparatus 100. The processing of each part of the logical configuration of the image processing apparatus 100 may also be performed by software using a CPU or GPU (and memory) built into the image processing apparatus 100. In addition, the image processing apparatus 100 may have one or more dedicated processing hardware separate from the CPU 201, and the dedicated processing hardware may execute at least a part of the processing performed by the CPU 201.

[0054] The processing of the image acquisition unit 101 is the same as that of the image acquisition unit 101 according to the first embodiment, so a description is omitted. The feature acquisition unit 102 divides each captured image into multiple sub-images and acquires image features corresponding to each sub-image. The method for acquiring image features corresponding to each sub-image in the feature acquisition unit 102 is the same as the method for acquiring image features corresponding to the captured image in the feature acquisition unit 102 according to the first embodiment, so a description is omitted.

[0055] The position acquisition unit 103 acquires position information for each small image divided by the feature acquisition unit 102, specifically the distance from the imaging viewpoint to the part of the object included as an image in the small image. Specifically, first, the position acquisition unit 103 estimates the three-dimensional shape of the object using the view volume cross-eyed method or the like, using multiple imaging images acquired by the image acquisition unit 101 and camera parameters corresponding to each imaging image. Subsequently, the position acquisition unit 103 uses the estimated three-dimensional shape to acquire position information for each small image, specifically the distance from the imaging viewpoint to the part of the object included as an image in the small image. Alternatively, the position acquisition unit 103 may acquire position information for each small image by acquiring depth values ​​from the depth image using the same method as in the modified example 1 of the first embodiment.

[0056] The setting unit 104 sets multiple sub-learning regions in the learning region and sets the parameters of the learning model for each sub-learning region based on the image features corresponding to each sub-image acquired by the feature acquisition unit 102. Specifically, first, the setting unit 104 calculates the spatial resolution around the point where the light ray corresponding to each pixel of the sub-image intersects with the object, based on the image features of each sub-image acquired by the feature acquisition unit 102 and the position information of each sub-image acquired by the position acquisition unit 103. Next, the setting unit 104 sets the parameters of the learning model for each of the multiple sub-learning regions set in the learning region based on the estimated spatial resolution. The learning unit 105 estimates spatial information corresponding to the entire learning region by training the learning model for each sub-learning region set by the setting unit 104.

[0057] <Operation of the image processing device> The operation of the image processing device 100 will be described with reference to Figures 10 to 14. Figure 10 is a flowchart showing an example of the processing flow of the image processing device 100 according to the second embodiment. Figure 11 is a diagram illustrating an example of the image feature acquisition process in the feature acquisition unit 102 according to the second embodiment. Figure 12 is a diagram illustrating an example of the location information acquisition process in the location acquisition unit 103 according to the second embodiment. Figure 13 is a diagram illustrating an example of the spatial resolution estimation process according to a small image in the setting unit 104 according to the second embodiment. Figure 14 is a diagram illustrating an example of the learning model parameter setting process in the setting unit 104 according to the second embodiment. First, the image processing device 100 executes the process S301 shown in Figure 3.

[0058] Next, in S1001, the feature acquisition unit 102 divides each captured image acquired in S301 to generate multiple sub-images. Figure 11(a) shows an example of the captured image 1100. The feature acquisition unit 102 generates multiple sub-images by dividing the captured image 1100. Figure 11(b) shows an example of a sub-image 1101 generated by dividing the captured image 1100. In this embodiment, as shown as an example in Figure 11(b), the sub-images 1101 are generated such that there are no overlapping regions between the sub-images by dividing the captured image 1100, but the method of generating sub-images is not limited to this. For example, the feature acquisition unit 102 may generate sub-images such that parts of the image regions in the captured image 1100 overlap between adjacent sub-images.

[0059] Next, in S1002, the feature acquisition unit 102 analyzes each small image 1101 in the same manner as in the first embodiment and acquires the highest spatial frequency among the spatial frequencies whose power spectrum of spatial frequencies is above a threshold as an image feature of each small image 1101. Figure 11(c) shows an example of a spatial frequency domain image 1102 corresponding to each small image 1101, obtained as a result of a two-dimensional discrete Fourier transform on each small image 1101. Figure 11(d) shows an example of image features 1103 and 1104 corresponding to each small image 1101. Note that for small images that do not contain an image of an object, such as image feature 1104, a label or value indicating this may be assigned.

[0060] Next, in S1003, the position acquisition unit 103 generates a silhouette image by extracting the region containing the image of the object (foreground region) from each captured image acquired in S301, and obtains the general shape of the object based on the generated silhouette image. Specifically, for example, the position acquisition unit 103 obtains the general shape of the object by the viewing volume cross-eyed method or the like. Figure 12(a) shows an example of an captured image 1200. The position acquisition unit 103 generates a silhouette image corresponding to each captured image by extracting the foreground region from each captured image. Figure 12(b) shows an example of a silhouette image 1201 showing the foreground region 1202 generated by extracting the foreground region from the captured image 1200.

[0061] Next, the position acquisition unit 103 acquires the approximate shape of the object using the silhouette image corresponding to each captured image and the camera parameters corresponding to each captured image. Figure 12(c) is a diagram illustrating an example of the process for acquiring the approximate shape of an object. The position acquisition unit 103 acquires the approximate shape 1203 of the object, for example, by the viewing volume cross-eyed method. Specifically, the position acquisition unit 103 acquires the approximate shape 1203 of the object by projecting the background region in the silhouette image 1201 corresponding to each imaging viewpoint 501 into three-dimensional space using the camera parameters corresponding to the captured image of each imaging viewpoint 501. The position acquisition unit 103 may also acquire the approximate shape 1203 of the object by projecting the foreground region 1202 in the silhouette image 1201 into three-dimensional space.

[0062] More specifically, for example, the position acquisition unit 103 illuminates the three-dimensional space with light rays corresponding to each pixel (hereinafter referred to as "background pixel") included in the background region of the silhouette image 1201 corresponding to each imaging viewpoint 501. The position acquisition unit 103 acquires the approximate shape 1203 of the object by estimating the region containing the object as the three-dimensional space that does not intersect with any of the light rays corresponding to the background pixels. In the silhouette images 1201 shown in Figures 12(b) and (c), the region represented in black is the foreground region 1202, and the region represented in white is the background region. Also, although Figure 12(c) shows a cross-section of the silhouette image 1201, in reality, when acquiring the approximate shape of a three-dimensional object, the entire silhouette image 1201 is used to estimate the approximate shape of the object.

[0063] In S1004, the position acquisition unit 103 calculates the distance from the imaging viewpoint 501 to the point of intersection by calculating the position of the point where the light ray corresponding to each pixel included in the foreground region of each small image (hereinafter referred to as "foreground pixel") intersects with the general shape of the object. Next, in S1005, the setting unit 104 calculates the spatial resolution of each small image by performing the same processing as the spatial resolution estimation processing of the first embodiment (processing in S304 shown in Figure 3) for each small image.

[0064] The processes of S1004 and S1005 will be explained using Figure 13. First, the setting unit 104 calculates the coordinates of the points where the light rays 1314 and 1324 corresponding to the foreground pixels in the captured image intersect with the surface of the approximate shape 1302 of the object. Next, the setting unit 104 acquires the distances 1313 and 1323 between the imaging viewpoint 1301 and the intersection points as positional information indicating the positional relationship. In Figure 13, distance 1304 is the focal length of the optical system, such as a lens, of the imaging device located at the imaging viewpoint 1301. In Figure 13, s1 and s2 represent the sensor size on the sensor 1300 corresponding to one wavelength of the image features (spatial frequency) corresponding to each of the two small images. Also, in Figure 13, distances 1313 and 1323 represent the distance from the imaging viewpoint 1301 to the approximate shape 1302 of the object, corresponding to each of the two small images. The setting unit 104 calculates the spatial resolutions 1316 and 1326 corresponding to the two small images by using formula (2), etc., similar to the first embodiment. As shown in Figure 13, if the distances 1313 and 1323 to the sensor size or the approximate shape 1302 of the object corresponding to the image features of each small image are different, then different values ​​are calculated as the spatial resolutions 1316 and 1326.

[0065] Furthermore, if the direction of the normal to the point where the optical axis of the imaging device located at the imaging viewpoint 1301 intersects with the approximate shape 1302 of the object is deviated from the direction of the optical axis, the surface of the object is inclined with respect to a plane perpendicular to the optical axis. Therefore, the captured image has a higher spatial frequency than the actual texture on the surface of the object. If the spatial resolution 1316,1326 is sufficiently small for the distances 1313,1323 to the approximate shape 1302 of the object, then, if θ is the angle between the normal and the optical axis, the spatial resolution can be approximated by multiplying the spatial resolution according to the first embodiment by 1 / cosθ. This approximation can correct the distortion that occurs when the texture of the surface of an object is imaged from an oblique angle, resulting in an increased spatial resolution of the texture.

[0066] Next, in S1006, the setting unit 104 sets up multiple sub-learning regions in the learning region based on the general shape of the object, and sets the parameters of the learning model for each sub-learning region based on the spatial resolution calculated in S1005. The setting process of the sub-learning model in the setting unit 104 will be explained using Figure 14. First, the setting unit 104 sets up multiple sub-learning regions 1401 to 1404 in the learning region 1400 that encompasses the general shape 1410 of the object. Below, as an example, a configuration in which four sub-learning regions 1401 to 1404 are set in the learning region 1400 that encompasses the general shape 1410 of the object will be explained. The number of sub-learning regions to be set in the learning region 1400 may be three or fewer, or five or more. Furthermore, the method of setting up the sub-learning regions may be to set up a sub-learning region of a predetermined size in the learning region, or to set up multiple sub-learning regions by dividing the learning region into a predetermined number of parts. Alternatively, the sub-learning regions may be set up based on the size or shape of the general shape 1203 of the object acquired by the position acquisition unit 103.

[0067] Next, the setting unit 104 assigns spatial resolutions 1411 to 1414, calculated based on the small images of each imaging viewpoint 1421 and 1422, to the small learning regions 1401 to 1404. Subsequently, the setting unit 104 sets the parameters of the learning model for each small learning region 1401 to 1404 based on the assigned spatial resolutions 1411 to 1414. Note that small learning region 1402 is assigned two spatial resolutions 1412 and 1413. In cases where multiple spatial resolutions are assigned to a single small learning region, the setting unit 104 sets the parameters of the learning model for that small learning region based, for example, on the highest spatial resolution among the multiple spatial resolutions assigned to that small learning region.

[0068] Next, in S1007, the learning unit 105 learns a learning model for each sub-learning region 1401 to 1404 whose parameters were set in S1006, based on each captured image acquired in S301 and the camera parameters corresponding to each captured image. As a result of this learning, spatial information corresponding to each sub-learning region 1401 to 1404 is estimated. In this embodiment, the learning model has grid-like color information and density information, similar to the first embodiment. Furthermore, the learning process in the learning unit 105 performs the same learning process for the learning model for each sub-learning region 1401 to 1404 as the learning unit 105 in the first embodiment.

[0069] With the image processing device 100 configured as described above, parameters can be set for the learning model relating to multiple sub-learning regions set in the learning region encompassing the general shape of an object, in accordance with the high-frequency components contained in the captured image. As a result, high-precision spatial information can be estimated while appropriately controlling the computational load required for training the learning model.

[0070] In this embodiment, the position acquisition unit 103 is described as acquiring the approximate shape of an object using the view volume cross-eyed method, but the method for acquiring the approximate shape of an object is not limited to the view volume cross-eyed method. For example, the approximate shape of an object may be estimated by a multi-view stereo method using multiple captured images obtained from multiple imaging viewpoints. Also, for example, if the object is a human-shaped object, the approximate shape of an object may be estimated using captured images by bone estimation or pose estimation and deformation of a standard human-shaped model. Furthermore, if multiple objects exist in the space that can be targeted as the learning area, the feature acquisition unit 102 may generate small images based on the position of each object. In this case, the setting unit 104 may set the learning area or small learning area based on the position of each object. Furthermore, the setting unit 104 may set initial values ​​for parameters related to the density of the learning model based on the information regarding the approximate shape of the object acquired by the position acquisition unit 103.

[0071] [Modification 1 of the second embodiment] The image processing device 100 according to the second embodiment sets up multiple sub-learning regions in the learning region and sets the parameters of the learning model for each sub-learning region. However, the learning region may be recursively divided using an octree or the like, and the parameters of the learning model may be set for each of the recursively divided regions. Figure 15 is a diagram illustrating an example of the setting process for setting parameters of the learning model by octree spatial partitioning in the setting unit 104 according to Modification 1 of the second embodiment. Specifically, Figure 15(a) shows the learning region 1500 divided so that its area is reduced to 1 / 4. Figure 15(b) shows the learning region 1500 being recursively divided so that its area is reduced to 1 / 4.

[0072] As shown in Figure 15, the setting unit 104 recursively divides the learning area 1500 so that the area is reduced to 1 / 4, etc. Specifically, as shown in Figure 15(a), the setting unit 104 recursively divides each of the four divided areas 1501 to 1504 of the learning area 1500 until the area is smaller than the estimated spatial resolution, as shown in Figure 15(b). In Figure 15, shape 1510 is the three-dimensional shape of the object. The sizes of rectangles 1521 to 1524 represent the spatial resolutions 1511 to 1514 of the object. More specifically, as shown in Figure 15(b), for example, the learning area corresponding to the vicinity of the surface of an object having a spatial resolution 1512 represented by rectangle 1522 is divided until the division size is less than or equal to the size of rectangle 1522. Also, as shown in Figure 15(b), for example, the learning area corresponding to the vicinity of the surface of an object having a spatial resolution represented by rectangle 1513 is divided until the division size is less than or equal to the size of rectangle 1523.

[0073] By setting the parameters of a learning model that has color information and density information based on this octree-like partitioning information, it is possible to set up a learning model that has different expressive powers for each location in the learning space.

[0074] [Other embodiments] In the above-described embodiment, the image processing device 100 was described as having a learning unit 105 as part of its logical configuration. However, the image processing device 100 does not necessarily have to have a learning unit 105. In this case, the image processing device 100 outputs a learning model, whose parameters are set in the setting unit 104, to an information processing device having a learning unit 105 as part of its logical configuration, and the information processing device performs learning on the learning model. Here, the information processing device may be composed of, for example, one or more server devices or personal computers.

[0075] In addition to the logical configuration shown in Figure 1, the image processing device 100 may also have a viewpoint acquisition unit and an image generation unit as logical configurations, which are not shown in Figure 1. The processing of the viewpoint acquisition unit and the image generation unit is performed, for example, by hardware such as a CPU built into the image processing device 100. Here, the viewpoint acquisition unit is a logical configuration that acquires virtual viewpoint information, which includes information about the position of a virtual viewpoint and the direction of the line of sight at the virtual viewpoint. The image generation unit is a logical configuration that generates an image (virtual viewpoint image) corresponding to the view from the virtual viewpoint, based on spatial information obtained as a result of learning the learning model in the learning unit 105 and the virtual viewpoint information acquired by the viewpoint acquisition unit. In this case, the image processing device 100 may output the generated virtual viewpoint image in addition to, or instead of, the estimated spatial information.

[0076] This disclosure can also be implemented by supplying a program that implements one or more of the functions of the embodiments described above to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be implemented by a circuit (e.g., an ASIC) that implements one or more of the functions.

[0077] Furthermore, within the scope of this disclosure, the technologies described herein allow for free combination of each embodiment, modification of any component of each embodiment, or omission of any component in each embodiment.

[0078] [Technical Features of This Disclosure] This disclosure includes the following configurations, methods, and programs.

[0079] <Configuration 1> Image acquisition means for acquiring multiple images obtained by capturing a three-dimensional space containing an object from multiple directions, A feature acquisition means that acquires image features related to the image by analyzing the aforementioned image, A position acquisition means for acquiring position information indicating the position of the aforementioned object, A setting means for setting parameters of a learning model that estimates spatial information relating to the learning region included in the three-dimensional space based on the position information and the image features, An image processing apparatus characterized by having <Configuration 2> A learning means for training the aforementioned learning model, The image processing apparatus according to configuration 1, further comprising the above.

[0080] <Structure 3> The setting means sets the parameters in the learning model that can represent a high spatial resolution in the learning domain when the spatial frequency of the image is high. An image processing apparatus according to configuration 1 or 2, characterized by the above.

[0081] <Structure 4> The setting means sets the parameters in the learning model that can represent high spatial resolution in the learning region when the distance from the position where the image was taken to the object or the learning region is short. An image processing apparatus according to any one of configurations 1 to 3 characterized by the above.

[0082] <Composition 5> The setting means sets the parameters in the learning model to be able to represent high spatial resolution in the learning domain when the angle between the optical axis of the imaging device corresponding to the image and the normal on the surface of the object intersecting the optical axis is small. An image processing apparatus according to any one of configurations 1 to 4 characterized by the above.

[0083] <Composition 6> The setting means sets the parameter that can represent the highest spatial resolution among the multiple estimated spatial resolutions for the learning region or object, when there are multiple spatial resolutions estimated based on the image features corresponding to each of the multiple images, in the learning model. An image processing apparatus according to any one of configurations 1 to 5 characterized by the above.

[0084] <Composition 7> The setting means sets the parameters based on the spatial resolution estimated for each of the multiple learning areas in which different parameters can be set, for each of the learning models corresponding to each of the multiple learning areas in which different parameters can be set, or for each of the multiple sub-regions included in the learning area in which different parameters can be set. An image processing apparatus according to any one of configurations 1 to 6 characterized by the above.

[0085] <Structure 8> The feature acquisition means acquires the frequency characteristics related to spatial frequencies obtained by analyzing the image as the image features. An image processing apparatus according to any one of configurations 1 to 7 characterized by the above.

[0086] <Composition 9> The feature acquisition means acquires the minimum value of the width in the foreground region as the image feature, based on the widths of a plurality of pixels constituting the foreground region including the image of the object in the image. An image processing apparatus according to any one of configurations 1 to 8 characterized by the above.

[0087] <Configuration 10> The feature acquisition means generates a plurality of sub-images from the image and acquires the image features related to the image for each sub-image. An image processing apparatus according to any one of configurations 1 to 9 characterized by the above.

[0088] <Composition 11> The position acquisition means acquires as position information a depth image showing the distance from the position where the image was taken to the object, or information regarding the three-dimensional shape of the object estimated based on the plurality of images. An image processing apparatus according to any one of configurations 1 to 10 characterized by the above.

[0089] <Composition 12> The position acquisition means acquires the position information corresponding to each of the multiple objects when multiple objects exist in the learning area. An image processing apparatus according to any one of configurations 1 to 11 characterized by the above.

[0090] <Composition 13> The learning model is such that information about each position in the learning domain is represented in at least one of the following forms: a multilayer neural network, a three-dimensional grid, a three-dimensional grid composed of an octvine structure, a tetrahedron group, a three-dimensional point cloud, and 3D Gaussian splatting. An image processing apparatus according to any one of configurations 1 to 12 characterized by the above.

[0091] <Composition 14> The spatial information includes at least one of the following for each of the multiple locations in the learning domain: information about density, information about the signed distance from the surface of the object, information about color, and information about color corresponding to each of the multiple directions. An image processing apparatus according to any one of configurations 1 to 13 characterized by the above.

[0092] <Composition 15> A viewpoint acquisition means for acquiring information about a virtual viewpoint, Image generation means for generating a virtual viewpoint image corresponding to the view from the virtual viewpoint based on the spatial information obtained as a result of training the learning model and the information relating to the virtual viewpoint, Having further, An image processing apparatus according to any one of configurations 1 to 14 characterized by the above.

[0093] <Method> Image acquisition means for acquiring multiple images obtained by capturing a three-dimensional space containing an object from multiple directions, A feature acquisition means that acquires image features related to the image by analyzing the aforementioned image, A position acquisition means for acquiring position information indicating the position of the aforementioned object, A setting means for setting parameters of a learning model that estimates spatial information relating to the learning region included in the three-dimensional space based on the position information and the image features, An image processing method characterized by including

[0094] <Program> A program for causing a computer to function as an image processing device described in any one of configurations 1 to 15. [Explanation of Symbols]

[0095] 100 Image Processing Devices 101 Image acquisition unit 102 Feature Acquisition Unit 103 Position acquisition part 104 Settings Section 105 Learning Department

Claims

1. Image acquisition means for acquiring multiple images obtained by capturing a three-dimensional space containing an object from multiple directions, A feature acquisition means that acquires image features related to the image by analyzing the aforementioned image, A position acquisition means for acquiring position information indicating the position of the aforementioned object, A setting means for setting parameters of a learning model that estimates spatial information relating to the learning region included in the three-dimensional space based on the position information and the image features, An image processing apparatus characterized by having

2. A learning means for training the aforementioned learning model, The image processing apparatus according to claim 1, further comprising the above.

3. The setting means sets the parameters in the learning model that can represent a high spatial resolution in the learning domain when the spatial frequency of the image is high. The image processing apparatus according to claim 1, characterized in that

4. The setting means sets the parameters in the learning model that can represent high spatial resolution in the learning region when the distance from the position where the image was taken to the object or the learning region is short. The image processing apparatus according to claim 1, characterized in that

5. The setting means sets the parameters in the learning model that can represent high spatial resolution in the learning domain when the angle between the optical axis of the imaging device corresponding to the image and the normal on the surface of the object intersecting the optical axis is small. The image processing apparatus according to claim 1, characterized in that

6. The setting means sets the parameter that can represent the highest spatial resolution among the multiple estimated spatial resolutions for the learning region or object, when there are multiple spatial resolutions estimated based on the image features corresponding to each of the multiple images, in the learning model. The image processing apparatus according to claim 1, characterized in that

7. The setting means sets the parameters based on the spatial resolution estimated for each of the multiple learning areas in which different parameters can be set, for each of the learning models corresponding to each of the multiple learning areas in which different parameters can be set, or for each of the multiple sub-regions included in the learning area in which different parameters can be set. The image processing apparatus according to claim 1, characterized in that

8. The feature acquisition means acquires the frequency characteristics related to spatial frequencies obtained by analyzing the image as the image features. The image processing apparatus according to claim 1, characterized in that

9. The feature acquisition means acquires the minimum value of the width in the foreground region as the image feature, based on the widths of a plurality of pixels constituting the foreground region including the image of the object in the image. The image processing apparatus according to claim 1, characterized in that

10. The feature acquisition means generates a plurality of sub-images from the image and acquires the image features related to the image for each sub-image. The image processing apparatus according to claim 1, characterized in that

11. The position acquisition means acquires as position information a depth image showing the distance from the position where the image was taken to the object, or information regarding the three-dimensional shape of the object estimated based on the plurality of images. The image processing apparatus according to claim 1, characterized in that

12. The position acquisition means acquires the position information corresponding to each of the multiple objects when multiple objects exist in the learning area. The image processing apparatus according to claim 1, characterized in that

13. The learning model is such that information about each position in the learning domain is represented in at least one of the following forms: a multilayer neural network, a three-dimensional grid, a three-dimensional grid composed of an octvine structure, a tetrahedron group, a three-dimensional point cloud, and 3D Gaussian splatting. The image processing apparatus according to claim 1, characterized in that

14. The spatial information includes at least one of the following for each of the multiple locations in the learning domain: information about density, information about the signed distance from the surface of the object, information about color, and information about color corresponding to each of the multiple directions. The image processing apparatus according to claim 1, characterized in that

15. A viewpoint acquisition means for acquiring information about a virtual viewpoint, Image generation means for generating a virtual viewpoint image corresponding to the view from the virtual viewpoint based on the spatial information obtained as a result of training the learning model and the information relating to the virtual viewpoint, Having further, The image processing apparatus according to claim 1, characterized in that

16. An image acquisition process that acquires multiple images obtained by capturing a three-dimensional space containing an object from multiple directions, A feature acquisition step involves analyzing the aforementioned image to acquire image features related to the aforementioned image, A position acquisition step to acquire position information indicating the position of the object, A setting step of setting parameters for a learning model that estimates spatial information relating to the learning region included in the three-dimensional space based on the position information and the image features, An image processing method characterized by including [a certain element].

17. A program for causing a computer to function as an image processing device according to any one of claims 1 to 15.

Images