Image processing apparatus, image processing method, and storage medium

Abstract

To reduce an operation amount required for estimating a three dimensional field.SOLUTION: The image processing apparatus 102 according to the present disclosure acquires a plurality of captured images obtained by imaging from a plurality of directions, sets, for each object, a three dimensional space including the object as a learning space on the basis of the plurality of captured images, and learns, for each set learning space, a three dimensional field corresponding to the learning space on the basis of the plurality of captured images. Here, when the image processing apparatus 102 learns a three dimensional field corresponding to a learning space based on a plurality of captured images obtained by synchronized imaging at a certain time point, for a learning space in which an object included in the learning space is a stationary object, the image processing apparatus 200 learns a feature amount of a three dimensional field corresponding to a learning space including a moving object using a feature amount of a three dimensional field already obtained as a result of learning based on a plurality of captured images obtained by synchronized imaging at another time point as a feature amount of a three dimensional field corresponding to the learning space.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present disclosure relates to techniques for estimating a three-dimensional field corresponding to a three-dimensional space of an object. [Background technology]

[0002] There is a technology that estimates a three-dimensional field corresponding to a scene in a three-dimensional space of an object using data from multiple captured images (hereinafter referred to as "multi-viewpoint images") obtained by capturing images from multiple different viewpoints. There is also a technology that uses the estimated three-dimensional field to generate an image (hereinafter referred to as a "virtual viewpoint image") corresponding to the view of the scene from an arbitrary virtual viewpoint (hereinafter referred to as a "virtual viewpoint"). Non-Patent Document 1 discloses, as an example of a three-dimensional field estimation technology, a technology that estimates radiance fields using Neural Radiance Fields (NeRF) composed of a deep learning neural network. By inputting virtual viewpoint information indicating the position of an arbitrary virtual viewpoint and the line of sight direction at the virtual viewpoint into a trained NeRF obtained as a result of training the NeRF using multi-viewpoint images, a virtual viewpoint image corresponding to the view of the scene from the virtual viewpoint is obtained. Specifically, by inputting the virtual viewpoint information into the trained NeRF, the color and volume density corresponding to the scene are estimated. The pixel value of the virtual viewpoint image is obtained by integrating these colors and volume densities. Here, volume density is an index representing the opacity of a color.

[0003] When training NeRF, the following series of processes are repeatedly executed. First, information indicating the position of the imaging device (hereinafter referred to as the "imaging position") and the optical axis direction of the imaging device (hereinafter referred to as the "attitude") is input to the NeRF during training. Based on this input information, NeRF executes a process similar to the virtual viewpoint image generation process described above, thereby generating an image corresponding to the captured image obtained by imaging with the imaging device. Next, using the captured image data as training data, the weight parameters of the neural network that constitutes NeRF are updated so that the difference between corresponding pixel values ​​of the image generated by NeRF and the captured image becomes smaller. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Ben Mildenhall, 5 others, “NeRF:Representing Scene As Neural Radiance Fields For View Synthesis”, [online], August 3, 2020, arXiv, [searched on July 3, 2020], Internet<https: / / arxiv.org / pdf / 2003.08934.pdf> Summary of the Invention [Problem to be solved by the invention]

[0005] In order to estimate 3D fields such as radiance fields with high accuracy using NeRF, it is necessary to repeat the above-mentioned learning process using a large number of multi-viewpoint images. As a result, there is a problem in that estimating 3D fields requires a huge amount of calculations. [Means for solving the problem]

[0006] The image processing device according to the present disclosure comprises an acquisition means for acquiring a plurality of captured images obtained by synchronized imaging of an imaging space from a plurality of directions; a setting means for setting, for each object present in the imaging space, a three-dimensional space including the object as a learning space based on the plurality of captured images; and a learning means for learning, for each set learning space, a three-dimensional field corresponding to the learning space based on the plurality of captured images. When the learning means learns the three-dimensional field corresponding to the learning space based on the plurality of captured images obtained by synchronized imaging at a certain point in time, for a learning space in which the object included in the learning space is a stationary object, the learning means uses the three-dimensional field features already obtained as a result of learning based on the plurality of captured images obtained by synchronized imaging at another point in time as the three-dimensional field features corresponding to the learning space to learn the three-dimensional field features corresponding to the learning space in which a moving object is included. [Effects of the Invention]

[0007] According to the present disclosure, the amount of calculation required to estimate a three-dimensional field can be reduced. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of an imaging system according to a first embodiment. [Figure 2] 1 is a block diagram showing an example of a hardware configuration of an image processing device according to a first embodiment. [Figure 3] 3A to 3C are diagrams for explaining a method for estimating a three-dimensional field in the image processing device according to the first embodiment. [Figure 4] 1 is a block diagram showing an example of a functional configuration of an image processing device according to a first embodiment. [Figure 5] 4 is a flowchart showing an example of a processing flow of the image processing device according to the first embodiment. [Figure 6] FIG. 2 is a diagram for explaining an example of a feature amount according to the first embodiment. [Figure 7]5 is a flowchart showing an example of the flow of learning processing by a learning unit according to the first embodiment. [Figure 8] FIG. 4 is a diagram for explaining an example of a learning process of a learning unit according to the first embodiment. [Figure 9] 10 is a flowchart showing an example of a processing flow of an image processing device according to a second embodiment. [Figure 10] 10 is a flowchart showing an example of the flow of learning processing by a learning unit according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that the following embodiments do not limit the means for solving the problems according to the present disclosure, and not all of the combinations of features described in the present embodiments are necessarily essential to the means for solving the problems according to the present disclosure. Note that the same components will be described with the same reference numerals.

[0010] [Embodiment 1] In embodiment 1, we describe an aspect in which, in estimating a three-dimensional field, a static space in a scene is identified based on multi-view images obtained by synchronized imaging at two different points in time, and an already estimated three-dimensional field is reused for the identified static space.

[0011] <Image capture system configuration> FIG. 1 is a diagram showing an example of the configuration of an imaging system according to the first embodiment. The imaging system includes a plurality of imaging devices 101, an image processing device 102, a user interface (hereinafter referred to as "UI") panel 103, a storage device 104, and a display device 105. Each imaging device 101 is configured as a digital still camera, a digital video camera, or the like, and is arranged in a different position. Each imaging device 101 captures synchronized images of an object 107 and an object 108 present in an imaging space 106 from different viewpoints according to set imaging conditions. Each imaging device 101 generates and outputs captured image data corresponding to each viewpoint through the imaging.

[0012] Note that "synchronized imaging" refers to imaging performed after synchronization processing, and includes imaging performed at approximately the same time. Captured image data obtained by imaging device 101 may be still image data, moving image data, or both still image and moving image data. Hereinafter, the term "image" will be explained as including both "still image" and "moving image" unless otherwise specified. Captured image data generated by each imaging device 101 is transmitted to image processing device 102.

[0013] The image processing device 102 acquires data of multiple captured images (multi-viewpoint images) transmitted from multiple imaging devices 101, and estimates a three-dimensional field corresponding to a three-dimensional space including objects 107 and 108 present in an imaging space 106 using the acquired multi-viewpoint images. Information on the three-dimensional field estimated by the image processing device 102 is output to the storage device 104. The image processing device 102 also generates a virtual viewpoint image based on the estimated three-dimensional field and a set virtual camera path. The virtual camera path is data including information indicating the position of a virtual viewpoint and the line of sight direction at the virtual viewpoint (hereinafter referred to as the "direction of the virtual viewpoint") in time series. The virtual viewpoint image generated by the image processing device 102 is output to the storage device 104, the display device 105, or the like.

[0014] 1, the description will be given assuming that each of the multiple imaging devices 101 and the image processing device 102 are connected to one another, but the method of connection between the imaging devices 101 and the image processing device 102 is not limited to this. Specifically, for example, the multiple imaging devices 101 may be cascade-connected by connecting adjacent imaging devices 101 to one another, and at least one of the multiple imaging devices 101 may be connected to the image processing device 102.

[0015] The UI panel 103 includes a display device such as a liquid crystal panel, and displays a GUI (Graphical User Interface) on the display device to present information such as the imaging conditions of the imaging device 101 and the processing settings of the image processing device 102 to the user. The UI panel 103 may also include an input device such as a touch panel or buttons. In this case, the UI panel 103 receives instructions from the user regarding the setting or change of the above-mentioned imaging conditions or processing conditions via the input device. The UI panel 103 may also receive instructions from the user regarding the setting of a virtual viewpoint when a virtual viewpoint image is generated based on an estimated three-dimensional field. Note that the user does not necessarily have to input using the input device included in the UI panel; for example, the user may input using an input device such as a mouse or keyboard connected to the UI panel 103 or the image processing device 102.

[0016] The storage device 104 is configured with a hard disk drive or the like, and stores data of captured images obtained by synchronized imaging by each imaging device 101, and information on the estimated three-dimensional field related to the objects 107, 108 output from the image processing device 102. The storage device 104 may also store data of virtual viewpoint images output from the image processing device 102. The display device 105 is configured with a liquid crystal display or the like, and displays the estimated three-dimensional field and the virtual viewpoint image generated and output by the image processing device 102 based on the set virtual camera path.

[0017] <Hardware configuration of image processing device> 2 is a block diagram showing an example of the hardware configuration of the image processing device 102 according to embodiment 1. The image processing device 102 has, as its hardware configuration, a CPU 201, a main memory 202, a storage device 203, an input device 204, a display device 205, and an external I / F 206. The various units that the image processing device 102 has as its hardware configuration are connected to each other via a bus 207 so as to be able to communicate with each other.

[0018] The CPU 201 is an arithmetic processing unit that controls the image processing device 102 overall, and executes various programs stored in the storage device 203 or the like to perform various processes. The main memory 202 temporarily stores data, parameters, etc. used in the various processes, and is also used as a work area for the CPU 201. The storage device 203 is a large-capacity storage device that stores various programs and various data necessary for displaying a GUI (graphical user interface). The storage device 203 is configured, for example, by a non-volatile memory such as a hard disk drive or a silicon disk drive. Note that the processing of each step shown in the flowchart described below is realized when program code stored in the storage device 203 or the like is expanded in the main memory 202 and executed by the CPU 201.

[0019] The input device 204 is configured with a keyboard, mouse, electronic pen, touch panel, or the like, and accepts operation input from a user. The display device 205 is configured with a liquid crystal panel or the like, and displays a GUI, etc. The external I / F 206 is an interface for communicating with external devices such as the image capture devices 101. For example, the image processing device 102 and each image capture device 101 are connected via the external I / F 206 and a LAN (local area network) 208, and captured image data, control signal data, etc. are transmitted and received via the external I / F 206 and the LAN 208. The LAN 208 is not limited to a local area network, and may be configured with an SDI (Serial Digital Interface) or an HDMI (High-Definition Multimedia Interface (registered trademark)), etc.

[0020] Each imaging device 101 starts and stops imaging, changes settings of imaging conditions such as shutter speed or aperture, and outputs data of the captured image obtained by imaging, based on a control signal output from the image processing device 102. Note that the image processing device 102 may have various components in addition to the hardware configuration described above, but since this is not the main focus of this disclosure, a description of the other hardware configuration will be omitted.

[0021] Hereinafter, a learning method for a three-dimensional field model will be described, assuming that the estimation of the three-dimensional field is performed by learning a learning model (hereinafter referred to as a "three-dimensional field model") that models the three-dimensional field in the imaging space 106. In addition, in this embodiment, as an example, the three-dimensional field model is an NeRF configured by a multi-layer perceptron, and the three-dimensional field is described as being expressed by a radiance field, but the configuration of the three-dimensional field model and the three-dimensional field are not limited to this.

[0022] The method of representing a three-dimensional field varies depending on the learning content. Specifically, for example, the three-dimensional field model may be constructed using InstantNGP, a high-speed method similar to NeRF. Furthermore, the three-dimensional field model is not limited to one constructed using a multilayer perceptron, but may be constructed using Plenoxels or TensoRF (Tensorial Radiance Fields), which explicitly represent three-dimensional fields. Furthermore, the three-dimensional field model may be constructed using NeuS, which improves the accuracy of shape estimation by representing three-dimensional fields using SDF (Signed Distance Field). Furthermore, the three-dimensional field model may be constructed using various methods, such as 3D Gaussian Splatting, which represents three-dimensional fields using a set of points with a spread.

[0023] <Outline of 3D field estimation method> Fig. 3 is a diagram for explaining a method for estimating a three-dimensional field in the image processing device 102 according to the first embodiment. Specifically, Fig. 3(a) shows an example of a method for estimating a radiance field in a reference frame, and Fig. 3(b) shows an example of a method for estimating a radiance field in a new frame. An overview of the method for estimating a radiance field in the image processing device 102 will be described with reference to Fig. 3.

[0024] Here, the reference frame refers to a plurality of captured images (multi-view images) obtained by synchronized imaging at a reference time point (hereinafter referred to as the "reference time point") of each imaging device 101. For example, if the captured images are a moving image, the reference frame is a multi-view image composed of a plurality of frames obtained by synchronized imaging at the reference time point of each imaging device 101. The radiance field in the reference frame is a radiance field estimated at the reference time point, obtained as a result of learning using the reference frame. The new frame is a plurality of captured images (multi-view images) obtained by synchronized imaging at a time point (hereinafter referred to as the "new time point") different from the reference time point of each imaging device 101. For example, if the captured images are a moving image, the new frame is a multi-view image composed of a plurality of frames obtained by synchronized imaging at a new time point of each imaging device 101, similar to the reference frame. The radiance field in the new frame is a radiance field estimated at the new time point, obtained as a result of learning using at least the new frame.

[0025] An outline of a method for estimating a radiance field at a reference time point will be described with reference to FIG. 3(a). At a reference time point, the image processing device 102 sets learning spaces 301 and 302 for three-dimensional spaces that include the objects 107 and 108 in the imaging space 106. For example, the image processing device 102 first uses the reference frame to acquire position coordinates of the three-dimensional spaces in which the objects 107 and 108 exist, thereby identifying three-dimensional spaces that include the objects (hereinafter referred to as "object spaces"). Next, the image processing device 102 sets spaces that include each of the identified one or more object spaces as learning spaces 301 and 302. Details of a method for acquiring position coordinates of the three-dimensional spaces in which the objects 107 and 108 exist will be described later.

[0026] Next, the image processing device 102 assigns new pre-learning NeRFs 311 and 312 to the respective learning spaces 301 and 302. In the following description, it is assumed that NeRF 311 is assigned to learning space 301, and NeRF 312 is assigned to learning space 302. Next, the image processing device 102 uses the reference frame to train the NeRFs 311 and 312 assigned to the learning spaces 301 and 302. As a result of this training, a trained NeRF 311 is obtained as an estimation result of the radiance field corresponding to learning space 301, and a trained NeRF 312 is obtained as an estimation result of the radiance field corresponding to learning space 302.

[0027] An outline of a method for estimating a radiance field at a new time point will be described with reference to Fig. 3(b). At the new time point, the image processing device 102 determines whether or not each of the objects 107 and 108 remains stationary relative to the reference time point. In the following description, it is assumed that the object 108 included in the learning space 301 remains stationary, and the object 107 included in the learning space 302 does not remain stationary but has moved.

[0028] Next, the image processing device 102 assigns the trained NeRF 311 obtained as a result of training using the reference frame to the training space 301 containing the object 108 that remains stationary (hereinafter referred to as the "stationary object"). Meanwhile, the image processing device 102 sets a new training space 322 that contains the three-dimensional space in which the moving object 107 exists (hereinafter referred to as the "moving object"), and assigns the new NeRF 332 before training to the new training space 322. The image processing device 102 trains only the new NeRF 332 while fixing the weight parameters of the three-dimensional field model without retraining the trained NeRF 311. Through this training, the trained NeRF 332 is obtained as an estimation result of the radiance field corresponding to the training space 322 at the new time point.

[0029] In this way, the learning result of the NeRF assigned to the learning space including the stationary object at the reference time point, i.e., the estimation result of the radiance field corresponding to the learning space, is reused as the estimation result of the radiance field corresponding to the learning space at the new time point. Therefore, this learning method can reduce part of the learning process in estimating the radiance field at the new time point, and as a result, the amount of calculation required to estimate the radiance field at the new time point can be reduced.

[0030] <Functional configuration of image processing device> FIG. 4 is a block diagram showing an example of the functional configuration of the image processing device 102 according to the first embodiment. The image processing device 102 has, as its functional configuration, an imaging parameter acquisition unit 401, an image acquisition unit 402, a setting unit 403, a determination unit 404, a learning unit 405, a feature output unit 406, and a feature acquisition unit 407. In addition to the above-described functional configuration, the image processing device 102 also has a virtual camera parameter acquisition unit 408, a generation unit 409, and an image output unit 410. Each unit included in the functional configuration of the image processing device 102 is realized by the CPU 201 executing a program stored in the storage device 203 or the like, using the main memory 202 as a working area. Note that not all of the processes described below necessarily need to be realized by the CPU 201 executing a program. The image processing device 102 may be configured so that part or all of the processes are executed by one or more processing circuits other than the CPU 201. The image acquisition unit 402 acquires data of multi-view images obtained by synchronized imaging by the imaging devices 101. Using the multi-viewpoint images acquired by the image acquisition unit 402, NeRF, which is an example of a three-dimensional field model, is trained.

[0031] The imaging parameter acquisition unit 401 acquires imaging parameters of each imaging device 101. The imaging parameters include extrinsic parameters, intrinsic parameters, distortion parameters, etc. The extrinsic parameters are parameters that represent the position and orientation of the imaging device. The intrinsic parameters are parameters that represent the coordinates of the center of a captured image obtained by imaging with the imaging device and the focal length of a lens of the imaging device. The distortion parameters are parameters that indicate lens distortion. The imaging parameters of each imaging device 101 can be calculated from the results of camera calibration performed in advance. In the following description, the imaging parameters of each imaging device 101 are pre-stored in the storage device 203, and the imaging parameter acquisition unit 401 acquires the imaging parameters of each imaging device 101 by reading them from the storage device 203. Note that the imaging parameter acquisition unit 401 may calculate and acquire the imaging parameters of each imaging device 101 by performing camera calibration using multi-view images acquired by the image acquisition unit 402.

[0032] The setting unit 403 sets a learning space of a NeRF for each of the objects 107 and 108 based on the multi-view images acquired by the image acquisition unit 402. Furthermore, the setting unit 403 assigns features of a new NeRF or a trained NeRF to the learning space based on the determination result by the determination unit 404. The determination unit 404 determines whether each of the learning spaces set for the objects 107 and 108 by the setting unit 403 includes a still object, based on the multi-view images acquired by the image acquisition unit 402. For a learning space determined by the determination unit 404 to include a still object, the setting unit 403 assigns features of a trained NeRF. On the other hand, for a learning space determined not to include a still object, i.e., to include a moving object, the setting unit 403 assigns a new NeRF.

[0033] The learning unit 405 estimates a radiance field corresponding to a three-dimensional space including the objects 107 and 108 by learning a new NeRF assigned to the learning space by the setting unit 403. When a radiance field is estimated based on a reference frame, a trained NeRF does not yet exist. Therefore, after the learning unit 405 finishes learning based on the reference frame, the feature output unit 406 outputs the features of the trained NeRF to the storage device 104 or the like for storage.

[0034] Furthermore, when a radiance field is estimated based on a new frame, the features of the trained NeRF have already been stored in the storage device 104 or the like as the estimation result of the radiance field based on the reference frame. The feature acquisition unit 407 acquires the features of the trained NeRF stored in the storage device 104 or the like based on the determination result of the determination unit 404. The features of the trained NeRF acquired by the feature acquisition unit 407 are assigned to a training space including a stationary object by the setting unit 403. The training unit 405 estimates a radiance field corresponding to the training space by performing training on a new NeRF assigned to the training space using the new frame and the features of the trained NeRF assigned to the training space. The virtual camera parameter acquisition unit 408 acquires a virtual camera path. The generation unit 409 generates a virtual viewpoint image corresponding to the view from the virtual viewpoint based on the learning result by the learning unit 405 and the estimated radiance field obtained, i.e., the learned radiance field and the virtual camera path acquired by the virtual camera parameter acquisition unit 408. Specifically, when generating the virtual viewpoint image, volume rendering, which will be described later, is performed for each of a plurality of light rays from the virtual viewpoint. The virtual viewpoint image generated by the generation unit 409 is output to and displayed on the UI panel 103, the display device 105, or the like.

[0035] The generation unit 409 may output and store the feature amount calculated for each ray in volume rendering for generating a virtual viewpoint image corresponding to the reference frame to the storage device 104 or the like. In this case, the generation unit 409 may generate a virtual viewpoint image corresponding to a new frame using the feature amount stored in the storage device 104 or the like for a training space including a still object, based on the determination result of the determination unit 404. Note that, for a training space including a moving object, the generation unit 409 performs volume rendering using the learning result of a new NeRF based on the new frame by the learning unit 405, without using the feature amount stored in the storage device 104 or the like.

[0036] <Operation of image processing device> FIG. 5 is a flowchart showing an example of a processing flow of the image processing device 102 according to the first embodiment. The series of processing steps shown in the flowchart of FIG. 5 are realized by the CPU 201 reading a predetermined program from the storage device 203, expanding it in the main memory 202, and executing it. First, in S500, the virtual camera parameter acquisition unit 408 acquires a virtual camera path. Note that the virtual camera path acquisition process may be executed at any timing before the virtual viewpoint image generation process in S507, which will be described later. Next, in S501, the imaging parameter acquisition unit 401 acquires imaging parameters of each imaging device 101. In the following description, it is assumed that the imaging parameters of each imaging device 101 do not change over time while the image processing device 102 is operating. Note that the imaging parameter acquisition process may be executed at any timing before the learning space setting process in S503, which will be described later.

[0037] Next, in S502, the image acquisition unit 402 acquires data of a multi-viewpoint image (reference frame) obtained by synchronized imaging by each imaging device 101 at a reference time point. Specifically, the data of the reference frame output from the multiple imaging devices 101 is temporarily stored in the main memory 202 via the LAN 208, the external I / F 206, and the bus 207. Here, the reference time point is, for example, a time point corresponding to the start frame of a scene for which a virtual viewpoint image is to be generated. The reference time point is not limited to this, and may be, for example, a time point in a state where no moving object exists, before the time point corresponding to the start frame of a scene for which a virtual viewpoint image is to be generated.

[0038] ffdfdsfdffdfd Next, in S503, the setting unit 403 sets a three-dimensional space including the object as a learning space of NeRF based on the imaging parameters acquired in S501 and the reference frame acquired in S502. Specifically, the setting unit 403 identifies a three-dimensional space including the object for each object based on the imaging parameters and the reference frame, and sets a space including each of the three-dimensional spaces identified for each object as a learning space of NeRF.

[0039] For example, the setting unit 403 estimates the three-dimensional shape of each object based on the imaging parameters and the reference frame, and sets a rectangular parallelepiped circumscribing the three-dimensional shape of each estimated object as the learning space. In this case, for the rectangular parallelepiped circumscribing the three-dimensional shape, the size of the learning space may be set to be a predetermined size larger than the circumscribing shape, for example. Setting the learning space to be larger in this manner can reduce the possibility of so-called artifacts occurring. Examples of methods for estimating the three-dimensional shape of an object include the VH (Visual Hull) method. In the VH method, a region containing an image of the object is extracted as a silhouette region from each captured image constituting a multi-viewpoint image, and the three-dimensional shape of the object is obtained from the extracted silhouette region and the imaging parameters used when the captured image was captured. Methods for extracting the silhouette region of an object include a background subtraction method, which obtains the difference between a previously acquired background image and the captured image, and a method that performs segmentation processing on the captured image. Note that the imaging parameters have already been acquired by the imaging parameter acquisition unit 401 in S501. The setting unit 403 projects the silhouette area of ​​the object in each captured image onto a three-dimensional space based on the corresponding imaging parameters, and obtains the intersection of the projected areas as the three-dimensional shape of the object.

[0040] Specifically, the setting unit 403 first defines a three-dimensional space filled with voxels of a given size. Then, for all voxels in the three-dimensional space, the setting unit 403 projects each voxel from its three-dimensional coordinates onto each of the two-dimensional captured images that constitute the multi-viewpoint image. Then, the setting unit 403 determines whether each projected voxel overlaps with the silhouette region of an object in each captured image. Then, the setting unit 403 determines that a voxel that is determined to overlap with the silhouette region of an object in the number of captured images that is equal to or greater than a given threshold is a voxel that constitutes part of the three-dimensional shape of the object. For example, the setting unit 403 assigns "0" to the flags of all voxels as an initial value, indicating that the voxel is an OFF voxel. Then, the setting unit 403 changes the value of the flag of a voxel determined to constitute part of the three-dimensional shape of the object to "1," indicating that the voxel is an ON voxel. A group of voxels whose flag value is set to "1" (ON voxels) constitutes the voxels that make up the three-dimensional shape of the object.

[0041] In this embodiment, the VH algorithm is used to estimate the three-dimensional shape of an object, but the method for estimating the three-dimensional shape of an object is not necessarily limited to the VH algorithm. For example, the three-dimensional shape of an object may be estimated based on a small number of captured images obtained by capturing images with one or more image capture devices 101, using a trained model obtained as a result of learning through deep learning. Alternatively, the three-dimensional shape of an object may be estimated by identifying the position of the surface of the object in three-dimensional space as a point cloud using a distance measurement device such as LiDAR.

[0042] After S503, in S504, the setting unit 403 assigns a new NeRF to each of the learning spaces set for each object in S503. Next, in S505, the learning unit 405 trains the new NeRF assigned to each learning space in S504. Specifically, as described above in the background art, the learning unit 405 trains the new NeRF assigned to each learning space using multi-view images.

[0043] The general training method of NeRF will be described. Given an arbitrary position (x, y, z) in the training space and a line-of-sight direction (θ, φ) relative to that position, NeRF estimates the corresponding color c and volumetric density σ (volumetric scene density). Specifically, NeRF first sets a ray corresponding to the direction from the imaging position toward each pixel in the captured image. Next, multiple sampling points are set on the set ray. Next, the color c and volumetric density σ at each set sampling point are estimated. Next, the estimated color c and density σ at each sampling point on the same ray are integrated from the imaging position to determine the pixel value (pixel value) corresponding to each ray, thereby generating an image corresponding to the captured image. This type of image generation is generally called volume rendering. Next, the weight parameters of the neural network are updated to minimize the difference between the image generated by volume rendering and the captured image serving as the ground truth data corresponding to that image.

[0044] In this embodiment, since an NeRF is assigned to each learning space corresponding to each object, two or more learning spaces may exist in the imaging space, and thus two or more NeRFs may be assigned to the imaging space. When two or more NeRFs are assigned to the imaging space, the above-mentioned integration process is performed as many times as the number of learning spaces where the ray intersects to generate an image.

[0045] For example, when a ray corresponding to a pixel passes through the learning space 301 and the learning space 302 in order, multiple sampling points are generated in the learning space 301 and the learning space 302 by the NeRFs assigned to each of them. Next, the color and density at each sampling point in each of the learning spaces 301 and 302 are estimated by the NeRFs assigned to each of them. Next, volume rendering is performed by sequentially accumulating the colors and densities at each estimated sampling point in the learning space 301 and the learning space 302, thereby generating an image. The learning method and volume rendering method for each NeRF when two or more NeRFs are assigned to the imaging space are described in detail in the following reference 1. Note that this method is not the main focus of this disclosure, so a detailed description of this method will be omitted.

[0046] <Reference 1> Daniel Rebain1, 5 others, “DeRF:Decomposed Radiance Fields”, [online], November 25, 2020, arXiv, [searched on July 3, 2020], Internet<https: / / arxiv.org / pdf / 2011.12490.pdf>

[0047] After the learning process for each NeRF in S505 is completed, in S506, the feature output unit 406 outputs and stores the feature for each NeRF obtained as a result of the learning process in S505 to the storage device 104 or the like. Here, the condition for ending the learning process for each NeRF in S505 is, for example, when the difference between the captured image as the ground truth data and the image generated by volume rendering corresponding to the captured image becomes smaller than a given threshold. Note that the condition for ending is not limited to this, and may be, for example, when the number of supervised learnings using each captured image as the ground truth data reaches a given number, or when the learning process has been performed for a given period of time.

[0048] FIG. 6 is a diagram illustrating an example of NeRF features according to the first embodiment. The NeRF features include, for example, weight parameters for each NeRF 311, 312, as shown in FIG. 6( a). The NeRF features may be values ​​of color c and density σ estimated as a result of learning at each sampling point set in each learning space 301, 302, as shown in FIG. 6( b). Here, assuming that the image capture position is k, the pixel position in the captured image is (w, h), and an identifier such as a number that can uniquely identify the NeRF is r, the features represented by the color c and density σ can be expressed as c(k, r, w, h) and σ(k, r, w, h), respectively. By storing such features in the storage device 104 or the like, it is not necessary to derive features for a learning space containing a stationary object in the subsequent NeRF learning process.

[0049] Furthermore, the NeRF feature may be a value obtained by integrating the estimated color c and density σ at each sampling point set on the same light ray and in the same learning space from the imaging position, as shown in Fig. 6(c) as an example. The feature C expressed by the integrated value of the color and the integrated value of the density can be calculated using, for example, the following formulas (1) and (2).

[0050]

number

[0051]

number

[0052] where T i is the cumulative transmittance at each sampling point. As described above, k is the imaging position, (w, h) is the pixel position in the captured image, and r is the NeRF identifier. N is the total number of sampling points, and δ i is the distance from the i-th sampling point i to the next i+1-th sampling point i+1.

[0053] Furthermore, the integrated value of density in the NeRF feature may be expressed using an integrated value W of weights obtained by converting density into weights w. The integrated value W of weights can be calculated using, for example, the following formulas (3) and (4).

[0054]

number

[0055]

number

[0056] In addition, the NeRF features may include various features, such as the color c and density σ values ​​estimated at each sampling point, as well as the integrated values ​​of the color and density of each sampling point set on the same light ray and in the same learning space.

[0057] After S506, in S507, the generation unit 409 generates a virtual viewpoint image based on the trained NeRF corresponding to each learning space obtained as a result of the learning process in S505, i.e., the estimated radiance field, and the virtual camera path acquired in S500. The volume rendering technique described above in the description of S505 may be used to generate the virtual viewpoint image.

[0058] After S507, in S511, the image acquisition unit 402 acquires data of a multi-view image (new frame) obtained by synchronized imaging by the image capture devices 101 at a new time point. Specifically, data of the new frame output from the multiple image capture devices 101 is temporarily stored in the main memory 202 via the LAN 208, the external I / F 206, and the bus 207. Here, the new frame is a multi-view image obtained by synchronized imaging by the image capture devices 101 at a time point later than the reference time point, and is synchronously captured at a time point different from the reference frame.

[0059] Next, in S512, the setting unit 403 sets a three-dimensional space including the object as a NeRF learning space based on the imaging parameters acquired in S501 and the new frame acquired in S511. Specifically, the setting unit 403 identifies a three-dimensional space including the object for each object based on the imaging parameters and the new frame, and sets a space including each of the three-dimensional spaces identified for each object as a NeRF learning space. The process of setting the learning space based on the new frame in S512 is similar to the process of setting the learning space based on the reference frame in S503, and therefore detailed description thereof will be omitted.

[0060] Next, in S513, the determination unit 404 determines whether the object included in each learning space is a still object, for each learning space set to include each object. If it is determined in S513 that the object included in at least one learning space is a still object, the determination unit 404 executes the process of S514. In this case, in S514, the determination unit 404 outputs information indicating that the NeRF features stored in the storage device 104 or the like will be used for the learning space as the determination result for the learning space to the learning unit 405, the setting unit 403, and the feature acquisition unit 407. If it is determined in S513 that the objects included in all learning spaces are not still objects, i.e., that there are moving objects, the determination unit 404 executes the process of S515. In this case, in S515, the determination unit 404 outputs information indicating that the NeRF features stored in the storage device 104 or the like will not be used for the learning space, but that a new NeRF will be assigned to the learning space and learning will be performed for the learning space as the determination result for the learning space. Specifically, the determination unit 404 outputs the determination result of the learning space to the learning unit 405 and the setting unit 403 .

[0061] As a method for determining whether an object is a stationary object or not, for example, there is a method for determining based on the amount of movement of the three-dimensional shape of the object estimated by the VH method. FsIn addition, the vertices of the three-dimensional shape included in the learning space di set based on the new frame Fp are denoted by V Fp For example, the determination unit 404 calculates the amount of movement of a group of vertices of a three-dimensional shape included in the same learning space di from the reference time point to the new time point. Next, the determination unit 404 determines whether the calculated amount of movement is greater than a given threshold value V th If the calculated amount of movement is greater than the threshold V, the determining unit 404 determines that the learning space di does not include a stationary object, that is, that the learning space di includes a moving object. th If the following holds, the learning space di is determined to be a learning space including a stationary object. Note that the vertices of the three-dimensional shapes of the object in the reference frame Fs and the new frame Fp can be associated with each other by vertex tracking, nearest vertex search processing, or the like.

[0062]

number

[0063] Furthermore, if a three-dimensional shape included in the learning space di has a surface, the determination unit 404 may calculate the amount of movement of the surface in the same way as for the vertices, and use the calculated amount of movement of the surface to determine whether the learning space di is a learning space including a stationary object. Such amounts of movement are called inter-shape distances, and are generally called the Hausdorff distance or the Chamfer distance. Furthermore, the determination unit 404 may obtain the amount of movement of the three-dimensional shape using a general three-dimensional shape tracking method.

[0064] Furthermore, the determination unit 404 may use the position or shape of the silhouette region of the object in each captured image, which is used for estimating the three-dimensional shape in the VH method, for this determination. Specifically, the determination unit 404 first labels the silhouette region corresponding to the object included in each learning space in each captured image. Next, the determination unit 404 obtains the amount of movement of the labeled silhouette region in the captured image from the reference time point to the new time point, and determines the object corresponding to the silhouette region whose amount of movement is equal to or greater than a given threshold as a moving object. A method such as optical flow may be used to calculate the amount of movement. If a stationary object is predetermined in a certain scene, the user may tag the object and specify the learning space including the stationary object in advance before estimating the radiance field.

[0065] Note that if the number of learning spaces set by the setting unit 403 based on the reference frame and the new frame differs, the determination unit 404 executes the following process. Specifically, in this case, the determination unit 404 first associates one or more learning spaces set based on the reference frame with one or more learning spaces set based on the new frame. For example, the determination unit 404 associates learning spaces that are closest in position, shape, size, or the like. Next, the determination unit 404 determines, for each pair of associated learning spaces, whether the learning space contains a still object using the determination method described above. Note that, for a learning space for which no corresponding learning space exists, the determination unit 404 determines that the learning space contains, for example, a moving object.

[0066] After S514, in S516, the feature acquisition unit 407 acquires, from the storage device 104 or the like, features of the NeRF corresponding to the learning spaces determined in S513 to contain a still object based on the learning space determination result output in S514. Next, in S517, the setting unit 403 assigns the NeRF features acquired in S516, i.e., the trained NeRF, color and density values ​​at sampling points on the ray, or their integrated values, to the learning spaces determined in S513 to contain a still object. This is because the radiance field corresponding to the learning spaces containing a still object has already been estimated based on the reference frame, and there is no need to assign a new NeRF to the learning spaces containing a still object and perform learning again. Next, in S518, the setting unit 403 assigns a new NeRF to each of all learning spaces determined in S513 to contain no still objects, i.e., to contain a moving object.

[0067] On the other hand, after S515, in S519, setting unit 403 assigns a new NeRF to each of all learning spaces that are determined in S513 not to contain still objects, i.e., to contain moving objects, based on the learning space determination result output in S515. Note that the weight of each node at the start of learning of the new NeRF assigned in S504, S518, and S519 is assigned "0" or a random value generated by a random number generator or the like.

[0068] After S518 or S519, in S520, the training unit 405 performs training on the new NeRF assigned in S518 or S519. Details of the training process in the training unit 405 will be described later with reference to FIG. 7. As a result of the training, features of the trained NeRF corresponding to all the training spaces set in S512 are obtained. After the training process by the training unit 405 in S520, in S521, the generation unit 409 generates a virtual viewpoint image. Specifically, the generation unit 409 generates the virtual viewpoint image based on the features of the trained NeRF obtained as a result of the training process in S520, i.e., the estimated radiance field, and the virtual camera path acquired in S500. The volume rendering technique described above in the description of S505 may be used as a method for generating the virtual viewpoint image.

[0069] After S521, the image processing device 102 ends the processing of the flowchart shown in Fig. 5. Thereafter, the image processing device 102 repeatedly executes the processing from S511 to S521 shown in the flowchart of Fig. 5 every time a captured image that constitutes a new frame is output from each image capturing device 101. Furthermore, the image processing device 102 repeatedly executes the processing from S500 to S521 shown in the flowchart of Fig. 5 every time a captured image that constitutes a new base frame is output from each image capturing device 101. In this case, if there is no addition or change of the virtual camera bus, the image processing device 102 may omit the processing of S500. Furthermore, if there is no change in the imaging parameters of all image capturing devices 101, the image processing device 102 may omit the processing of S501.

[0070] In the present embodiment, the generation unit 409 generates a virtual viewpoint image based on the results of the learning process in S520 and the virtual camera path in S521. However, the method for generating the virtual viewpoint image in S521 is not limited to this. For example, the image processing device 102 may generate a virtual viewpoint image in S521 as follows. Specifically, the image processing device 102 first outputs feature values ​​for each learning space calculated for each ray in the virtual viewpoint image generation process in S507, i.e., pixel values ​​corresponding to each ray obtained by volume rendering, and stores the feature values ​​in the storage device 104 or the like. Next, in the virtual viewpoint image generation process in S521, the image processing device 102 first acquires feature values ​​stored in the storage device 104 or the like for a learning space determined to include a still object. Next, in the generation process, the image processing device 102 generates a virtual viewpoint image using the acquired feature values ​​and feature values ​​obtained by volume rendering a trained NeRF corresponding to a learning space including a moving object, i.e., pixel values.

[0071] <Learning process in the learning unit> 7 is a flowchart showing an example of the flow of learning processing in the learning unit 405 according to the first embodiment, and is a flowchart showing an example of the processing flow in S520. The flowchart shown in FIG. 7 is executed after S518 or S519. First, in S701, the learning unit 405 sets a plurality of rays emitted in a direction from the imaging position toward each pixel in the captured image. Next, in S702, the learning unit 405 selects an arbitrary ray from the plurality of rays set in S701.

[0072] Next, in S703, the learning unit 405 determines whether the ray selected in S702 (hereinafter referred to as the "selected ray") passes through each of the learning spaces set in S512. If it is determined in S703 that the selected ray does not pass through one or more learning spaces, the learning unit 405 executes the processing of S706, which will be described later. If it is determined in S703 that the selected ray passes through one or more learning spaces, the learning unit 405 determines, in this determination, which learning spaces the selected ray will pass through and in what order. Information regarding the identified learning spaces through which the selected ray will pass and the order of passage is temporarily stored, for example, in the main memory 202, as a result of the pass determination processing.

[0073] If it is determined in S703 that the selected ray passes through one or more learning spaces, the learning unit 405 executes the processing of S704. In this case, in S704, the learning unit 405 determines whether the selected ray passes through only learning spaces that include still objects, based on the result of the pass determination processing in S703 and the determination result of the learning spaces output from the determination unit 404 in S514 or S515. If it is determined in S704 that the selected ray passes through only learning spaces that include still objects, the learning unit 405 executes the processing of S706, which will be described later.

[0074] If it is determined in S704 that the selected ray does not pass only through a learning space containing still objects, i.e., that it passes through a learning space containing at least a moving object, the learning unit 450 executes the process of S705. In S705, the learning unit 450 performs learning on the new NeRF assigned to the learning space in S518 or S519. Here, if the selected ray passes through both a learning space containing still objects and a learning space containing moving objects, the learning unit 450 uses the feature amount assigned to the learning space in S517 to perform learning on the new NeRF assigned to the learning space in S518.

[0075] In this way, when estimating a radiance field based on a new frame, the radiance field estimation result based on the reference frame is reused for a training space including a stationary object, and only the NeRF assigned to the training space including a moving object is trained. Therefore, this type of training can reduce the amount of calculation required for training the NeRF when estimating a radiance field based on a new frame. Note that if it is determined in S704 that the selected ray passes only through a training space including a stationary object, the radiance field estimation result based on the reference frame is already reused for that training space, and therefore the processing of S705 is omitted. After S705, the training unit 405 executes the processing of S706.

[0076] In S706, the learning unit 405 determines whether all of the rays set in S701 have been selected in S702. If it is determined in S706 that at least some of all of the rays have not yet been selected, the learning unit 405 returns to the process of S702 and repeatedly executes the processes of S702 to S706 until it is determined in S706 that all of the rays have been selected. Note that in this repeated process, the learning unit 405 selects, for example, an arbitrary ray from one or more rays that have not yet been selected among all of the rays in S702. If it is determined in S706 that all of the rays have been selected, the learning unit 405 ends the process of the flowchart shown in FIG. 7, i.e., the process shown in S520 of FIG. 5.

[0077] Fig. 8 is a diagram for explaining an example of the learning process of the learning unit 405 according to the first embodiment, and is a diagram for explaining an example of the process of S706 shown in the flowchart of Fig. 7. With reference to Fig. 8, a case will be described in which, in NeRF learning based on a new frame, trained NeRF weight parameters obtained as a result of learning based on a reference frame are assigned to a learning space including a stationary object. In Fig. 8, learning space 301 includes object 108, which is a stationary object, and learning space 302 and learning space 322 include object 107, which is a moving object.

[0078] Fig. 8(a) shows trained NeRF311, which is the training result for training space 301, and trained NeRF312, which is the training result for training space 302, as an example of the results of the training process based on the reference frame in S505 shown in Fig. 5. Fig. 8(b) shows trained NeRF311 as a feature assigned to training space 301 in S517 in Fig. 5, and new NeRF332 assigned to training space 322 in S518. In Fig. 8, black circles indicate sampling points for which learning for color and density has been completed, and open circles indicate sampling points for which learning for color and density has not been performed.

[0079] In learning based on a new frame, first, in S518, the setting unit 403 assigns a new NeRF 332 to the learning space 322 containing a moving object. In addition, in S517, the setting unit 403 assigns a trained NeRF 311 obtained as a result of learning based on the reference frame to the learning space 301 containing a stationary object. Next, the colors and densities of the sampling points a, b, and c in the learning space 301 are calculated using the trained NeRF 311 obtained as a result of learning based on the reference frame. Next, the colors and densities of the sampling points d, e, and f in the learning space 322 are calculated using the new NeRF 322.

[0080] Next, the colors and densities calculated at sampling points a, b, c, d, e, and f are integrated to perform volume rendering, thereby calculating pixel values ​​(pixel values) corresponding to rays passing through these sampling points. Finally, while maintaining the weights of the trained NeRF 311, the error between the value of the pixel and the value of the pixel corresponding to the pixel in the captured image is fed back to the NeRF 332, thereby updating the weights of the NeRF 332. In this way, in training based on a new frame, the training unit 405 performs training using the weight parameters of the trained NeRF 311 obtained as a result of training based on the reference frame.

[0081] Next, a case will be described in which, in learning based on a new frame, the color and density values ​​of learned sampling points obtained as a result of learning based on a reference frame are assigned as features to a learning space including a still object. In learning based on a new frame, first, in S518, the setting unit 403 assigns a new NeRF 332 to the learning space 322 including a moving object. Also, in S518, the setting unit 403 assigns the learned color and density values ​​obtained as a result of learning based on the reference frame as features to the color and density of sampling points a, b, and c in the learning space 301 including a still object.

[0082] Next, the colors and densities of sampling points d, e, and f in the learning space 322 containing the moving object are calculated using the new NeRF 332. Next, the colors and densities of sampling points a, b, c, d, e, and f are integrated to perform volume rendering, thereby calculating pixel values ​​(pixel values) corresponding to rays passing through these sampling points. Finally, the error between the pixel value and the pixel value corresponding to the pixel in the captured image is fed back to the NeRF 332, thereby updating the weight parameters of the NeRF 332. In this way, the learning unit 405 can also perform learning based on a new frame by reusing the learned colors and densities obtained as a result of learning based on the reference frame.

[0083] Finally, a case will be described in which, in learning based on a new frame, the integrated values ​​of the learned colors and densities of sampling points obtained as a result of learning based on a reference frame are assigned as features to a learning space including a still object. In learning based on a new frame, first, in S518, the setting unit 403 assigns a new NeRF 332 to the learning space 322 including a moving object. Also, in S518, the setting unit 403 assigns the integrated values ​​of the learned colors and densities obtained as a result of learning based on the reference frame as the integrated values ​​of the colors and densities of sampling points a, b, and c in the learning space 301 including a still object.

[0084] Next, the color and density of sampling points d, e, and f in the learning space 322 containing the moving object are calculated using the new NeRF 332. Next, volume rendering is performed by integrating the color and density of sampling points d, e, and f while assigning the integrated value of the learned color and density obtained as a result of learning based on the reference frame as the integrated value of the color and density of sampling points a, b, and c. This volume rendering calculates the values ​​of pixels corresponding to rays passing through sampling points a, b, c, d, e, and f. Finally, the weights of NeRF 332 are updated by feeding back to NeRF 332 the error between the value of the pixel and the value of the pixel corresponding to the pixel in the captured image. In this way, the learning unit 405 can also use the integrated value of the learned color and density obtained as a result of learning based on the reference frame in learning based on the new frame.

[0085] In the above description, one type of feature is assigned to the learning space including the still object as the trained NeRF feature, but two or more types of feature may be assigned to the space. Specifically, for example, the setting unit 403 may assign, to the learning space including the still object, a feature indicating the trained color and density values ​​of each sampling point and a feature indicating the integrated value of the trained color and density.

[0086] For example, the relative positions of a still object and a moving object may change relative to the imaging position. When a still object is closer to the imaging position than a moving object, the learning unit 405 first refers to a feature indicating an integrated value of learned densities among the features assigned to the learning space including the still objects corresponding to each ray. When the integrated value of learned densities corresponding to a certain ray is equal to or greater than a given threshold, i.e., when a still object on the path taken by the ray is not transparent or translucent, the learning unit 405 omits learning for the ray in the learning space including the moving object. This is because the learning space including the moving object is blocked by the still object when viewed from the imaging position in the direction in which the ray travels.

[0087] Furthermore, if a moving object is closer to the imaging position than a still object, the learning unit 405 first calculates the integrated value of the density of the learning space including the moving object corresponding to each ray. If the integrated value corresponding to a certain ray is equal to or greater than a given threshold, i.e., if the moving object is not transparent or translucent along the path of the ray, the learning unit 405 performs volume rendering only on the learning space including the moving object. This is because, when viewed from the imaging position in the direction of the ray, the learning space including the still object is occluded by the moving object. Next, the learning unit 405 feeds back to NeRF332 the error between the pixel value calculated by the volume rendering and the pixel value corresponding to that pixel in the captured image.

[0088] On the other hand, if the integrated value of the density in the learning space containing the moving object corresponding to a certain light ray is less than the threshold, the learning unit 405 calculates the sum of the integrated value of the density in the learning space containing the moving object and the integrated value of the density in the learning space containing the still object. In this case, if this sum is less than a given threshold, the learning unit 405 calculates the integrated value of the color and density in the learning space containing the moving object, and calculates the sum of this integrated value and the integrated value of the color and density assigned to the learning space containing the still object. Next, the learning unit 405 feeds back to the NeRF 332 the error between this sum and the pixel value of the captured image.

[0089] If the sum of the integrated value of the density in the learning space including the moving object and the integrated value of the density in the learning space including the still object is equal to or greater than a threshold, the learning unit 405 executes the following process. In this case, the learning unit 405 first calculates the integrated value of the color and density in the learning space including the moving object. Next, the learning unit 405 integrates the color and density of the sampling points in the learning space including the still object with respect to the integrated value, in order of proximity to the learning space including the moving object, until the integrated value of the density is equal to or greater than a threshold. Next, the learning unit 405 feeds back to the NeRF 332 the error between the pixel value obtained by this integration and the pixel value of the captured image.

[0090] <Effects of image processing devices> As described above, in this embodiment, the image processing device 102 is configured to identify a learning space including a stationary object in a scene based on a reference frame and a new frame. Also, in this embodiment, in estimating a three-dimensional field based on a new frame, the image processing device 102 is configured to reuse the estimation result of the three-dimensional field estimated based on the reference frame for the learning space including the stationary object. According to the image processing device 102 configured as described above, in estimating a three-dimensional field based on a new frame, it is possible to reduce the amount of calculation required to learn a three-dimensional field model for the estimation.

[0091] [Embodiment 2] In the first embodiment, a static space in a scene is identified based on a reference frame and a new frame, and the learning results of NeRF based on the reference frame are used for the static space in estimating a three-dimensional field (radiance field) based on the new frame. In the second embodiment, instead of estimating a three-dimensional field using a three-dimensional field model such as NeRF as in the first embodiment, a three-dimensional field is estimated by grid-based learning as described in the following document 2. Note that the configurations of the imaging system and image processing device according to the second embodiment are the same as those of the first embodiment, and therefore, the same configurations will be described below using the reference numerals assigned to FIGS. 1, 2, or 4.

[0092] <Reference 2> Alex Yu, 5 others, “Plenoxels: Radiance Fields without Neural Networks”, [online], November 25, 2021, arXiv, [searched on July 3, 2020], Internet<https: / / arxiv.org / pdf / 2112.05131.pdf>

[0093] In grid-based 3D field learning, a 3D space is divided into equally spaced voxel grids, and a 3D field corresponding to the 3D space is reproduced by assigning features to each lattice point of the voxel grid. Here, the features assigned to each lattice point are, for example, values ​​related to color and density at the lattice point. Details of grid-based 3D field learning are described in the above-mentioned reference 2.

[0094] <Operation of image processing device> 9 is a flowchart showing an example of a processing flow of the image processing device 102 according to the second embodiment (hereinafter simply referred to as "image processing device 102"). A series of processing steps shown in the flowchart of FIG. 9 is realized by the CPU 201 reading a predetermined program from the storage device 203, loading it into the main memory 202, and executing it. In the following description, processing steps similar to those shown in the flowchart of FIG. 5 are denoted by the same reference numerals, and description thereof will be omitted.

[0095] First, the image processing device 102 executes the processes from S500 to S502. After S502, in S903, the setting unit 403 divides the imaging space 106 into equally spaced voxel grids. Next, in S904, the setting unit 403 sets, for each object, a space including the object among the multiple voxel grids divided in S903 based on the imaging parameters acquired in S501 and the reference frame acquired in S502, as a learning space. For example, the setting unit 403 estimates the three-dimensional shape of the object using the VH method or the like described above based on the imaging parameters and the reference frame, and sets, as the learning space, a voxel grid included in a space corresponding to a rectangular parallelepiped circumscribing the estimated three-dimensional shape of the object.

[0096] Next, in S905, the learning unit 405 uses the imaging parameters acquired in S501 and the reference frame acquired in S502 to learn the feature amounts of each lattice point of the voxel grid included in the learning space set in S904. In learning the feature amounts of the lattice points of the voxel grid, the difference between the pixel value obtained as a result of volume rendering and the pixel value of the captured image is fed back, similar to the learning in the learning unit 405 according to the first embodiment.

[0097] In grid-based 3D field learning, the procedure for volume rendering involves first setting rays corresponding to the direction from the imaging position to each pixel in the captured image. Next, multiple sampling points are set on each of the set rays, and the color and density of each sampling point are calculated using the colors and densities of lattice points of a voxel grid that are located near the sampling point. The feature quantities of the sampling points can be calculated, for example, by trilinearly interpolating the feature quantities of the lattice points corresponding to the eight vertices that make up the voxel containing the sampling point. The method for calculating the feature quantities of the sampling points is not limited to this. For example, the feature quantities related to the color of the sampling points can be calculated using the color feature quantities directly assigned to each lattice point, or by assigning coefficients of a spherical harmonic function to each lattice point and then inputting the trilinearly interpolated coefficients into the spherical harmonic function.

[0098] Next, the calculated color and density values ​​of each sampling point are integrated to generate an image by volume rendering. The learning unit 405 updates the feature amount of each grid point so that the difference between the generated image and the captured image as the correct data becomes small, thereby learning the feature amount of each grid point.

[0099] After the learning process of the feature amounts of each grid point in S905 is completed, the feature amount output unit 406 executes the process of S906. In S906, the feature amount output unit 406 outputs and stores, among the feature amounts of the grid points obtained as a result of the learning process in S905, the feature amounts of the grid points included in the learning space set in S904, to the storage device 104 or the like. Here, the condition for ending the learning process of the feature amounts of each grid point in S905 is, for example, when the difference between the captured image as the ground truth data and the image generated by volume rendering corresponding to the captured image becomes smaller than a given threshold. Note that the condition for ending the learning process is not limited thereto, and may be, for example, when the number of supervised learning processes using each captured image as the ground truth data reaches a given number, or when the learning process has been performed for a given period of time.

[0100] After S906, in S907, the generation unit 409 generates a virtual viewpoint image based on the feature values ​​of each trained lattice point corresponding to each learning space obtained as a result of the learning process in S905 and the virtual camera path acquired in S500. The volume rendering technique described above in the description of S905 may be used as a method for generating the virtual viewpoint image. After S907, the image processing device 102 executes the process of S511. After S511, in S912, the setting unit 403 sets a space including an object from among the multiple voxel grids divided in S903 as a learning space based on the imaging parameters acquired in S501 and the new frame acquired in S511. The process of S912 is similar to the process of setting the learning space based on the imaging parameters and the reference frame in S904, and therefore description thereof will be omitted. Note that the feature values ​​of each lattice point included in the learning space set in S904 and S912 are assigned an initial value of “0” or a random value generated by a random number generator or the like. After S912, the image processing apparatus 102 executes the process of S513.

[0101] If it is determined in S513 that the object included in the learning space is a still object, the determination unit 404 executes the processing of S914. In this case, in S914, the determination unit 404 outputs information indicating that the feature quantities of the lattice points stored in the storage device 104 or the like will be used for the learning space as the determination result for the learning space to the learning unit 405, the setting unit 403, and the feature acquisition unit 407. If it is determined in S513 that the object included in the learning space is not a still object, that is, that there is a moving object, the determination unit 404 executes the processing of S915. In this case, in S915, the determination unit 404 outputs information indicating that new learning will be performed on the feature quantities of the lattice points without using the feature quantities of the lattice points stored in the storage device 104 or the like for the learning space as the determination result for the learning space to the learning unit 405 and the setting unit 403.

[0102] After S914, in S916, the feature acquisition unit 407 acquires, from the storage device 104 or the like, the feature of each grid point included in the learning space determined in S513 to include a still object. Next, in S917, the setting unit 403 assigns the feature of each grid point acquired in S916, i.e., the color and density values ​​at each grid point, to the learning space determined in S513 to include a still object. This is because the feature of each grid point included in the learning space including a still object has already been learned based on the reference frame, and it is not necessary to newly learn the feature of each grid point included in the learning space including a still object.

[0103] After S917 or S915, in S920, the learning unit 405 learns the feature amounts of each lattice point included in the learning space set in S912. Details of the learning process by the learning unit 405 will be described later with reference to FIG. 10. As a result of this learning, learned feature amounts are obtained for each lattice point included in all of the learning spaces set in S512. After the learning process by the learning unit 405 in S920, in S921, the generation unit 409 generates a virtual viewpoint image. Specifically, the generation unit 409 generates the virtual viewpoint image based on the learned feature amounts of each lattice point obtained as a result of the learning process in S920, i.e., the estimated three-dimensional field, and the virtual camera path acquired in S500. The volume rendering technique described above in the description of S905 may be used as a method for generating the virtual viewpoint image.

[0104] After S921, the image processing device 102 ends the processing of the flowchart shown in Fig. 9. Thereafter, the image processing device 102 repeatedly executes the processing from S511 to S921 shown in the flowchart of Fig. 9 every time a captured image that constitutes a new frame is output from each image capturing device 101. Furthermore, the image processing device 102 repeatedly executes the processing from S500 to S921 shown in the flowchart of Fig. 9 every time a captured image that constitutes a new base frame is output from each image capturing device 101. In this case, if there is no addition or change of the virtual camera bus, the image processing device 102 may omit the processing of S500. Furthermore, if there is no change in the imaging parameters of all image capturing devices 101, the image processing device 102 may omit the processing of S501.

[0105] <Learning process in the learning unit> FIG. 10 is a flowchart showing an example of the flow of learning processing in the learning unit 405 according to the second embodiment, and is a flowchart showing an example of the processing flow in S920. The flowchart shown in FIG. 10 is executed after S917 or S915. Note that in the following description, processing steps similar to processing steps shown in the flowchart of FIG. 7 are denoted by the same reference numerals, and description thereof will be omitted. First, the learning unit 405 executes the processing of S701 to S703. If it is determined in S703 that the selected ray does not pass through one or more learning spaces, the learning unit 405 executes the processing of S706. If it is determined in S703 that the selected ray passes through one or more learning spaces, the learning unit 405, in this determination, identifies which learning spaces and in what order the selected ray will pass through. Information regarding the identified learning spaces through which the selected ray will pass and the order of passage is temporarily stored, for example, in the main memory 202, as a result of the pass determination processing.

[0106] If it is determined in S703 that the selected ray passes through one or more learning spaces, the learning unit 405 executes the processing of S704. If it is determined in S704 that the selected ray passes only through learning spaces containing still objects, the learning unit 405 executes the processing of S706. If it is determined in S704 that the selected ray does not pass only through learning spaces containing still objects, i.e., that the selected ray passes through learning spaces containing at least moving objects, the learning unit 450 executes the processing of S1005. In S1005, the learning unit 450 learns the feature points of each lattice point included in the learning space determined in S513 to include a moving object, out of the lattice points included in the learning space set in S512. Here, if the selected ray passes through both a learning space containing a still object and a learning space containing a moving object, the learning unit 450 uses the feature amount of each lattice point assigned in S917 in the learning.

[0107] In this case, the learning unit 450 first calculates the color and density of sampling points in the learning space determined in S513 to include a still object, using the trained feature amounts of each lattice point included in the learning space obtained as a result of learning based on the reference frame. Next, the learning unit 450 calculates the color and density of sampling points in the learning space determined in S513 to include a moving object, using the feature amounts of the lattice points included in the learning space. Next, the learning unit 450 performs volume rendering by integrating the calculated color and density at each sampling point, and calculates the value of the pixel corresponding to the selected ray (pixel value).

[0108] Next, the learning unit 450 feeds back the error between the value of the pixel obtained by volume rendering (pixel value) and the value of the pixel corresponding to that pixel in the captured image (pixel value) to the learning space determined in S513 to include a moving object. By performing this feedback, the learning unit 450 fixes the feature amount of each lattice point included in the learning space determined in S513 to include a still object, while updating the feature amount of each lattice point included in the learning space determined in S513 to include a moving object. After S1005, the learning unit 405 executes the process of S706.

[0109] If it is determined in S706 that at least some of all the rays have not yet been selected, the learning unit 405 returns to the process of S702 and repeatedly executes the processes of S702 to S706 until it is determined in S706 that all the rays have been selected. Note that in this repeated process, the learning unit 405 selects, for example, an arbitrary ray from one or more rays that have not yet been selected among all the rays in S702. If it is determined in S706 that all the rays have been selected, the learning unit 405 ends the process of the flowchart shown in FIG. 10, i.e., the process shown in S920 in FIG. 9.

[0110] <Effects of image processing devices> As described above, in this embodiment, the image processing device 102 is configured to identify a learning space containing a still object in a scene based on the reference frame and a new frame. Furthermore, in estimating a three-dimensional field based on a new frame, the image processing device 102 is configured to reuse, for the learning space containing a still object, the feature quantities of the grid points obtained as a result of learning based on the reference frame, i.e., the estimation result of the three-dimensional field based on the reference frame. According to the image processing device 102 configured as described above, in estimating a three-dimensional field based on a new frame, it is possible to reduce the amount of calculation required for learning a three-dimensional field model for the estimation.

[0111] [Other embodiments] The present disclosure can also be realized by a process in which a program that realizes one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (such as an ASIC) that realizes one or more functions.

[0112] It should be noted that within the scope of the present disclosure, the embodiments may be freely combined, any component of the embodiments may be modified, or any component of the embodiments may be omitted.

[0113] [Configuration of the present disclosure] The present disclosure includes the following configurations, methods, and programs.

[0114] <Configuration 1> an acquisition means for acquiring a plurality of captured images obtained by synchronously capturing images of an imaging space from a plurality of directions; a setting means for setting, for each object present in the image capture space, a three-dimensional space including the object based on the plurality of captured images as a learning space; a learning means for learning a three-dimensional field corresponding to each of the set learning spaces based on the plurality of captured images; and when the learning means learns the three-dimensional field corresponding to the learning space based on the plurality of captured images obtained by synchronized imaging at a certain point in time, for a learning space in which the object included in the learning space is a stationary object, the learning means uses, as the feature amount of the three-dimensional field corresponding to the learning space, the feature amount of the three-dimensional field that has already been obtained as a result of learning based on the plurality of captured images obtained by synchronized imaging at another point in time, to learn the feature amount of the three-dimensional field corresponding to the learning space in which a moving object is included; An image processing device characterized by:

[0115] <Configuration 2> the feature quantities of the three-dimensional field include values ​​indicating color and density corresponding to positions and directions in the learning space; 2. The image processing device according to configuration 1,

[0116] <Configuration 3> the feature amount of the three-dimensional field includes a value indicating transparency or opacity corresponding to a position and a direction in the learning space; 3. The image processing device according to configuration 2,

[0117] <Configuration 4> the feature amount of the three-dimensional field includes a network parameter of a learning model related to the three-dimensional field corresponding to the learning space; 4. The image processing device according to any one of configurations 1 to 3, characterized in that:

[0118] <Configuration 5> the feature amount of the three-dimensional field includes an integrated value obtained by volume rendering the three-dimensional field corresponding to the learning space on a predetermined ray; 5. The image processing device according to any one of configurations 1 to 4,

[0119] <Configuration 6> the learning means estimates the three-dimensional field corresponding to the learning space by learning at least one of a learning model assigned to each of the learning spaces, feature amounts of lattice points included in each of the learning spaces, and functions assigned to lattice points included in each of the learning spaces; 6. The image processing device according to any one of configurations 1 to 5, characterized in that:

[0120] <Configuration 7> the setting means sets the learning space for each of the objects based on a position of the object in the imaging space; 7. The image processing device according to any one of configurations 1 to 6,

[0121] <Configuration 8> a determination means for determining whether the stationary object is included in the learning space; and the learning means learns the feature quantity of the three-dimensional field corresponding to the learning space based on the result of the determination; 8. The image processing device according to any one of configurations 1 to 7, characterized in that:

[0122] <Configuration 9> the determining means makes the determination based on optical flows in the plurality of captured images; 9. The image processing device according to configuration 8,

[0123] <Configuration 10> the determination means makes the determination based on a change in the three-dimensional shape of the object acquired based on the plurality of captured images; 10. The image processing device according to configuration 8 or 9,

[0124] <Configuration 11> the three-dimensional field being a radiance field; 11. The image processing device according to any one of configurations 1 to 10, characterized in that:

[0125] <Configuration 12> a generating means for generating an image corresponding to an appearance from an arbitrary virtual viewpoint based on the result of the learning and the obtained three-dimensional field corresponding to the learning space; further comprising: 12. The image processing device according to any one of configurations 1 to 11,

[0126] <Method> an acquisition step of acquiring a plurality of captured images obtained by synchronously capturing images of an imaging space from a plurality of directions; a setting step of setting, for each object present in the image capture space, a three-dimensional space including the object based on the plurality of captured images as a learning space; a learning step of learning a three-dimensional field corresponding to each of the set learning spaces based on the plurality of captured images; Including, In the learning step, when learning the three-dimensional field corresponding to the learning space based on the plurality of captured images obtained by synchronized imaging at a certain point in time, for the learning space in which the object included in the learning space is a stationary object, feature quantities of the three-dimensional field that have already been obtained as a result of learning based on the plurality of captured images obtained by synchronized imaging at another point in time are used as feature quantities of the three-dimensional field corresponding to the learning space in which a moving object is included, to learn feature quantities of the three-dimensional field corresponding to the learning space; An image processing method comprising:

[0127] <Program> 13. A program for causing a computer to function as the image processing device according to any one of configurations 1 to 12. [Explanation of symbols]

[0128] 102 Image processing device 402 Image acquisition unit 403 Settings 405 Learning Department

Claims

1. an acquisition means for acquiring a plurality of captured images obtained by synchronously capturing images of an imaging space from a plurality of directions; a setting means for setting, for each object present in the image capture space, a three-dimensional space including the object based on the plurality of captured images as a learning space; a learning means for learning a three-dimensional field corresponding to each of the set learning spaces based on the plurality of captured images; and when the learning means learns the three-dimensional field corresponding to the learning space based on the plurality of captured images obtained by synchronized imaging at a certain point in time, for a learning space in which the object included in the learning space is a stationary object, the learning means uses, as the feature amount of the three-dimensional field corresponding to the learning space, the feature amount of the three-dimensional field that has already been obtained as a result of learning based on the plurality of captured images obtained by synchronized imaging at another point in time, to learn the feature amount of the three-dimensional field corresponding to the learning space in which a moving object is included; An image processing device characterized by:

2. the feature quantities of the three-dimensional field include values ​​indicating color and density corresponding to positions and directions in the learning space; 2. The image processing device according to claim 1, wherein:

3. the feature amount of the three-dimensional field includes a value indicating transparency or opacity corresponding to a position and a direction in the learning space; 3. The image processing device according to claim 2, wherein:

4. the feature amount of the three-dimensional field includes a network parameter of a learning model related to the three-dimensional field corresponding to the learning space; 2. The image processing device according to claim 1, wherein:

5. the feature amount of the three-dimensional field includes an integrated value obtained by volume rendering the three-dimensional field corresponding to the learning space on a predetermined ray; 2. The image processing device according to claim 1, wherein:

6. the learning means estimates the three-dimensional field corresponding to the learning space by learning at least one of a learning model assigned to each of the learning spaces, feature amounts of lattice points included in each of the learning spaces, and functions assigned to lattice points included in each of the learning spaces; 2. The image processing device according to claim 1, wherein:

7. the setting means sets the learning space for each of the objects based on a position of the object in the imaging space; 2. The image processing device according to claim 1, wherein:

8. a determination means for determining whether the stationary object is included in the learning space; and the learning means learns the feature quantity of the three-dimensional field corresponding to the learning space based on the result of the determination; 2. The image processing device according to claim 1, wherein:

9. the determining means makes the determination based on optical flows in the plurality of captured images; The image processing device according to claim 8 ,

10. the determination means makes the determination based on a change in the three-dimensional shape of the object acquired based on the plurality of captured images; The image processing device according to claim 8 ,

11. the three-dimensional field being a radiance field; 2. The image processing device according to claim 1, wherein:

12. a generating means for generating an image corresponding to an appearance from an arbitrary virtual viewpoint based on the result of the learning and the obtained three-dimensional field corresponding to the learning space; further comprising:

2. The image processing device according to claim 1, wherein:

13. an acquisition step of acquiring a plurality of captured images obtained by synchronously capturing images of an imaging space from a plurality of directions; a setting step of setting, for each object present in the image capture space, a three-dimensional space including the object based on the plurality of captured images as a learning space; a learning step of learning a three-dimensional field corresponding to each of the set learning spaces based on the plurality of captured images; Including, In the learning step, when learning the three-dimensional field corresponding to the learning space based on the plurality of captured images obtained by synchronized imaging at a certain point in time, for the learning space in which the object included in the learning space is a stationary object, feature quantities of the three-dimensional field that have already been obtained as a result of learning based on the plurality of captured images obtained by synchronized imaging at another point in time are used as feature quantities of the three-dimensional field corresponding to the learning space in which a moving object is included, to learn feature quantities of the three-dimensional field corresponding to the learning space; An image processing method comprising:

14. A program for causing a computer to function as the image processing device according to any one of claims 1 to 12.

Images