Image processing apparatus, image processing method and program

Abstract

To perform learning for generating an image or the like corresponding to a virtual viewpoint from multi-viewpoint images at a higher speed.SOLUTION: Shape data representing a three-dimensional shape of an object is generated on the basis of a plurality of captured images obtained by a plurality of image capturing devices. Then, a learning area is set for each object based on the generated shape data, and a three-dimensional field in accordance with a captured image is learned by taking the learning area set for each object as a target.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present disclosure relates to an image processing technique for generating data corresponding to a virtual viewpoint from multi-viewpoint images.

Background Art

[0002] There is a technique called NeRF (Neural Radiance Fields) that takes as input multi-viewpoint images with known camera parameters and generates an image corresponding to an arbitrary virtual viewpoint (Non-Patent Document 1, Patent Document 1). This NeRF is a neural network that outputs a volume density σ and an emitted radiance from a five-dimensional input variable {arbitrary spatial position coordinates (x, y, z) and direction (θ, φ)}. In the learning of this neural network, the pixel values of the multi-viewpoint images are used as the teacher, and the difference between the pixel values of the rendering result is used as the loss. Therefore, in the learning process, rendering, loss calculation, and error backpropagation are performed for the number of images included in the multi-viewpoint images, and this is repeated, so a lot of time is required for learning. For example, at least 12 hours or more are required to learn a scene with 100-view 1K images as the teacher. To address the problem of requiring a lot of time for learning, techniques such as a method called VaxNeRF that aims to speed up by using the volume intersection method have been proposed (see Non-Patent Document 2). The volume intersection method is a method of obtaining the three-dimensional shape of an object by back-projecting the silhouette of the object extracted from multi-viewpoint images into three-dimensional space to form cones from each viewpoint and obtaining the intersection part of each cone. This volume intersection method has the feature that it is guaranteed that no object exists outside the obtained three-dimensional shape under the assumption that the extracted silhouette is correct. Utilizing this feature, in VaxNeRF, pixels located outside the silhouette are not used for learning, and the sampling points during rendering are limited to the inside of the three-dimensional shape obtained by the volume intersection method, thereby suppressing the computational amount during learning and speeding up the learning.

Prior Art Documents

Patent Documents

[0003] [Patent Document 1] U.S. Patent No. 11,308,659 [Non-Patent Document]

[0004] [Non-Patent Document 1] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. 3 Aug 2020. [Non-Patent Document 2] Naruya Kondo, Yuya Ikeda, Andrea Tagliasacchi, Yutaka Matsuo, Yoichi Ochiai, Shixiang Shane Gu. VaxNeRF: Revisiting the Classic for Voxel-Accelerated Neural Radiance Field. Nov 25 2021. [Non-Patent Document 3] Alex Yu, Sara Fridovich-Keil, Matthew Tancik, Qinhong Chen, Benjamin Recht, Angjoo Kanazawa. Plenoxels: Radiance Fields without Neural Networks. 9 Dec 2021. [Summary of the Invention] [Problems to be Solved by the Invention]

[0005] For example, in the case of a scene where many objects are sparsely distributed in a vast shooting area like a soccer game, even with the VaxNeRF method, it still takes an enormous amount of time for learning. This disclosure is made in view of the above-described problems, and aims to perform learning for generating an image corresponding to a virtual viewpoint from multi-viewpoint images at a higher speed.

Means for Solving the Problems

[0006] An image processing apparatus according to the present disclosure is an image processing apparatus that performs learning for obtaining virtual viewpoint data corresponding to a virtual viewpoint or a three-dimensional shape from a plurality of captured images obtained by a plurality of imaging devices, and includes an acquisition unit that acquires the plurality of captured images, a generation unit that generates schematic shape data representing the three-dimensional shape of an object based on the plurality of captured images acquired by the acquisition unit, a setting unit that sets a learning region for each object based on the schematic shape data generated by the generation unit, and a learning unit that learns a three-dimensional field corresponding to the captured image for the learning region set for each object by the setting unit. It is characterized by comprising the above.

Advantages of the Invention

[0007] According to the present disclosure, it becomes possible to perform learning for generating an image corresponding to a virtual viewpoint from multi-viewpoint images at a higher speed.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10A

Figure 10B

Figure 10C

Figure 10D

Figure 11

Figure 12

Figure 13

Figure 14A

Figure 14B

Figure 14C

Figure 14D

Embodiments for Carrying Out the Invention

[0009] Hereinafter, this embodiment will be described with reference to the drawings. Note that the following embodiments do not necessarily limit the present invention. Also, not all combinations of features described in this embodiment are essential for the solution means of the present invention. Further, as a common concept in each embodiment, in the present disclosure, a rough three-dimensional shape (approximate shape) of an object is used in the same manner as in Non-Patent Document 2. Then, a three-dimensional field (this "field" varies depending on the learning content. Hereinafter, it is referred to as "three-dimensional field" in this specification) in the imaging space to be learned is defined and learned independently for each object. Thereby, the amount of information held by each three-dimensional field is reduced, the convergence of learning for each three-dimensional field is accelerated, and high-speed learning is realized.

[0010] [Embodiment 1] <Configuration of Image Processing System> FIG. 1 is a diagram showing a configuration example of an image processing system that generates virtual viewpoint images according to the present embodiment. The image processing system includes a plurality of imaging devices (cameras) 101, an image processing device 102, a user interface (UI) panel 103, a storage device 104, and a display device 105. The plurality of cameras 101 synchronously capture an object 107 inside a shooting area 106 from multiple viewpoints according to shooting conditions, and acquire a plurality of captured image (multi-viewpoint image) data corresponding to each viewpoint. The captured images obtained by the cameras 101 may be still images, moving images, or both still images and moving images. In the present embodiment, unless otherwise specified, the term "image" includes both still images and moving images. The image processing device 102 generates three-dimensional shape data (3D model) of the object 107 based on the control of the plurality of cameras 101 and the plurality of captured images acquired from the plurality of cameras 101. The UI panel 103 is a display device such as a liquid crystal display, and functions as a user interface for conveying the current shooting conditions and processing settings to the user. Further, the UI panel 103 may be provided with an input device such as a touch panel or buttons, and can receive instructions from the user regarding the shooting conditions and processing settings. Note that the input device may be provided separately from the UI panel 103, such as a mouse or a keyboard. The storage device 104 stores the three-dimensional shape data of the object acquired from the image processing device 102. The display device acquires and displays the three-dimensional shape data of the object from the image processing device 102. The shooting area 106 is a space (three-dimensional space) surrounded by the plurality of cameras 101 installed in the studio, and the frame indicated by the solid line shows the contours of the shooting area 106 in the front-rear direction and the left-right direction on the floor surface.

[0011] <Hardware Configuration of Image Processing Device> FIG. 2 is a diagram showing a hardware configuration example of the image processing apparatus 102. The image processing apparatus 102 includes a CPU 201, a RAM 202, a ROM 203, a storage unit 204, a control interface (I / F) 205, an input interface (I / F) 206, an output interface (I / F) 207, and a main bus 208. The CPU 201 is a processor that comprehensively controls each part of the image processing apparatus 102. The RAM 202 functions as a main memory, a work area, etc. of the CPU 201. The ROM 203 stores a group of programs executed by the CPU 201. The storage unit 204 stores applications executed by the CPU 201, data used for image processing, etc. The control (I / F) 205 is connected to a plurality of cameras 101 and is an interface for setting shooting conditions and performing controls such as shooting start and stop. The input I / F 206 is a serial bus interface such as SDI or HDMI (registered trademark), and multi-viewpoint image data is acquired from a plurality of cameras 101 via the input I / F 206. The output I / F 207 is a serial bus interface such as USB or IEEE 1394, and the subject shape is output to the storage device 104 or the display device 105 via the output I / F 207. The main bus 208 is a transmission path connecting each module of the image processing apparatus 102.

[0012] In the present embodiment, one or a plurality of objects are photographed from the surroundings using eight cameras installed in a studio. Also, it is assumed that camera parameters such as internal parameters, external parameters, and distortion parameters of the cameras are stored in the storage unit 204. The internal parameters represent the coordinates of the image center and the lens focal length, and the external parameters represent the position and orientation of the camera. Note that the camera parameters of a plurality of cameras do not all have to be common, and for example, the angle of view may be different.

[0013] <Operation of the Image Processing Apparatus> Next, the operation of the image processing apparatus 120 according to the present embodiment will be described. FIG. 3(a) is a block diagram showing the functional configuration of the image processing apparatus 102 in the learning phase, and FIG. 4 is a flowchart showing the operation flow of the image processing apparatus 102. As shown in FIG. 3, the image processing apparatus 102 includes a learning unit 300, an image input unit 301, a schematic shape generation unit 302, and a learning area setting unit 303. The learning unit 300 includes a drawing unit 306, a three-dimensional field storage unit 305, and a drawing unit 306. Hereinafter, the operations of the respective units included in the image processing apparatus 120 will be described along the flowchart of FIG. 4. In the following description, the symbol "S" means step. When the input captured image is a moving image, it will be executed in units of frames.

[0014] In S401, the image input unit 301 receives and acquires a multi-viewpoint image from a plurality of cameras 101 via the input I / F 206. Alternatively, the data of the multi-viewpoint image stored in the storage unit 204 may be read and acquired. The acquired multi-viewpoint image is held in the RAM 202.

[0015] In S402, the outline shape generating unit 302 generates outline shape data representing a rough three-dimensional shape of an object captured in the acquired multi-view image. In this embodiment, the shape of the object is derived by a volume intersection method as the outline shape data, and three-dimensional shape data expressed as a voxel set is generated. In the volume intersection method, first, for each captured image constituting the multi-view image, foreground / background separation is performed using an image (background image) in which the object is not captured, and an image (silhouette image) representing the silhouette of the object is obtained. The background image is prepared, for example, by capturing an image of the shooting area 106 in which there is no object in advance. Next, based on the camera parameters of each of the multiple cameras 101, each voxel included in the voxel set corresponding to the shooting area 106 is projected onto each silhouette image of the object captured from the multi-view image. Then, a voxel set consisting of only voxels projected into the silhouette in all silhouette images is regarded as the outline shape of the object. FIG. 5 is a two-dimensional view of the shooting area 106 from above. In Fig. 5, a rectangle 501 outside the dashed line indicates the wall of the studio where the camera 101 is installed. Moreover, a rectangle 502 inside the dashed line indicates the range (capture area) where an object may exist, and solid-line triangles 506-513 indicate the positions, directions, and angles of view of each of the eight cameras 101. Fig. 6 shows how a rough shape 601 is obtained by the volume intersection method for an object 600 with a concave center.

[0016] In S403, the learning area setting unit 303 sets, for each object, a rectangular parallelepiped circumscribing the outline shape of the object as a target three-dimensional area (learning area) for learning, based on the outline shape data for each object generated in S402. The shape of this learning area may be a solid that contains the outline shape of the object, for example, a sphere or an oval sphere. Alternatively, a three-dimensional area of a coarser outline shape made up of a voxel set made up of voxels larger than the voxels that are elements that make up the outline shape data may be set as the learning area.

[0017] In S404, the three-dimensional field update unit 304 secures a memory area in the RAM 202 as the three-dimensional field storage unit 305, corresponding to the learning area set for each object in S403. Now, assuming that the "three-dimensional field" is the radiance field in NeRF (a vector field that associates volume density (≈ opacity) and radiance (≈ color) with each coordinate in space), the following steps will be described. When the three-dimensional field is the radiance field of NeRF, the value representing the volume density (hereinafter simply referred to as "density") at an arbitrary position in space and the value representing the anisotropic color that varies for each direction are stored in the secured memory area.

[0018] In S405, the rendering unit 306 renders (volume rendering) an image corresponding to each shooting viewpoint having the same viewing angle as each shooting image, based on the camera parameters corresponding to each shooting image constituting the multi-viewpoint image and the radiance field stored in the three-dimensional field storage unit 305. Specifically, a process of obtaining the pixel value C(r) of each pixel corresponding to the ray r viewed from the same viewpoint as the shooting image is performed, for example, using the following formula (1).

[0019]

Equation

[0020] In S406, the three-dimensional field update unit 304 performs a process of obtaining the color difference between the captured image and the drawing image that are in a correspondence relationship with each other for the learning region of interest among the learning regions set in S403, and updating the radiance field so that the color difference becomes smaller. This step corresponds to the error calculation and error backpropagation in deep learning. In this embodiment, the color difference between the two images is calculated for each corresponding pixel, and the value is defined by the Euclidean distance squared of the color (RGB).

[0021] In S407, it is determined whether the update process of the radiance field has been completed for all the learning regions set in S403. If there is a learning region for which the update process of the radiance field has not been completed, the process returns to S405, the next learning region of interest is determined, and the same process is executed. On the other hand, if the update process of the radiance field has been completed for all the learning regions, the process proceeds to S408.

[0022] In S408, it is determined whether the update has sufficiently converged for all radiance fields. Whether it has converged is determined, for example, by summing the errors calculated for each pixel at all viewpoints and determining that convergence has occurred when the decrease rate relative to the previously calculated total value is less than a threshold value (e.g., 0.1%). Alternatively, it may be determined that convergence has occurred when the error calculated for each pixel is less than a predetermined threshold value, or it may be determined that convergence has occurred when the number of update processes for the radiance field is counted and a predetermined number of times is reached. Further, for example, some of a plurality of cameras may be used as evaluation cameras that are not used for updating the radiance field, and it may be determined that convergence has occurred when the error with the captured image of the camera starts to increase (overfitting occurs). Furthermore, determination may be made using a combination of these. If the update has converged, this process ends, and if the update has not converged, the process returns to S405 and the same process is repeated.

[0023] The above is the flow of operations in the image processing apparatus 120 according to the present embodiment. FIG. 3(b) is a block diagram showing the functional configuration of the image processing apparatus 102' in the case of generating a virtual viewpoint image using the radiance field determined to have converged (inference phase). As shown in FIG. 3(b), the image processing apparatus 102' responsible for the inference phase has an inference unit 310 composed of the aforementioned three-dimensional field storage unit 305 and rendering unit 306. In the rendering unit 306 of the inference unit 310, the camera parameters of the virtual viewpoint are input instead of the camera parameters of the shooting viewpoint of the imaging apparatus 101. Then, in the rendering unit 306, volume rendering is performed according to the parameters of the virtual viewpoint using the radiance field after convergence held in the three-dimensional field storage unit 305, and a rendered image (virtual viewpoint image) corresponding to the virtual viewpoint is generated. Note that in the flow of FIG. 4 described above, both the processes of S405 and S406 are repeated in units of learning regions, but it is not limited to this. For example, after performing the process of S405 for all learning regions to obtain a rendered image, the process of S406 may be performed collectively to update the radiance field corresponding to each learning region. Also, in the case of a moving image, object tracking may be performed between frames and the setting of the learning region (S403) may be performed based on the result, and the generation of the approximate shape by the volume intersection method may be omitted for some frames. Thereby, a series of processes in the case of a moving image can be performed more efficiently.

[0024] <Difference from the prior art> Here, the difference between NeRF and VaxNeRF and the method of the present embodiment will be described by taking the case where three objects exist in the capture area 502 (see FIG. 7) as an example.

[0025] ≪In the case of NeRF≫ (a) and (b) of FIG. 8 are diagrams for explaining learning when NeRF is applied. In the case of NeRF that defines the update range (learning area) of the radiance field during learning, the entire capture area 502 is set as the update range of the radiance field. In FIG. 8(a), the small dot area 801 represents the learning area set as the update range of the radiance field, and currently, it is equal to the capture area 502. Then, learning is performed using the pixel values of the pixels corresponding to the capture area 502 as teacher data. In FIG. 8(a), the thick lines in the triangles representing each of the eight cameras 506 to 513 represent the image planes corresponding to the set learning area 801 (= capture area 502). In FIG. 8(b), the large dot 811 within the small dot area 801 represents a sampling point on the ray corresponding to a certain pixel on the captured image of the camera 506. In NeRF, sampling points are provided throughout the update range of the radiance field on the ray for rendering, so learning is performed for many pixels, and as a result, the processing takes time. Note that in NeRF as well, a temporary measure for high-speed processing such as reducing the sampling points in places where the density (volume density) of the object is continuously low has been made.

[0026] <<In the case of VaxNeRF>> (a) and (b) of FIG. 9 are diagrams for explaining learning when VaxNeRF is applied. In FIG. 9(a), the three small dot areas 901 to 903 represent the learning areas set as the update range of the radiance field. As shown in FIG. 9(a), in the case of VaxNeRF, the learning area of the radiance field is set by limiting it to the range where the object exists within the capture area 502. Also, in FIG. 9(b), the large dots 911 and 912 within the three small dot areas 901 to 903 represent the sampling points on the ray corresponding to a certain pixel on the captured image of the camera 506. As shown in FIG. 9(b), since the sampling points are limited to the inside of the approximate shape of the object, the amount of calculation during rendering is reduced.

[0027] <<In the case of this method>> Figures 10A - D are diagrams for explaining the learning when the method of this embodiment is applied. Now, there are three objects in the capture area 502. Therefore, three schematic shapes corresponding to the three objects can be obtained, and learning regions 1001 - 1003 that are the update ranges of the radiance fields are set for each object as shown in Figures 10A - C respectively. Different from the above - mentioned NeRF and VaxNeRF, since the radiance field is independently defined for each object, the pixels used for learning for each radiance field are determined. Here, since the update of the radiance field for each object (three in this example) is performed separately, in some cases, the learning for the same pixel may overlap, and the learning efficiency will decrease accordingly. However, by independently defining the radiance field for each object, the convergence of learning for each radiance field is accelerated, and an improvement in efficiency exceeding the decrease due to the overlap of learning for some pixels can be expected, so faster learning can be achieved overall. In Figure 10D, the large dots 1011 and 1012 represent the sampling points on the ray corresponding to a certain pixel on the captured image of the camera 506. As shown in Figure 10D, during drawing, color integration is also performed for the radiance fields other than the radiance field to be learned. Note that if the silhouette shown in the silhouette image is correct, it is not necessary to sample outside the schematic shape. However, in practical use, there are many cases where there are partial defects in the silhouette shown in the silhouette image, and the schematic shape obtained by estimating the object shape contains errors. Therefore, it is preferable to be able to absorb such errors by sampling the entire inside of the learning region enclosing the schematic shape. However, sampling may be performed only inside the schematic shape as in VaxNeRF.

[0028] As described above, according to this embodiment, a learning region is set for each object based on the schematic shape of the object, and the update of the radiance field independently defined for each object is performed. Thereby, the amount of information of each radiance field can be reduced, so the convergence is accelerated and the learning can be performed faster.

[0029] [Embodiment 2] Next, as a representation form of the radiance field defined independently for each object, a method of performing learning more quickly will be described as Embodiment 2 by using Plenoxels (Non-Patent Document 3). Plenoxels is a method of directly representing the radiance field with direct parameters and optimizing this without using a neural network. Therefore, it becomes possible to more directly control the density and color values for any position in space. Below, the explanation will be centered on the differences from Embodiment 1.

[0030] <Overview of Plenoxels>[[]] Before entering the explanation of this embodiment, an overview of Plenoxels will be explained. In Plenoxels, first, the space is divided into a coarse voxel grid. FIG. 11(a) shows one voxel 1101 in the coarsely divided voxel grid. Then, as shown in FIG. 11(b), at the positions of the corners (8 vertices) 1103 of each voxel, the density σ and the color c for each direction are held as parameters of the spherical harmonic function. Here, when obtaining the value of an arbitrary point 1104 other than the vertex 1103 corresponding to the corner of the voxel, the values of the 8 vertices of the voxel containing that point are obtained, and the density σ and color c of the arbitrary point 1104 are obtained by trilinear interpolation of those values. By integrating the pixel values of the sampling points on the ray corresponding to a certain pixel in the same way as NeRF using the parameters thus obtained, drawing is performed, and the radiance field is updated so that the difference between the obtained drawn image and the photographed image becomes small. The objective function for this update is defined as in the following formula (2).

[0031] [Number] TIFF2025107617000005.tif12150 TIFF2025107617000006.tif12150 ··· Formula (2) In the above formula (2), "L recon " is a term that reduces the difference from the pixel value of the photographed image, and "L TV" is a term that reduces the difference in values between neighboring parameters. For optimizing the parameters by the objective function, for example, the RMSProp method, the steepest descent method, the Adam method, the SGD method, etc. are used.

[0032] Then, the radiance field with the coarse voxel grid shown in Fig. 11(a) is optimized, and the range to be learned by density is specified. Next, for the specified range, the radiance field with a finer voxel grid with unnecessary parts removed as shown in Fig. 11(c) is optimized. The above is the outline of Plenoxels.

[0033] <The method of this embodiment> In this embodiment, the above optimization process in Plenoxels is simplified using the volume intersection method. Here, the estimation of the "coarse radiance field" in Plenoxels has the same role as the acquisition of the schematic shape described in Embodiment 1. That is, according to the schematic shape acquired by the schematic shape generation unit 302, the object shape with finer voxels can be defined. Furthermore, by obtaining in advance the initial values of the parameters assigned to each of the fine voxels, the optimization can converge faster. Further, in this embodiment, a method of further reducing the pixels used for learning by using the visibility determination (occlusion determination) of the object will also be described. Note that the idea of further reducing the pixels used for learning by using the visibility determination is also applicable to Embodiment 1 in the same way.

[0034] <Operation of the image processing apparatus> Subsequently, the operation of the image processing apparatus 120 according to this embodiment will be described. Fig. 12 is a flowchart showing the operation flow of the image processing apparatus 102 according to this embodiment. The method of this embodiment can also be realized by the image processing apparatus 102 having each functional unit shown in Fig. 3. However, there are some parts that are partially different in the function of the learning area setting unit 303. Hereinafter, the description will focus on the different parts.

[0035] Since S1201 and S1202 are the same as S401 and S402 in the flow of FIG. 4 of Embodiment 1, the description thereof will be omitted.

[0036] In S1203, the learning area setting unit 303 sets, based on the approximate shape acquired in S1202, a three-dimensional area composed of coarse voxels as a learning area for performing update of the radiance field.

[0037] In S1204, the three-dimensional field update unit 304 secures a memory area corresponding to the learning area set for each object in S1203. Specifically, in the RAM 202 as the three-dimensional field storage unit 305, a memory area for storing parameters representing the radiance field for fine voxels obtained by dividing the coarse voxels is secured.

[0038] In S1205, the three-dimensional field update unit 304 performs a process of calculating an initial value of the radiance field defined for each object. Details of this initial value calculation process will be described later.

[0039] In S1206, the drawing unit 306 performs the same process as S405 in the flow of FIG. 4 of Embodiment 1. That is, based on the camera parameters corresponding to each captured image constituting the multi-viewpoint image and the three-dimensional field (here, the radiance field) stored in the three-dimensional field storage unit 305, an image having the same viewing angle as the captured image is drawn.

[0040] In S1207, the drawing unit 306 performs a process of obtaining a difference in pixel values between the captured image and the drawn image for the target learning area of interest and updating the radiance field so that the difference in pixel values becomes small. At this time, in the first S1206 immediately after the start of the process, the initial value generated in S1205 is used. And this initial value is set based only on the pixels determined to be visible by the visibility determination described later.

[0041] Since S1208 and S1209 are the same as S407 and S408 in the flow of FIG. 4 of Embodiment 1, the description thereof will be omitted.

[0042] The above is the flow of operations in the image processing apparatus 120 according to the present embodiment.

[0043] <Details of Initial Value Calculation Processing of Radiance Field> Subsequently, with reference to the flowchart of FIG. 13, the initial value calculation processing of the radiance field defined for each object will be described in detail. Each of the processes S1301 to S1303 is performed collectively for all objects.

[0044] In S1301, the density of the radiance field is initialized. Specifically, for each voxel constituting a fine voxel grid, a process of assigning values of σ = 1 when inside the approximate shape, σ = 0 when outside, and σ = 0.5 when on the surface is performed.

[0045] In S1302, the surface voxels in the approximate shape obtained in S1202 are extracted. The extraction of these surface voxels is realized, for example, by selecting voxels that are located inside the approximate shape and are adjacent to voxels located outside the approximate shape.

[0046] In S1303, the surface voxels for all objects extracted in S1302 are projected onto the image plane using the camera parameters of all shooting viewpoints of the multi-viewpoint image, and a depth map at all shooting viewpoints is generated. The generated depth map is held in the RAM 102.

[0047] In S1304, for each surface voxel, its center coordinates are projected onto the shooting viewpoint, and a process of comparing the depth value (d’) obtained by the projection with the depth value (d) in the depth map to determine the visibility from each viewpoint is performed. Specifically, it is determined to be visible (not blocked) when d’ ≦ d + m. Here, “m” is a constant, and a value such as a size one size larger than the voxel in the fine voxel grid, for example, 1 to 2 mm, is entered.

[0048] In S1305, for each surface voxel, based on the pixel value of the captured image corresponding to the viewpoint determined to have visibility, the color information of the radiance field is initialized. In Plenoxels, different colors for each direction are represented by the parameters of the spherical harmonic function for each RGB component. Therefore, for example, the base component of the spherical harmonic function (the average value of values in all directions) is set to the average value of the pixel values of the captured image corresponding to the viewpoint determined to have visibility, and the component representing the color change in each other direction is set to 0 as the initialization process.

[0049] <Comparison with Embodiment 1> Figures 14A to 14D are diagrams for explaining the learning when the method of this embodiment is applied, and correspond to Figures 10A to 10D of Embodiment 1 respectively. As shown in Figures 14A to 14C respectively, the three radiance fields 1401 to 1403 are each optimized using only the pixels for which the corresponding object is visible. This enables updates that exclude the influence of other radiance fields, enables optimization by processing multiple radiance fields in parallel, and can further reduce the amount of information held by each radiance field. Furthermore, since there is no overlap of pixels used for optimization between objects, the amount of computation is further reduced.

[0050] As described above, according to this embodiment, based on Plenoxels that handle the radiance field with direct parameters, after calculating the initial values of each radiance field, each radiance field is updated, so that the convergence of optimization can be accelerated. In addition, by additionally performing visibility determination, the amount of computation can be further reduced, and a significant speedup of learning can be achieved.

[0051] <Modification Example> In Embodiments 1 and 2, as the three-dimensional field, an example was given of a radiance field that associates different colors with density and direction for each coordinate in space, but it is not limited to this. For example, the color information associated with the coordinates in space may be an isotropic color (color filed) that does not depend on the direction. Also, the three-dimensional field is not limited to the radiance field. For example, it may be an occupancy field that represents volume density. It may also be a field represented by a bidirectional reflectance distribution function (BRDF: Bidirectional Reflectance Distribution Function) that represents the distribution characteristics of reflected light with respect to incident light. Furthermore, it may be a field that represents the amount of ambient light entering (Light Visibility). Similar to the radiance field described in Embodiments 1 and 2, these can be learned with multi-viewpoint images as input, and by inputting the camera parameters of the virtual viewpoint into each field after learning, the following virtual viewpoint data can be obtained.

[0052] · Occupancy field: A map representing the opacity when viewed from the virtual viewpoint · Bidirectional reflectance distribution function field: A map representing the bidirectional reflectance distribution function when viewed from the virtual viewpoint · Ambient light entry amount field: A map representing the degree of visibility when viewed from the virtual viewpoint Furthermore, the three-dimensional field may be a signed distance field where the inside of the object is negative and the outside is positive, or a binary field where the inside of the object is 0 and the outside is 1 (Surface Filed). Furthermore, it may be a field of the normal direction of the object surface (Normal Field). In these cases, in addition to the multi-viewpoint images, learning is performed with the depth map corresponding to the multi-viewpoint images as input, and based on each field after learning, the following virtual viewpoint data can be obtained.

[0053] · Signed distance field where the inside of the object is negative and the outside is positive: The depth map when viewed from the virtual viewpoint · Binary field where the inside of the object is 0 and the outside is 1: The depth map when viewed from the virtual viewpoint · Field in the normal direction of the object surface: Normal map as seen from the virtual viewpoint.

[0054] For example, the above-mentioned Surface Filed is used in Pixel NeRf and Double Filed. Also, BRDF, Light Visibility, and Normal Field are used in NeRFactor. Also, Signed Distance Filed is used in NeuS. The method of the present disclosure is applicable in these cases as well. So far, the method of generating virtual viewpoint data based on virtual viewpoint parameters has been described. However, the learning of the three-dimensional field is also effective for obtaining three-dimensional shape data. For example, Occupancy Field and Signed Distance Filed can be taken out as voxels, and mesh data can be obtained by using the Marching Cubes method for the taken-out voxels.

[0055] [Other Embodiments] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

[0056] Also, the present disclosure includes the following configurations and methods.

[0057] [Configuration 1] An image processing apparatus that performs learning to obtain virtual viewpoint data corresponding to a virtual viewpoint or a three-dimensional shape from a plurality of captured images obtained by a plurality of imaging devices, an acquisition means for acquiring the plurality of captured images, a generation means for generating schematic shape data representing the three-dimensional shape of an object based on the plurality of captured images acquired by the acquisition means, Setting means for setting a learning region for each object based on the schematic shape data generated by the generation means; Learning means for learning a three-dimensional field corresponding to the captured image, with respect to the learning region set for each object by the setting means; An image processing apparatus, characterized by comprising the above.

[0058] [Configuration 2] The learning means includes: Storage means for storing the three-dimensional field; Drawing means for drawing an image corresponding to each imaging viewpoint having the same angle of view as each captured image, based on the camera parameters corresponding to each of the plurality of captured images and the three-dimensional field stored by the storage means; Updating means for updating the three-dimensional field based on the plurality of drawn images obtained by the drawing means and the plurality of captured images; The image processing apparatus according to Configuration 1, characterized by having the above.

[0059] [Configuration 3] The updating means targets a learning region of interest among the learning regions set for each object by the setting means, obtains a color difference between a captured image and a drawn image in a corresponding relationship with each other, and updates the three-dimensional field so that the color difference becomes smaller. The image processing apparatus according to Configuration 2 is characterized by the above.

[0060] [Configuration 4] The updating means obtains the color difference for each pixel in a corresponding relationship between the two images and performs the update. The image processing apparatus according to Configuration 3 is characterized by the above.

[0061] [Configuration 5] The updating means obtains the color difference from the pixels of the captured image having visibility with respect to the elements constituting the schematic shape data and the pixels of the corresponding drawn image, and performs the update. The image processing apparatus according to Configuration 4 is characterized by the above.

[0062] [Configuration 6] The updating means determines an initial value of the three-dimensional field based on pixel values of a captured image of a viewpoint from which the element is visible, and performs the update, and the image processing apparatus according to Configuration 5, characterized in that.

[0063] [Configuration 7] The setting means sets, as the learning region, a solid that circumscribes the three-dimensional shape of the object represented by the schematic shape data, and the image processing apparatus according to any one of Configurations 1 to 6, characterized in that.

[0064] [Configuration 8] The schematic shape data is data that specifies the three-dimensional shape of the object by a set of a plurality of elements, The setting means sets, as the learning region, a three-dimensional region represented by a set of elements having a size larger than that of the elements, and the image processing apparatus according to any one of Configurations 1 to 6, characterized in that.

[0065] [Configuration 9] The generation means generates the shape data representing the schematic shape of the object by a volume intersection method using the plurality of captured images, and the image processing apparatus according to any one of Configurations 1 to 8, characterized in that.

[0066] [Configuration 10] The three-dimensional field is a radiance field that associates a volume density and an anisotropic color with each coordinate in the imaging space of the plurality of imaging devices, The virtual viewpoint data is a virtual viewpoint image representing the view from the virtual viewpoint, and the image processing apparatus according to any one of Configurations 1 to 9, characterized in that.

[0067] [Configuration 11] The three-dimensional field is a radiance field that associates a volume density and an isotropic color with each coordinate in the imaging space of the plurality of imaging devices, The virtual viewpoint data is a virtual viewpoint image representing the view from the virtual viewpoint, and the image processing apparatus according to any one of Configurations 1 to 9, characterized in that.

[0068] [Configuration 12] The three-dimensional field is a field of opacity representing volume density, The virtual viewpoint data is a map representing the opacity when viewed from the virtual viewpoint, The image processing apparatus according to any one of Configurations 1 to 9, characterized in that.

[0069] [Configuration 13] The three-dimensional field is a field of bidirectional reflectance distribution function, The virtual viewpoint data is a map representing the bidirectional reflectance distribution function when viewed from the virtual viewpoint, The image processing apparatus according to any one of Configurations 1 to 9, characterized in that.

[0070] [Configuration 14] The three-dimensional field is a field of the amount of ambient light entering, The virtual viewpoint data is a map representing the degree of visibility when viewed from the virtual viewpoint, The image processing apparatus according to any one of Configurations 1 to 9, characterized in that.

[0071] [Configuration 15] The three-dimensional field is a floating-point field representing the inside of the object as negative and the outside as positive, or a binary field representing the inside of the object as 0 and the outside as 1, The virtual viewpoint data is a depth map when viewed from the virtual viewpoint, The image processing apparatus according to any one of Configurations 1 to 9, characterized in that.

[0072] [Configuration 16] The three-dimensional field is a field of the normal direction of the object surface, The virtual viewpoint data is a normal map when viewed from the virtual viewpoint, The image processing apparatus according to any one of Configurations 1 to 9, characterized in that.

[0073] [Configuration 17] An image processing apparatus comprising inference means for performing inference using a three-dimensional field learned by the image processing apparatus according to any one of Configurations 1 to 16 and outputting the virtual viewpoint data.

[0074] [Configuration 18] The inference means storage means for holding the learned three-dimensional field; generation means for generating the virtual viewpoint data based on the three-dimensional field held in the storage means according to camera parameters of a virtual viewpoint; The image processing apparatus according to Configuration 17, characterized by comprising the above.

[0075] [Method 1] An image processing method for obtaining virtual viewpoint data corresponding to a virtual viewpoint or a three-dimensional shape from a plurality of captured images obtained by a plurality of imaging devices, the method comprising: an acquisition step of acquiring the plurality of captured images; a generation step of generating schematic shape data representing a three-dimensional shape of an object based on the plurality of captured images acquired in the acquisition step; a setting step of setting a learning region for each object based on the schematic shape data generated by the generation means; a learning step of learning a three-dimensional field corresponding to the captured image for the learning region set for each object in the setting step; The image processing method characterized by including the above.

[0076] [Configuration 20] A program for causing a computer to function as each means of the image processing apparatus according to any one of Configurations 1 to 18.

Claims

1. An image processing device that performs learning to obtain virtual viewpoint data corresponding to a virtual viewpoint from multiple captured images obtained by multiple imaging devices, A first acquisition means for acquiring the plurality of captured images and camera parameters corresponding to each of the captured images, A second acquisition means for acquiring three-dimensional shape data representing the three-dimensional shape of an object, which is generated based on the aforementioned plurality of captured images, Based on the three-dimensional shape data and the camera parameters, a means for identifying objects to identify the pixels in which the object is captured in the plurality of captured images, A learning means that learns using the multiple captured images, targeting the pixels identified for each object, An image processing apparatus characterized by having

2. The image processing apparatus according to claim 1, characterized in that the second acquisition means generates a silhouette image of the object based on the plurality of captured images and acquires the three-dimensional shape data by generating the three-dimensional shape data using the silhouette image.

3. The aforementioned three-dimensional shape data is data that represents the three-dimensional shape of the object by a collection of multiple elements, The aforementioned identification means identifies pixels using the element that is the surface of the three-dimensional shape data among the elements. The image processing apparatus according to feature 1.

4. The aforementioned specifying means is, The elements that are the surface of the three-dimensional shape data are projected onto each captured image based on the camera parameters to generate a depth map. By determining visibility using the depth value obtained by projecting the element, which is the surface of the three-dimensional shape data, and the depth value in the depth map, pixels are identified. The image processing apparatus according to claim 3.

5. The learning means is A recording means for recording the aforementioned virtual viewpoint data, A drawing means that generates a drawing image based on the camera parameters and the virtual viewpoint data, An update means for updating the virtual viewpoint data based on the difference in pixel values ​​between the drawing image and the captured image, The image processing apparatus according to claim 1, characterized by having the following features.

6. The image processing apparatus according to claim 5, characterized in that the updating means calculates the difference in pixel values ​​for each corresponding pixel and updates the virtual viewpoint data so that the difference becomes smaller.

7. The image processing apparatus according to claim 5, characterized in that the learning means does not use pixels other than those identified by the identification means for updating the virtual viewpoint data.

8. The image processing apparatus according to claim 5, characterized in that the learning means determines the initial value of the virtual viewpoint data based on the pixel value of the captured image corresponding to the pixel identified by the identification means, and then updates the virtual viewpoint data.

9. The aforementioned virtual viewpoint data is represented by storing parameters that represent volume density and color according to direction for each vertex of a voxel grid. The learning means updates the virtual viewpoint data by optimizing the parameters. The image processing apparatus according to feature 5.

10. The color corresponding to the aforementioned direction is expressed by the parameters of spherical harmonics. The learning means initializes the basis components of the spherical harmonics based on the average value of the pixel values ​​corresponding to the viewpoint determined to be visible. The image processing apparatus according to feature 9.

11. The image processing apparatus according to claim 5, characterized in that the learning means updates each virtual viewpoint data corresponding to a plurality of objects independently of each other.

12. The image processing apparatus according to claim 1, wherein the identifying means identifies pixels based on visibility such that the pixels identified for each object do not overlap with pixels identified for other objects.

13. The image processing apparatus according to claim 1, further comprising inference means for performing inference using the virtual viewpoint data and outputting the virtual viewpoint data according to the camera parameters of the virtual viewpoint.

Images