Image processing apparatus, image processing method, and program
By setting learning areas for each object and independently defining three-dimensional fields, the image processing device accelerates the learning process for generating images from virtual viewpoints in scenes with sparsely distributed objects.
Patent Information
- Application Number
- JP2023078633
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-11
- Publication Date
- 2024-11-21
- Estimated Expiration
- 2043-05-11
AI Technical Summary
Existing methods for generating images from virtual viewpoints using multi-view images, such as NeRF and VaxNeRF, are inefficient for scenes with sparsely distributed objects, requiring significant time for learning.
An image processing device that sets learning areas for each object based on rough shape data, independently defines a three-dimensional field for each object, and updates the radiance field using a visual volume intersection method to reduce the amount of information and accelerate learning.
This approach allows for faster learning and image generation from virtual viewpoints by reducing the amount of information and accelerating convergence of the learning process.
Smart Images

Figure 2024162762000001_ABST
Abstract
Description
[Technical field]
[0001] The present disclosure relates to an image processing technique for generating data corresponding to a virtual viewpoint from a multi-viewpoint image. [Background technology]
[0002] There is a technology called NeRF (Neural Radiance Fields) that generates an image corresponding to an arbitrary virtual viewpoint by inputting a multi-view image with known camera parameters (Non-Patent Document 1, Patent Document 1). This NeRF is a neural network that outputs volume density σ and emitted radiance from five-dimensional input variables {arbitrary spatial position coordinates (x, y, z) and direction (θ, φ)}. In learning this neural network, the pixel values of the multi-view image are used as teachers, and the difference between the pixel values of the drawing result is used as loss. Therefore, in the learning process, drawing, loss calculation, and error backpropagation are performed for the number of images contained in the multi-view image, and these are repeated, so learning takes a long time. For example, at least 12 hours are required to learn a scene using 1K images from 100 viewpoints as teachers. To address the issue of long learning times, a method called VaxNeRF has been proposed that uses a volume intersection method to speed up the learning process (see Non-Patent Document 2). The volume intersection method is a method to obtain the 3D shape of an object by constructing a cone from each viewpoint by back-projecting the silhouette of the object extracted from a multi-view image into a 3D space, and obtaining the intersection of each cone. The volume intersection method has the characteristic that it guarantees that the object does not exist outside the obtained 3D shape, assuming that the extracted silhouette is correct. By utilizing this characteristic, VaxNeRF does not use pixels outside the silhouette in learning, and limits the sampling points during drawing to the inside of the 3D shape obtained by the volume intersection method, thereby reducing the amount of calculations during learning and speeding up learning. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] U.S. Pat. No. 1,130,8659 [Non-patent literature]
[0004] [Non-Patent Document 1] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. 3 Aug 2020. [Non-Patent Document 2] Naruya Kondo, Yuya Ikeda, Andrea Tagliasacchi, Yutaka Matsuo, Yoichi Ochiai, Shixiang Shane Gu. VaxNeRF: Revisiting the Classic for Voxel-Accelerated Neural Radiance Field. Nov 25 2021. [Non-Patent Document 3] Alex Yu, Sara Fridovich-Keil, Matthew Tancik, Qinhong Chen, Benjamin Recht, Angjoo Kanazawa. Plenoxels: Radiance Fields without Neural Networks. 9 Dec 2021. Summary of the Invention [Problem to be solved by the invention]
[0005] For example, when targeting a scene in which many objects are sparsely distributed over a vast shooting area, such as a soccer game, even the VaxNeRF method still requires a huge amount of time for learning. The present disclosure has been made in consideration of the above-mentioned problems, and aims to perform faster learning for generating images corresponding to virtual viewpoints from multi-viewpoint images. [Means for solving the problem]
[0006] The image processing device according to the present disclosure is an image processing device that performs learning to obtain virtual viewpoint data corresponding to a virtual viewpoint or a three-dimensional shape from a plurality of captured images obtained by a plurality of imaging devices, and includes an acquisition means for acquiring the plurality of captured images, a generation means for generating outline shape data representing a three-dimensional shape of an object based on the plurality of captured images acquired by the acquisition means, a setting means for setting a learning area for each object based on the outline shape data generated by the generation means, and a learning means for learning a three-dimensional field corresponding to the captured images, with the learning area set for each object by the setting means as a target. The present invention is characterized by comprising: Effect of the Invention
[0007] According to the present disclosure, it becomes possible to perform learning for generating an image corresponding to a virtual viewpoint from a multi-viewpoint image more quickly. [Brief description of the drawings]
[0008] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of an image processing system. [Diagram 2] FIG. 2 is a diagram showing an example of the hardware configuration of an image processing apparatus. [Diagram 3] FIG. 2 is a block diagram showing the functional configuration of the image processing apparatus. [Figure 4] 5 is a flowchart showing the flow of operations of the image processing device according to the first embodiment. [Diagram 5] A two-dimensional view of the imaging area from above. [Figure 6]1A to 1C are diagrams for explaining acquisition of a rough shape using the visual hull method. [Figure 7] A diagram showing a state where there are three objects in the capture area. [Figure 8] (a) and (b) are diagrams explaining learning using NeRF. [Figure 9] (a) and (b) are diagrams explaining learning by VaxNeRF. [Figure 10A] FIG. 1 is a diagram for explaining learning by the method of the first embodiment. [Figure 10B] FIG. 1 is a diagram for explaining learning by the method of the first embodiment. [Figure 10C] FIG. 1 is a diagram for explaining learning by the method of the first embodiment. [Figure 10D] FIG. 1 is a diagram for explaining learning by the method of the first embodiment. [Figure 11] (a) to (c) are diagrams outlining Plenoxels. [Figure 12] 10 is a flowchart showing the flow of operations of an image processing device according to a second embodiment. [Figure 13] 11 is a flowchart showing details of a process for calculating an initial value of a radiance field. [Figure 14A] FIG. 10 is a diagram for explaining learning by the method of the second embodiment. [Figure 14B] FIG. 10 is a diagram for explaining learning by the method of the second embodiment. [Figure 14C] FIG. 10 is a diagram for explaining learning by the method of the second embodiment. [Figure 14D] FIG. 10 is a diagram for explaining learning by the method of the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0009] Hereinafter, the present embodiment will be described with reference to the drawings. Note that the following embodiment does not necessarily limit the present invention. Also, not all of the combinations of features described in the present embodiment are necessarily essential to the solution of the present invention. Also, as a common idea to each embodiment, in this disclosure, a rough three-dimensional shape (schematic shape) of an object is used as in Non-Patent Document 2. Then, a three-dimensional field (this "field" varies depending on the learning content. Hereinafter, referred to as "three-dimensional field" in this specification) in the shooting space to be learned is defined independently for each object and learned. This reduces the amount of information held by each three-dimensional field, speeding up the convergence of learning for each three-dimensional field, and realizing high-speed learning.
[0010] [Embodiment 1] <Image processing system configuration> FIG. 1 is a diagram showing an example of the configuration of an image processing system that generates a virtual viewpoint image according to this embodiment. The image processing system includes a plurality of imaging devices (cameras) 101, an image processing device 102, a user interface (UI) panel 103, a storage device 104, and a display device 105. The plurality of cameras 101 synchronously capture an object 107 in a capture area 106 from multiple viewpoints according to the capture conditions, and acquire a plurality of captured image (multi-viewpoint image) data corresponding to each viewpoint. The captured image acquired by the camera 101 may be a still image, a moving image, or both a still image and a moving image. In this embodiment, the term "image" includes both a still image and a moving image, unless otherwise specified. The image processing device 102 generates three-dimensional shape data (3D model) of the object 107 based on the control of the plurality of cameras 101 and the plurality of captured images acquired from the plurality of cameras 101. The UI panel 103 is a display device such as a liquid crystal display, and functions as a user interface for notifying the user of the current capture conditions and processing settings. The UI panel 103 may also include an input device such as a touch panel or buttons, and can receive instructions from a user regarding shooting conditions and processing settings. The input device may be provided separately from the UI panel 103, such as a mouse or a keyboard. The storage device 104 stores the three-dimensional shape data of the object acquired from the image processing device 102. The display device acquires and displays the three-dimensional shape data of the object from the image processing device 102. The shooting area 106 is a space (three-dimensional space) surrounded by multiple cameras 101 installed in a studio, and the frame indicated by a solid line indicates the contours of the shooting area 106 on the floor in the front-rear and left-right directions.
[0011] <Hardware configuration of image processing device> 2 is a diagram showing an example of the hardware configuration of the image processing device 102. The image processing device 102 includes a CPU 201, a RAM 202, a ROM 203, a storage unit 204, a control interface (I / F) 205, an input interface (I / F) 206, an output interface (I / F) 207, and a main bus 208. The CPU 201 is a processor that performs overall control of each unit of the image processing device 102. The RAM 202 functions as the main memory, work area, etc. of the CPU 201. The ROM 203 stores a group of programs executed by the CPU 201. The storage unit 204 stores applications executed by the CPU 201, data used in image processing, etc. The control (I / F) 205 is connected to a plurality of cameras 101, and is an interface for setting shooting conditions, starting and stopping shooting, and the like. The input I / F 206 is a serial bus interface such as SDI or HDMI (registered trademark), and acquires multi-viewpoint image data from the multiple cameras 101 via the input I / F 206. The output I / F 207 is a serial bus interface such as USB or IEEE1394, and outputs the subject shape to the storage device 104 or display device 105 via the output I / F 207. The main bus 208 is a transmission path that connects the various modules of the image processing device 102.
[0012] In this embodiment, eight cameras installed in a studio are used to capture images of one or more objects from around the object. Camera parameters such as internal parameters, external parameters, and distortion parameters of the cameras are assumed to be stored in the storage unit 204. The internal parameters represent the coordinates of the center of the image and the lens focal length, and the external parameters represent the position and orientation of the camera. The camera parameters of the multiple cameras do not have to be the same for all of them, and for example, the angles of view may be different.
[0013] <Operation of image processing device> Next, the operation of the image processing device 120 according to this embodiment will be described. FIG. 3(a) is a block diagram showing the functional configuration of the image processing device 102 in the learning phase, and FIG. 4 is a flowchart showing the flow of the operation of the image processing device 102. As shown in FIG. 3, the image processing device 102 has a learning unit 300, an image input unit 301, a rough shape generating unit 302, and a learning area setting unit 303. The learning unit 300 has a drawing unit 306, a three-dimensional field storage unit 305, and a drawing unit 306. Below, the operation of each unit of the image processing device 120 will be described along the flowchart in FIG. 4. In the following description, the symbol "S" means a step. In addition, when the input captured image is a moving image, it is executed on a frame-by-frame basis.
[0014] In S401, the image input unit 301 receives and acquires multi-viewpoint images from the multiple cameras 101 via the input I / F 206. Alternatively, the multi-viewpoint images may be acquired by reading data of the multi-viewpoint images stored in the storage unit 204. The acquired multi-viewpoint images are held in the RAM 202.
[0015] In S402, the outline shape generating unit 302 generates outline shape data representing a rough three-dimensional shape of an object captured in the acquired multi-view image. In this embodiment, the shape of the object is derived by a volume intersection method as the outline shape data, and three-dimensional shape data expressed as a voxel set is generated. In the volume intersection method, first, for each captured image constituting the multi-view image, foreground / background separation is performed using an image (background image) in which the object is not captured, and an image (silhouette image) representing the silhouette of the object is obtained. The background image is prepared, for example, by capturing an image of the shooting area 106 in which there is no object in advance. Next, based on the camera parameters of each of the multiple cameras 101, each voxel included in the voxel set corresponding to the shooting area 106 is projected onto each silhouette image of the object captured from the multi-view image. Then, a voxel set consisting of only voxels projected into the silhouette in all silhouette images is regarded as the outline shape of the object. FIG. 5 is a two-dimensional view of the shooting area 106 from above. In Fig. 5, a rectangle 501 outside the dashed line indicates the wall of the studio where the camera 101 is installed. Moreover, a rectangle 502 inside the dashed line indicates the range (capture area) where an object may exist, and solid-line triangles 506-513 indicate the positions, directions, and angles of view of each of the eight cameras 101. Fig. 6 shows how a rough shape 601 is obtained by the volume intersection method for an object 600 with a concave center.
[0016] In S403, the learning area setting unit 303 sets, for each object, a rectangular parallelepiped circumscribing the outline shape of the object as a target three-dimensional area (learning area) for learning, based on the outline shape data for each object generated in S402. The shape of this learning area may be a solid that contains the outline shape of the object, for example, a sphere or an oval sphere. Alternatively, a three-dimensional area of a coarser outline shape made up of a voxel set made up of voxels larger than the voxels that are elements that make up the outline shape data may be set as the learning area.
[0017] In S404, the three-dimensional field update unit 304 secures a memory area corresponding to the learning area set for each object in S403 in the RAM 202 as the three-dimensional field storage unit 305. Each step below will be explained assuming that the "three-dimensional field" is a radiance field in NeRF (a vector field that associates volume density (≒ opacity) with radiance (≒ color) for each coordinate in space). When the three-dimensional field is a NeRF radiance field, a value representing the volume density (hereinafter simply referred to as "density") at any position in the space and a value representing anisotropic color that differs for each direction are stored in the secured memory area.
[0018] In S405, the drawing unit 306 draws (volume renders) images corresponding to each shooting viewpoint having the same angle of view as each captured image, based on the camera parameters corresponding to each captured image constituting the multi-viewpoint image and the radiance field stored in the three-dimensional field storage unit 305. Specifically, a process is performed to obtain the pixel value C(r) of each pixel corresponding to the light ray r when viewed from the same viewpoint as the captured image, for example, using the following formula (1).
[0019]
number
[0020] In S406, the three-dimensional field update unit 304 performs a process of calculating the color difference between the captured image and the drawn image, which correspond to each other, for the learning area of interest among the learning areas set in S403, and updating the radiance field so as to reduce the color difference. This process corresponds to error calculation and error backpropagation in deep learning. In this embodiment, the color difference between both images is calculated for each corresponding pixel, and the value is defined as the squared Euclidean distance of the color (RGB).
[0021] In S407, it is determined whether the update process of the radiance field has been completed for all the learning regions set in S403. If there is a learning region for which the update process of the radiance field has not been completed, the process returns to S405, and the next learning region of interest is determined and the same process is executed. On the other hand, if the update process of the radiance field has been completed for all the learning regions, the process proceeds to S408.
[0022] In S408, it is determined whether the updates have sufficiently converged for all the radiance fields. For example, the errors calculated for each pixel are summed up for all viewpoints, and it is determined that the convergence has occurred when the reduction in the sum from the previous calculation falls below a threshold (for example, 0.1%). Convergence may also be determined when the error calculated for each pixel falls below a predetermined threshold, or when the number of times the update process for the radiance field is counted and reaches a predetermined number. For example, some of the multiple cameras may be used as evaluation cameras that are not used for updating the radiance field, and convergence may be determined when the error with the captured image of the camera starts to increase (overlearning occurs). Furthermore, it may be determined by a combination of these. If the updates have converged, this process is terminated, and if the updates have not converged, the process returns to S405 and the same process is repeated.
[0023] The above is the flow of the operation in the image processing device 120 according to the present embodiment. FIG. 3(b) is a block diagram showing the functional configuration of the image processing device 102′ in the case where a virtual viewpoint image is generated (inference phase) using a radiance field determined to have converged. As shown in FIG. 3(b), the image processing device 102′ in charge of the inference phase has an inference unit 310 consisting of the above-mentioned three-dimensional field storage unit 305 and drawing unit 306. Instead of the camera parameters of the shooting viewpoint of the imaging device 101, the camera parameters of the virtual viewpoint are input to the drawing unit 306 of the inference unit 310. Then, the drawing unit 306 performs volume rendering according to the parameters of the virtual viewpoint using the converged radiance field stored in the three-dimensional field storage unit 305, and generates a drawn image (virtual viewpoint image) corresponding to the virtual viewpoint. Note that, in the flow of FIG. 4 described above, both processes of S405 and S406 are repeated for each learning area, but this is not limited to this. For example, the process of S405 may be performed for all learning regions to obtain rendered images, and then the process of S406 may be performed collectively to update the radiance field corresponding to each learning region. In addition, in the case of a moving image, object tracking may be performed between frames, and learning regions may be set (S403) based on the results, and generation of outline shapes by the visual volume intersection method may be omitted for some frames. This allows a series of processes for the moving image to be performed more efficiently.
[0024] <Differences from conventional technology> Here, the difference between NeRF and VaxNeRF and the method of this embodiment will be described taking as an example a case where three objects exist in the capture area 502 (see FIG. 7).
[0025] In the case of NeRF 8(a) and 8(b) are diagrams for explaining learning when NeRF is applied. In the case of NeRF, which defines the update range (learning area) of the radiance field during learning, the entire capture area 502 is set as the update range of the radiance field. In FIG. 8(a), a small dot area 801 represents the learning area set as the update range of the radiance field, and is now equal to the capture area 502. Learning is performed using the pixel values of the pixels corresponding to the capture area 502 as teacher data. In FIG. 8(a), the thick line parts in the triangles representing the eight cameras 506 to 513 represent the image surface corresponding to the set learning area 801 (=capture area 502). In FIG. 8(b), a large dot 811 in the small dot area 801 represents a sampling point on the light ray corresponding to a pixel on the image captured by the camera 506. In NeRF, sampling points are set in the entire update range of the radiance field on the light ray for drawing, so learning is performed for many pixels, which results in a long processing time. NeRF also makes efforts to speed up processing by reducing the number of sampling points in areas where the volume density of objects is continuously low.
[0026] <In the case of VaxNeRF> 9(a) and (b) are diagrams for explaining learning when VaxNeRF is applied. In FIG. 9(a), three small dot regions 901 to 903 represent a learning region set as an update range of the radiance field. As shown in FIG. 9(a), in the case of VaxNeRF, the learning region of the radiance field is set limited to the range in the capture area 502 where an object exists. Also, in FIG. 9(b), large dots 911 and 912 in the three small dot regions 901 to 903 represent sampling points on a light ray corresponding to a pixel on an image captured by the camera 506. As shown in FIG. 9(b), the amount of calculation during drawing is reduced by the amount that the sampling points are limited to the inside of the outline shape of the object.
[0027] <<In the case of this method>> 10A to 10D are diagrams for explaining learning when the method of this embodiment is applied. Currently, three objects exist in the capture area 502. Therefore, three outline shapes corresponding to the three objects are obtained, and as shown in FIGS. 10A to 10C, learning regions 1001 to 1003, which are the update ranges of the radiance field, are set for each object. Unlike the above-mentioned NeRF and VaxNeRF, the radiance field is defined independently for each object, so pixels to be used for learning are determined for each radiance field. Here, since the radiance field for each object (three in this example) is updated separately, in some cases learning for the same pixel may overlap, and the efficiency of learning decreases accordingly. However, by defining the radiance field independently for each object, the convergence of learning for each radiance field is accelerated, and it is expected that the improvement in efficiency will exceed the decrease due to the overlap of learning for some pixels, so that faster learning is possible overall. In FIG. 10D, large dots 1011 and 1012 represent sampling points on a light ray corresponding to a certain pixel on the image captured by the camera 506. As shown in FIG. 10D, when drawing, radiance fields other than the radiance field to be learned are also superimposed and color-accumulated. If the silhouette shown in the silhouette image is correct, there is no need to sample the outside of the outline shape. However, in practice, there are many cases where the outline shape obtained by estimating the shape of an object contains errors, such as partial loss of the silhouette shown in the silhouette image. Therefore, it is preferable to sample the entire inside of the learning area that contains the outline shape so that such errors can be absorbed. However, sampling may be performed only on the inside of the outline shape, as in VaxNeRF.
[0028] As described above, according to this embodiment, a learning region is set for each object based on the outline shape of the object, and the radiance field defined independently for each object is updated. This reduces the amount of information contained in each radiance field, resulting in faster convergence and faster learning.
[0029] [Embodiment 2] Next, as a representation format of the radiance field defined independently for each object, Plenoxels (Non-Patent Document 3) is used to describe a method for performing learning more quickly as Embodiment 2. Plenoxels is a method of directly representing the radiance field with direct parameters without using a neural network and optimizing this. Therefore, it becomes possible to more directly control the density and color values for any position in space. Below, the explanation will be centered on the differences from Embodiment 1.
[0030] <Overview of Plenoxels>[[]]END]] Before entering the explanation of this embodiment, an overview of Plenoxels will be explained. In Plenoxels, first, the space is divided into a coarse voxel grid. FIG. 11(a) shows one voxel 1101 in the coarsely divided voxel grid. Then, as shown in FIG. 11(b), at the positions of the corners (8 vertices) 1103 of each voxel, the density σ and the color c for each direction are held as parameters of the spherical harmonic function. Here, when obtaining the value of an arbitrary point 1104 other than the vertex 1103 corresponding to the corner of the voxel, the values of the 8 vertices of the voxel containing that point are obtained, and by trilinear interpolation of those values, the density σ and color c of the arbitrary point 1104 are obtained. Using the parameters thus obtained, similar to NeRF, rendering is performed by integrating the pixel values of the sampling points on the ray corresponding to a certain pixel, and the radiance field is updated so that the difference between the obtained rendered image and the captured image becomes small. The objective function for this update is defined as in the following formula (2).
[0031] [Number] TIFF2024162762000005.tif12150 TIFF2024162762000006.tif12150 ··· Formula (2) In the above formula (2), "L recon " is a term that reduces the difference from the pixel value of the captured image, and "L TV" is a term that reduces the difference in values between nearby parameters. For example, the RMSProp method, the steepest descent method, the Adam method, the SGD method, etc. are used to optimize the parameters using the objective function.
[0032] Then, the radiance field based on the coarse voxel grid shown in Fig. 11(a) is optimized, and the range to be learned is identified based on the density. Next, for the identified range, the radiance field based on a finer voxel grid with unnecessary parts removed is optimized, as shown in Fig. 11(c). This concludes the overview of Plenoxels.
[0033] <Method of this embodiment> In this embodiment, the optimization process in Plenoxels is simplified by using a volume intersection method. Here, the estimation of the "coarse radiance field" in Plenoxels has the same role as the acquisition of the outline shape described in the first embodiment. That is, the object shape can be defined by finer voxels according to the outline shape acquired by the outline shape generating unit 302. Furthermore, the optimization can be converged faster by previously determining the initial values of the parameters assigned to each fine voxel. Furthermore, in this embodiment, a method of further reducing the pixels used for learning by using the visibility judgment (occlusion judgment) of the object is also described. The idea of further reducing the pixels used for learning by using the visibility judgment can be similarly applied to the first embodiment.
[0034] <Operation of image processing device> Next, the operation of the image processing device 120 according to this embodiment will be described. Fig. 12 is a flowchart showing the flow of the operation of the image processing device 102 according to this embodiment. The method of this embodiment can also be realized by the image processing device 102 having the functional units shown in Fig. 3. However, there are some differences in the functions of the learning area setting unit 303. The following description will focus on these differences.
[0035] S1201 and S1202 are the same as S401 and S402 in the flow of FIG. 4 in the first embodiment, and therefore a description thereof will be omitted.
[0036] In S1203, the learning area setting unit 303 sets a three-dimensional area consisting of coarse voxels as a learning area for updating the radiance field based on the rough shape acquired in S1202.
[0037] In S1204, the three-dimensional field update unit 304 secures a memory area corresponding to the learning area set for each object in S1203. Specifically, a memory area for storing parameters expressing a radiance field for fine voxels obtained by dividing coarse voxels is secured in the RAM 202 as the three-dimensional field storage unit 305.
[0038] In step S1205, the three-dimensional field update unit 304 performs a process of calculating an initial value of the radiance field defined for each object. The details of this initial value calculation process will be described later.
[0039] In S1206, the rendering unit 306 performs the same process as S405 in the flow of Fig. 4 of the first embodiment. That is, based on the camera parameters corresponding to each captured image constituting the multi-viewpoint image and the three-dimensional field (here, the radiance field) stored in the three-dimensional field storage unit 305, an image with the same angle of view as the captured image is rendered.
[0040] In S1207, the drawing unit 306 performs a process of calculating the difference in pixel values between the corresponding captured image and the drawn image for the learning area of interest, and updating the radiance field so as to reduce the difference in pixel values. In this case, the initial value generated in S1205 is used in the first S1206 immediately after the start of processing. This initial value is set based only on pixels that are determined to be visible by a visibility determination described later.
[0041] S1208 and S1209 are the same as S407 and S408 in the flow of FIG. 4 in the first embodiment, and therefore a description thereof will be omitted.
[0042] The above is the flow of operations in the image processing device 120 according to this embodiment.
[0043] <Details of the initial value calculation process for the radiance field> Next, the process of calculating the initial value of the radiance field defined for each object will be described in detail with reference to the flowchart in Fig. 13. Each process of S1301 to S1303 is performed collectively for all objects.
[0044] In S1301, the density of the radiance field is initialized by assigning a value of σ=1 to each voxel in the fine voxel grid if it is inside the outline shape, σ=0 if it is outside, and σ=0.5 if it is on the surface.
[0045] In S1302, surface voxels are extracted from the schematic shape obtained in S1202. The extraction of the surface voxels is realized, for example, by selecting voxels that are located inside the schematic shape and adjacent to voxels located outside the schematic shape.
[0046] In S1303, the surface voxels of all objects extracted in S1302 are projected onto the image plane using the camera parameters of all shooting viewpoints of the multi-view image, and a depth map for all shooting viewpoints is generated. The generated depth map is stored in the RAM 102.
[0047] In S1304, the center coordinates of each surface voxel are projected to the shooting viewpoint, and the depth value (d') obtained by the projection is compared with the depth value (d) in the depth map to determine the visibility from each viewpoint. Specifically, it is determined to be visible (not occluded) when d'≦d+m. Here, "m" is a constant, and is a value one size larger than a voxel in the fine voxel grid, for example, 1 to 2 mm.
[0048] In S1305, for each surface voxel, color information of the radiance field is initialized based on the pixel values of the captured image corresponding to the viewpoint determined to have visibility. In Plenoxels, the different colors for each direction are expressed by the parameters of a spherical harmonic function for each RGB component. Therefore, for example, the initialization process is performed by setting the basis component of the spherical harmonic function (average of values in all directions) to the average value of the pixel values of the captured image corresponding to the viewpoint determined to have visibility, and setting the other components representing color changes for each direction to 0.
[0049] <Comparison with embodiment 1> 14A to 14D are diagrams for explaining learning when the method of this embodiment is applied, and correspond to FIGS. 10A to 10D of the first embodiment, respectively. As shown in FIGS. 14A to 14C, each of the three radiance fields 1401 to 1403 is optimized using only pixels where the corresponding object is visible. This enables updating without the influence of other radiance fields, enables optimization by processing multiple radiance fields in parallel, and can further reduce the amount of information contained in each radiance field. Furthermore, there is no overlap of pixels used for optimization between objects, so the amount of calculation is further reduced.
[0050] As described above, according to this embodiment, the radiance field is treated as a direct parameter by Plenoxels, and the initial value of each radiance field is calculated, and then each radiance field is updated, so that the optimization can be converged more quickly. In addition, by additionally performing visibility determination, the amount of calculation can be further reduced, and the learning speed can be significantly increased.
[0051] <Modification> In the first and second embodiments, the three-dimensional field is described by taking a radiance field in which density and a different color for each direction are associated with each coordinate in space as an example, but is not limited thereto. For example, the color information associated with the coordinate in space may be an isotropic color (color field) that does not depend on the direction. In addition, the three-dimensional field is not limited to the radiance field. For example, it may be an occupancy field that expresses volume density. It may also be a field expressed by a bidirectional reflectance distribution function (BRDF) that expresses the distribution characteristics of reflected light with respect to incident light. Furthermore, it may be a field that expresses the amount of ambient light entering (light visibility). As with the radiance field described in the first and second embodiments, these can be learned by inputting a multi-viewpoint image, and by inputting the camera parameters of the virtual viewpoint into each field after learning, the following virtual viewpoint data can be obtained.
[0052] Opacity field: A map that represents the opacity as seen from the virtual viewpoint Bidirectional reflectance distribution function field: A map showing the bidirectional reflectance distribution function as seen from a virtual viewpoint Ambient light penetration field: A map that represents the degree of visibility from the virtual viewpoint Furthermore, the three-dimensional field may be a floating-point field (Signed Distance Fielded) in which the inside of the object is negative and the outside is positive, or a binary field (Surface Fielded) in which the inside of the object is 0 and the outside is 1. Furthermore, it may be a field in the normal direction of the object surface (Normal Field). In these cases, learning is performed using a depth map corresponding to the multi-view images as input in addition to the multi-view images, and the following virtual viewpoint data is obtained based on each field after learning.
[0053] · Floating point field with negative inside and positive outside of the object: Depth map as seen from the virtual eye A binary field with 0 inside the object and 1 outside: a depth map as seen from a virtual viewpoint · Object surface normal field: Normal map as seen from the virtual viewpoint.
[0054] For example, the above-mentioned Surface Field is used in Pixel NeRF and Double Field. BRDF, Light Visibility, and Normal Field are used in NeRFactor. Signed Distance Field is used in NeuS. The method of the present disclosure can also be applied to these. So far, we have explained how to generate virtual viewpoint data based on virtual viewpoint parameters, but learning 3D fields is also effective for acquiring 3D shape data. For example, by extracting the occupancy field or signed distance field as voxels and applying the Marching Cubes method to the extracted voxels, mesh data can be acquired.
[0055] [Other embodiments] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.
[0056] The present disclosure also includes the following configurations and methods.
[0057] [Configuration 1] An image processing device that performs learning to obtain virtual viewpoint data corresponding to a virtual viewpoint or a three-dimensional shape from a plurality of captured images obtained by a plurality of imaging devices, An acquisition means for acquiring the plurality of photographed images; a generating means for generating schematic shape data representing a three-dimensional shape of an object based on the plurality of photographed images acquired by the acquiring means; a setting means for setting a learning area for each object based on the schematic shape data generated by the generating means; a learning means for learning a three-dimensional field corresponding to the captured image, using a learning area set for each object by the setting means as a target; An image processing device comprising:
[0058] [Configuration 2] The learning means includes: A storage means for storing the three-dimensional field; a rendering means for rendering an image corresponding to each of the plurality of photographed images and each of the photographed viewpoints having the same angle of view as each of the photographed images, based on camera parameters corresponding to each of the plurality of photographed images and the three-dimensional field stored in the storage means; an updating means for updating the three-dimensional field based on a plurality of drawn images obtained by the drawing means and the plurality of captured images; 2. The image processing device according to claim 1, further comprising:
[0059] [Configuration 3] The image processing device described in configuration 2, characterized in that the update means calculates the color difference between a captured image and a drawn image that correspond to each other, for a learning area of interest among the learning areas set for each object by the setting means, and updates the three-dimensional field so as to reduce the color difference.
[0060] [Configuration 4] 4. The image processing device according to configuration 3, wherein the updating means performs the updating by calculating the color difference for each pixel that corresponds between both images.
[0061] [Configuration 5] The image processing device according to configuration 4, wherein the updating means performs the updating by determining the color difference from pixels of the captured image that are visible to elements constituting the schematic shape data and pixels of the drawn image that correspond thereto.
[0062] [Configuration 6] 6. The image processing device according to configuration 5, wherein the update means determines an initial value of the three-dimensional field based on pixel values of a captured image from a viewpoint where the element is visible, and performs the update.
[0063] [Configuration 7] 7. The image processing device according to any one of configurations 1 to 6, wherein the setting means sets a solid body circumscribing the three-dimensional shape of the object represented by the outline shape data as the learning area.
[0064] [Configuration 8] the schematic shape data is data that specifies a three-dimensional shape of the object by a collection of a plurality of elements; 7. The image processing device according to any one of configurations 1 to 6, wherein the setting means sets, as the learning area, a three-dimensional area represented by a set of elements having a size larger than that of the element.
[0065] [Configuration 9] 9. The image processing device according to any one of configurations 1 to 8, wherein the generating means generates the shape data representing a rough shape of the object by a volume intersection method using the plurality of captured images.
[0066] [Configuration 10] the three-dimensional field is a radiance field that associates a volume density and an anisotropic color with each coordinate in the imaging space of the multiple imaging devices; The virtual viewpoint data is a virtual viewpoint image representing a view from a virtual viewpoint. 10. The image processing device according to any one of configurations 1 to 9.
[0067] [Configuration 11] the three-dimensional field is a radiance field that associates a volume density and an isotropic color with each coordinate in the imaging space of the multiple imaging devices; The virtual viewpoint data is a virtual viewpoint image representing a view from a virtual viewpoint. 10. The image processing device according to any one of configurations 1 to 9.
[0068] [Configuration 12] the three-dimensional field being an opacity field representing volume density; The virtual viewpoint data is a map representing opacity when viewed from a virtual viewpoint. 10. The image processing device according to any one of configurations 1 to 9.
[0069] [Configuration 13] the three-dimensional field being a bidirectional reflectance distribution function field; The virtual viewpoint data is a map representing a bidirectional reflectance distribution function as viewed from a virtual viewpoint. 10. The image processing device according to any one of configurations 1 to 9.
[0070] [Configuration 14] The three-dimensional field is a field of ambient light penetration, The virtual viewpoint data is a map representing the degree of visibility when viewed from a virtual viewpoint. 10. The image processing device according to any one of configurations 1 to 9.
[0071] [Configuration 15] the three-dimensional field being a floating-point field with negative values inside the object and positive values outside the object, or a binary field with 0s inside the object and 1s outside the object; The virtual viewpoint data is a depth map as viewed from a virtual viewpoint. 10. The image processing device according to any one of configurations 1 to 9.
[0072] [Configuration 16] the three-dimensional field being a field normal to a surface of an object; The virtual viewpoint data is a normal map as viewed from a virtual viewpoint. 10. The image processing device according to any one of configurations 1 to 9.
[0073] [Configuration 17] 17. An image processing device comprising an inference means for performing inference using a three-dimensional field learned by the image processing device according to any one of configurations 1 to 16, and outputting the virtual viewpoint data.
[0074] [Configuration 18] The inference means includes: a storage means for storing said learned three-dimensional field; a generating means for generating the virtual viewpoint data based on the three-dimensional field stored in the storage means in accordance with camera parameters of a virtual viewpoint; 18. The image processing device according to claim 17, further comprising:
[0075] [Method 1] An image processing method for performing learning to obtain virtual viewpoint data corresponding to a virtual viewpoint or a three-dimensional shape from a plurality of captured images obtained by a plurality of image capturing devices, comprising: An acquisition step of acquiring the plurality of photographed images; a generating step of generating schematic shape data representing a three-dimensional shape of an object based on the plurality of photographed images acquired in the acquiring step; a setting step of setting a learning area for each object based on the schematic shape data generated by the generating means; a learning step of learning a three-dimensional field corresponding to the captured image, using the learning area set for each object in the setting step as a target; 13. An image processing method comprising:
[0076] [Configuration 20] A program for causing a computer to function as each of the means of the image processing device according to any one of configurations 1 to 18.
Claims
1. An image processing device that performs learning to obtain virtual viewpoint data corresponding to a virtual viewpoint or a three-dimensional shape from a plurality of captured images obtained by a plurality of imaging devices, An acquisition means for acquiring the plurality of photographed images; a generating means for generating schematic shape data representing a three-dimensional shape of an object based on the plurality of photographed images acquired by the acquiring means; a setting means for setting a learning area for each object based on the schematic shape data generated by the generating means; a learning means for learning a three-dimensional field corresponding to the captured image, using a learning area set for each object by the setting means as a target; An image processing device comprising:
2. The learning means includes: A storage means for storing the three-dimensional field; a rendering means for rendering an image corresponding to each of the plurality of photographed images and each of the photographed viewpoints having the same angle of view as each of the photographed images, based on camera parameters corresponding to each of the plurality of photographed images and the three-dimensional field stored in the storage means; an updating means for updating the three-dimensional field based on a plurality of drawn images obtained by the drawing means and the plurality of captured images; 2. The image processing apparatus according to claim 1, further comprising:
3. The image processing device according to claim 2, characterized in that the updating means calculates the color difference between a captured image and a drawn image that correspond to each other, for a learning area of interest among the learning areas set for each object by the setting means, and updates the three-dimensional field so as to reduce the color difference.
4. 4. The image processing apparatus according to claim 3, wherein said updating means performs said updating by determining said color difference for each pixel that is in a corresponding relationship between both images.
5. 5. The image processing apparatus according to claim 4, wherein said updating means performs said updating by determining said color difference from pixels of said photographed image having visibility to elements constituting said schematic shape data and pixels of said drawn image corresponding thereto.
6. 6. The image processing apparatus according to claim 5, wherein said updating means performs said updating by determining an initial value of said three-dimensional field based on pixel values of a photographed image from a viewpoint where said element is visible.
7. 2. The image processing apparatus according to claim 1, wherein said setting means sets, as said learning area, a solid body circumscribing the three-dimensional shape of said object represented by said outline shape data.
8. the schematic shape data is data that specifies a three-dimensional shape of the object by a collection of a plurality of elements; 2 . The image processing apparatus according to claim 1 , wherein the setting means sets, as the learning area, a three-dimensional area represented by a set of elements having a size larger than that of the element.
9. 2. The image processing apparatus according to claim 1, wherein the generating means generates the shape data representing a rough shape of the object by a volume intersection method using the plurality of captured images.
10. the three-dimensional field is a radiance field that associates a volume density and an anisotropic color with each coordinate in the imaging space of the multiple imaging devices; The virtual viewpoint data is a virtual viewpoint image representing a view from a virtual viewpoint.
10. The image processing device according to claim 1,
11. the three-dimensional field is a radiance field that associates a volume density and an isotropic color with each coordinate in the imaging space of the multiple imaging devices; The virtual viewpoint data is a virtual viewpoint image representing a view from a virtual viewpoint.
2. The image processing device according to claim 1,
12. the three-dimensional field being an opacity field representing volume density; The virtual viewpoint data is a map representing opacity when viewed from a virtual viewpoint.
2. The image processing device according to claim 1,
13. the three-dimensional field being a bidirectional reflectance distribution function field; The virtual viewpoint data is a map representing a bidirectional reflectance distribution function as viewed from a virtual viewpoint.
2. The image processing device according to claim 1,
14. The three-dimensional field is a field of ambient light penetration, The virtual viewpoint data is a map representing the degree of visibility when viewed from a virtual viewpoint.
2. The image processing device according to claim 1,
15. the three-dimensional field being a floating-point field with negative values inside the object and positive values outside the object, or a binary field with 0s inside the object and 1s outside the object; The virtual viewpoint data is a depth map as viewed from a virtual viewpoint.
2. The image processing device according to claim 1,
16. the three-dimensional field being a field normal to a surface of an object; The virtual viewpoint data is a normal map as viewed from a virtual viewpoint.
2. The image processing device according to claim 1,
17. 2. An image processing apparatus comprising: an inference unit that performs inference using the three-dimensional field learned by the image processing apparatus according to claim 1 and outputs said virtual viewpoint data.
18. The inference means includes: a storage means for storing said learned three-dimensional field; a generating means for generating the virtual viewpoint data based on the three-dimensional field stored in the storage means in accordance with camera parameters of a virtual viewpoint; 18. The image processing apparatus according to claim 17, further comprising:
19. An image processing method for performing learning to obtain virtual viewpoint data corresponding to a virtual viewpoint or a three-dimensional shape from a plurality of captured images obtained by a plurality of image capturing devices, comprising: An acquisition step of acquiring the plurality of photographed images; a generating step of generating schematic shape data representing a three-dimensional schematic shape of an object based on the plurality of photographed images acquired in the acquiring step; a setting step of setting a learning area for each object based on the schematic shape data generated by the generating means; a learning step of learning a three-dimensional field corresponding to the captured image, using the learning area set for each object in the setting step as a target; 13. An image processing method comprising:
20. A program for causing a computer to execute the image processing method according to claim 19.
Citation Information
Patent Citations
Neural 3D Video Synthesis
US20220239844A1
Latency-Resilient Cloud Rendering
US20220277510A1
View synthesis robust to unconstrained image data
US11308659B2
Cited By
Information processing system, information processing method, and program
WO2026140904A1
