Image processing apparatus, image processing method, and program

Abstract

To provide an image processor capable of performing volume rendering at a high speed.SOLUTION: An image processing apparatus (100) includes an acquisition unit (105) configured to acquire a supervisory image and a position of a camera corresponding to the supervisory image, a ray calculation unit (106) configured to calculate a ray corresponding to each pixel of the supervisory image by using the position of the camera, and a learning parameter calculation unit (107) configured to perform machine learning by sampling a point in the ray and using the supervisory image and calculate a learning parameter, wherein a sampling density of the ray within a range of a depth of field of the supervisory image is higher than a sampling density outside the range of the depth of field.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing device that performs three-dimensional modeling using machine learning. [Background technology]

[0002] Conventionally, there are known techniques for performing three-dimensional modeling using images of an object captured from various angles. Patent Document 1 discloses a technique for generating, with a small amount of calculation, an image of the object viewed from an angle different from that at which the object was captured. However, since the color of light reflected from an object essentially changes depending on the viewing angle, the technique disclosed in Patent Document 1 may produce an unnatural feeling when the angle at which the image is reconstructed is changed.

[0003] Non-Patent Document 1 discloses a technology for reconstructing realistic images that look natural, like a real photograph, by taking into account the direction of light rays in addition to three-dimensional positions in space, sampling points on the light rays, and performing volume rendering. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2018-205863 [Patent Document 2] Japanese Patent Application Laid-Open No. 2008-15754 [Non-patent literature]

[0005] [Non-Patent Document 1] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng, “NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis”, In ECCV, 2020. [Non-patent document 2] TIANYE LI, MIRA SLAVCHEVA, MICHAEL ZOLLHOEFER, SIMON GREEN, CHRISTOPH LASSNER, CHANGIL KIM, TANNER SCHMIDT, STEVEN LOVEGROVE, MICHAEL GOESELE, ZHAOYANG LV, “Neural 3D Video Synthesis”, arXiv:2103.02597, 2021 Summary of the Invention [Problem to be solved by the invention]

[0006] However, the technique disclosed in Non-Patent Document 1 requires volume rendering by sampling points on a ray from one end of the target space to the other, which increases the amount of calculation and requires a long processing time.

[0007] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide an image processing device, an image processing method, and a program that are capable of performing volume rendering at high speed. [Means for solving the problem]

[0008] An image processing device as one aspect of the present invention comprises a teacher image, an acquisition means for acquiring the position of a camera corresponding to the teacher image, a ray calculation means for calculating a ray corresponding to each pixel of the teacher image using the position of the camera, and a learning parameter calculation means for performing machine learning by sampling points on the ray and using the teacher image to calculate learning parameters, wherein the sampling density of the ray within the depth of field range of the teacher image is higher than the sampling density outside the depth of field range.

[0009] Other objects and features of the present invention will be described in the following embodiments. [Effects of the Invention]

[0010] According to the present invention, it is possible to provide an image processing device, an image processing method, and a program that are capable of performing volume rendering at high speed. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram of a personal computer according to a first embodiment. [Figure 2] 4 is a flowchart of 3D model learning in the first embodiment. [Figure 3] FIG. 3 is an explanatory diagram of capturing a teacher image in the first embodiment. [Figure 4] FIG. 2 is an explanatory diagram of a teacher image and a focus map in the first embodiment. [Figure 5] 10 is a flowchart of free viewpoint image rendering in the first embodiment. [Figure 6] FIG. 2 is an explanatory diagram of a free viewpoint camera according to the first embodiment. [Figure 7] FIG. 2 is an explanatory diagram of a teacher image and a focus map in the first embodiment. [Figure 8] FIG. 4 is an explanatory diagram of subject depth calculation in the first embodiment. [Figure 9] FIG. 3 is an explanatory diagram of calculation of coordinates of a three-dimensional point in the first embodiment. [Figure 10] FIG. 2 is an explanatory diagram of a teacher image and a low-resolution focus map according to the first embodiment. [Figure 11] FIG. 11 is an explanatory diagram of a teacher image taken by a free viewpoint camera in the second embodiment. [Figure 12] FIG. 10 is an explanatory diagram of a peripheral teacher image and a focus map in the second embodiment. [Figure 13] 10A and 10B are explanatory diagrams of a teacher image and a focus map in the second embodiment. [Figure 14] FIG. 10 is a block diagram of a personal computer according to a third embodiment. [Figure 15] 10 is a flowchart of dynamic 3D model learning in the third embodiment. [Figure 16] FIG. 11 is an explanatory diagram of capturing a teacher image in the third embodiment. [Figure 17] FIG. 11 is an explanatory diagram of capturing a teacher image in the third embodiment. [Figure 18] 13A and 13B are explanatory diagrams of a teacher image and a focus map in the third embodiment. [Figure 19] 13A and 13B are explanatory diagrams of a teacher image and a focus map in the third embodiment. [Figure 20] 13 is a flowchart of free viewpoint video rendering in the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0013] (First embodiment) First, a personal computer (image processing device) according to a first embodiment of the present invention will be described with reference to Fig. 1. Fig. 1 is a block diagram of a personal computer (image processing device) 100. Note that this embodiment will be described using a personal computer as an example of an image processing device, but the present invention is not limited to this and can also be applied to image processing devices other than personal computers.

[0014] The control unit 101 is, for example, a CPU, and reads out an operation program for each block of the personal computer 100 from the ROM 102, expands it into the RAM 103, and executes it to control the operation of each block of the personal computer 100. The ROM 102 is a rewritable non-volatile memory such as an SSD, and stores the operation program for each block of the personal computer 100 as well as parameters and the like required for the operation of each block. The RAM 103 is a rewritable volatile memory such as a DRAM, and is used as a temporary storage area for data output in the operation of each block of the personal computer 100. The data storage unit 104 is a recording medium such as a hard disk that reads and writes image data required for machine learning, metadata for each image, and the like.

[0015] The shooting camera position and orientation estimation unit 105 estimates the position and orientation of the shooting camera for each image from the image data group using a known technique such as SfM (Structure from Motion). That is, the shooting camera position and orientation estimation unit 105 is an acquisition unit that acquires the teacher image and the position of the camera corresponding to the teacher image.

[0016] The ray calculation unit 106 calculates the rays used in volume rendering using a method such as that disclosed in Non-Patent Document 1. In Non-Patent Document 1, the rays used in volume rendering are defined as r(t) = o + td. Here, o is the camera's principal point in the world coordinate system, d is the ray's direction vector expressed in the world coordinate system, and t is the distance from the camera's principal point to the sampling point on the ray. The ray direction vector d is obtained by calculating a three-dimensional vector directed from the camera's principal point to each pixel on the image plane. Furthermore, the camera's principal point o and the ray direction vector d are coordinate-transformed from the camera coordinate system to the world coordinate system using the camera position and orientation parameters. In other words, the ray calculation unit 106 is a ray calculation unit that calculates rays corresponding to each pixel of the teacher image using the camera position during learning, and calculates rays corresponding to each pixel of the arbitrary viewpoint camera using the camera position and orientation during inference.

[0017] The neural network unit 107 samples points on each ray, calculates the color and density of the corresponding points using a neural network, and performs volume rendering across the target space to determine the color of the pixel corresponding to each ray. For learning and inference, a technique such as that disclosed in Non-Patent Document 1 may be used. During learning, the learning weights are converged using backpropagation, with the L2 loss between the color calculated by volume rendering and the color of the captured image used as a loss function. That is, during learning, the neural network unit 107 is a learning parameter calculation means that performs machine learning by sampling points on the ray and using a training image to calculate learning parameters. During inference, a free-viewpoint image is rendered by volume-rendering each ray at the free camera position and orientation. That is, during inference, the neural network unit 107 is a rendering means that renders the camera image through machine learning using learning parameters that have been pre-trained by sampling points on the ray using a learning parameter calculation means.

[0018] FΘ:(x,d)→(c,σ) (1) In equation (1), FΘ is a neural network consisting of a multi-layer perceptron. It takes as input the three-dimensional coordinate x of the point on the sampled ray and the ray direction vector d. FΘ outputs the RGB color c and density σ for each sampling point. Volume rendering for ray r is expressed as Cvr(r) = ΣTi(1-exp(-σi·δi))ci, i = 1 to N. Here, N is the number of samples, i is the index number for each sampling point, ci and σi are the color and density corresponding to index i, and δi = t(i+1)-t(i), where t(i) is the distance from the camera principal point corresponding to index i to the sampling point on the ray. Also, Ti = exp(-Σσj·δj), j = 1 to i-1, which eliminates the influence of the color of objects occluded by occlusion.

[0019] The free camera position and orientation acquisition unit 108 is an acquisition unit that acquires the camera position and orientation (position of the free viewpoint camera) for the free viewpoint image. It may acquire camera position and orientation parameters calculated in advance by an external device, or may acquire the camera position and orientation specified by the user via an operating member such as a joystick.

[0020] Next, 3D model learning by the control unit 101 will be described with reference to FIG. 2. FIG. 2 is a flowchart of 3D model learning. First, in step S201, the shooting camera position and orientation estimation unit 105 estimates the camera position and orientation corresponding to each image used for learning. Learning is performed by updating learning weights through iteration processing, thereby converging to target learning weights. Next, in step S202, the control unit 101 determines whether the iteration processing is complete. If the iteration processing is complete, this flow ends. On the other hand, if the iteration processing is not complete, the flow proceeds to step S203.

[0021] In step S203, the control unit 101 first randomly selects rays equal to the batch size in each iteration. The batch size may be set to, for example, 4096 rays as shown in Non-Patent Document 1. In this embodiment, the calculation cost can be efficiently reduced by selecting rays while excluding rays that are not suitable for 3D modeling. This operation will be described with reference to FIGS. 3 and 4.

[0022] FIG. 3 is an explanatory diagram of capturing a teacher image, showing a bird's-eye view to explain the relationship between the subject, the capturing space, and the capturing camera. 301 is the capturing camera (image capturing device), 302 is the range of the capturing space to be 3D modeled, 303 is the main subject, and 304 is the background subject. The capturing camera 301 is focused on the main subject 303, and is focused within the angle of view and depth of field indicated by the hatched area 305. Multiple capturing cameras 301 are placed to capture the subject within the capturing space range 302 from various directions, but for simplicity, only the capturing camera 301 is shown in FIG.

[0023] Fig. 4 is an explanatory diagram of a teacher image and a focus map (distance distribution information), showing an image 401 acquired by the photographing camera 301 and a focus map 402, which is accompanying metadata. In Fig. 4, the degree of focus on the imaging surface is shown in the form of a grayscale map, with white in the foreground, black in the background, and 50% gray indicating in-focus. The focus map 402 may be configured to acquire a defocus map on the imaging surface as a focus map from a phase difference image obtained from an imaging sensor in which all pixels are phase difference pixels, as disclosed in Patent Document 2, for example.

[0024] Rays 306 and 307 in FIG. 3 are rays to be subjected to volume rendering. However, since there is no subject within the depth of field on ray 306, it is excluded. Whether or not it is an unnecessary ray can be determined based on focus map 402 in FIG. 4. Pixel 404 corresponding to ray 307 is an in-focus pixel of 50% gray, while pixel 403 corresponding to ray 306 is dark gray outside the depth of field, and therefore it is determined to be a ray to be excluded. In this manner, by excluding unnecessary rays, the volume rendering process can be speeded up. In this embodiment, the acquisition means acquires distance distribution information (focus map) corresponding to the teacher image, and the ray calculation means determines whether or not the ray is within the depth of field of the teacher image based on the distance distribution information.

[0025] In this embodiment, light rays outside the depth of field are excluded, but this is not limited to this. The light ray calculation unit 106 may select light rays efficiently by varying the density of light rays inside and outside the depth of field (the light ray calculation means may select light rays within the depth of field of the teacher image with a focus on that light). For example, it is possible to select all light rays within the depth of field, and to select light rays outside the depth of field that are 10% or less of the total light rays of the target camera.

[0026] 2, rays for the batch size are selected, and then in step S204, the ray calculation unit 106 calculates rays. As described above, the rays for volume rendering are defined as r(t) = o + td, where o is the principal point of the camera in the world coordinate system, d is the direction vector of the ray expressed in the world coordinate system, and t is the distance from the camera principal point to the sampling point on the ray.

[0027] The range of distance t sampled for volume rendering is limited within the depth of field indicated by the hatched area 305 in FIG. 3. The depth range of the hatched area 305 is expressed as Z-Df to Z+Db, where Df is the front depth of field, Db is the rear depth of field, and Z is the in-focus object distance. Df = (r·Av·Z^2) / (f^2+r·Av·Z) and Db = (r·Av·Z^2) / (f^2-r·Av·Z). Here, r is the permissible circle of confusion diameter, Av is the aperture value, and f is the focal length. The permissible circle of confusion diameter r is twice the pixel pitch. By limiting the range of distance t sampled for volume rendering within the depth of field, the volume rendering process can be sped up.

[0028] In this embodiment, the permissible circle of confusion diameter r is set to twice the pixel pitch, but is not limited to this and may be set to a larger or smaller value depending on the resolution used to render the free viewpoint image. In other words, the permissible circle of confusion diameter for determining the depth of field may be determined based on the resolution used to render using the learning parameters (rendering resolution).

[0029] In this embodiment, the depth of field is defined as Z-Df to Z+Db. However, this is not limiting. Taking into account errors in the focal length f and aperture value Av obtainable from the camera, a more generous range, such as Z-2·Df to Z+2·Db, may be used. In this embodiment, the ray range for volume rendering is limited to the depth of field. However, this is not limiting. Efficient sampling may be achieved by varying the sampling density within and outside the depth of field. That is, the sampling density of the ray within the depth of field of the teacher image may be higher than the sampling density outside the depth of field. For example, for the ray 307 in FIG. 3, 32 points are first sampled from the entire range covered by the 3D modeling target range 302, and an additional 128 points are sampled in the range covered by the hatched area 305. In this manner, sampling points may be densely arranged only within the depth of field.

[0030] 2, after the rays are calculated in step S204, the neural network unit 107 updates the learning weights in step S205. The control unit 101 determines the learning weights by repeating steps S202 to S205 until the learning weights converge. Note that the determination of the completion of the iterations in step S202 may be made based on whether or not the number of iterations reaches 100-300K, as disclosed in Non-Patent Document 1, for example.

[0031] Next, free viewpoint image rendering by the control unit 101 will be described with reference to Fig. 5. Fig. 5 is a flowchart of free viewpoint image rendering. First, in step S501, the free camera position and orientation acquisition unit 108 acquires the camera position and orientation of the free viewpoint to be rendered. Next, in step S502, the control unit 101 determines whether or not calculation of pixel values ​​(RGB values) by volume rendering has been completed for all pixels of the rendered image. If calculation of pixel values ​​has been completed for all pixels, this flow ends. On the other hand, if calculation of pixel values ​​has not been completed for all pixels, the process proceeds to step S503.

[0032] In step S503, the control unit 101 determines whether the corresponding 3D point for each pixel is within the depth of field of the training image, i.e., whether the ray is within the depth of field. If the 3D point is not within the depth of field, the process returns to step S502. On the other hand, if the 3D point is within the depth of field, the process proceeds to step S503, where volume rendering is performed. Note that a fixed pixel value, such as black, is assigned to pixels corresponding to rays outside the depth of field.

[0033] Fig. 6 is an explanatory diagram of a free viewpoint camera, showing a bird's-eye view illustrating the relationship between the subject, the shooting space, and each camera. Fig. 7 is an explanatory diagram of a teacher image and a focus map, showing an image 701 acquired by a shooting camera 601 and a focus map 702, which is accompanying metadata. In Fig. 6, 603 is the free viewpoint camera that performs rendering, and 301 and 601 are shooting cameras adjacent to the free viewpoint camera 603.

[0034] Pixel 404 in Fig. 4 and pixel 704 in Fig. 7 represent the same 3D point 607 in Fig. 6. Furthermore, ray 605 from free viewpoint camera 603 and ray 307 from shooting camera 301 represent the same 3D point 607 in Fig. 6. If the 3D coordinates of 3D point 607 are known in advance, it can be confirmed that ray 605 corresponds to ray 307, and this can be found from focus map 402 in Fig. 4.

[0035] 8 is an explanatory diagram of subject depth calculation, showing the procedure for calculating the subject depth Z+ΔZ from the defocus value def for each pixel position indicated by the focus map. 801 is the imaging optical system, 802 is the imaging plane position, 803 is the focused subject distance position, 804 is the defocused image position, and 805 is the subject distance position. From the lens formula, 1 / Z+1 / Z'=1 / f and 1 / (Z+ΔZ)+1 / (Z'+def)=1 / f hold, and from these, the subject depth Z+ΔZ can be calculated (f: focal length).

[0036] If the depth Z+ΔZ of the subject is known, then x / f=X / (Z+ΔZ) and y / f=Y / (Z+ΔZ) hold true from the geometric relationship of the triangle shown in Figure 9, which is an explanatory diagram of calculating the coordinates of a three-dimensional point. From these, the X coordinate, Y coordinate, and Z coordinate (Z+ΔZ) of the three-dimensional point 607 can be found.

[0037] From the above, it can be confirmed that ray 605 corresponds to ray 307. Since it is known that ray 307 during training is within the depth of field, ray 605 is subject to volume rendering. On the other hand, pixel 403 in FIG. 4 and pixel 703 in FIG. 7 represent the same 3D point, 606 in FIG. 6, and correspond to ray 306 and ray 604 in FIG. 6, respectively. However, since it is known that ray 306 during training is outside the depth of field and the photographing camera 601 did not capture the corresponding 3D point 606, ray 604 is excluded from volume rendering. In this way, by excluding rays corresponding to 3D points that have not been trained from volume rendering, it is possible to speed up the volume rendering process for free viewpoint images.

[0038] For the ray determined to be within the depth of field in step S503 of FIG. 5, the ray calculation unit 106 calculates the ray in step S504. As described above, the ray to be volume rendered is defined as r(t) = o + td. Here, o is the principal point of the camera in the world coordinate system, d is the direction vector of the ray expressed in the world coordinate system, and t is the distance from the camera principal point to the sampling point on the ray. The range of the distance t sampled for volume rendering is limited to the depth of field of the adjacent photographing cameras 301 and 601, as indicated by the hatched areas 305 and 602 in FIG. 6. In this way, by limiting the range of the distance t sampled for volume rendering to the depth of field of the adjacent photographing cameras, the volume rendering process of the free viewpoint image can be accelerated.

[0039] After the rays are calculated in step S504 in FIG. 5, in step S505, the neural network unit 107 executes volume rendering processing and calculates corresponding pixel values ​​(RGB values).

[0040] In this embodiment, the image (teacher image) from the capturing camera and the focus map (distance distribution information) have the same resolution, but this is not limiting and they may have different resolutions. For example, the resolution of the distance distribution information map may be lower than that of the teacher image. A map generated based on a disparity map, such as a focus map, is typically reduced in size by the template size because template matching of a predetermined size is performed for stereo correspondence point search. For example, if the template size is 16 × 16 pixels, the normal map size is 1 / 16 the size in both height and width. A focus map is also required when rendering a free viewpoint image. However, if the focus map stored at this time is a reduced version that is 1 / 16 the size in both height and width, the data volume required for rendering can be reduced.

[0041] 10 is an explanatory diagram of a teacher image and a low-resolution focus map, showing an image 401 acquired by the photographing camera 301 and a focus map 1001, which is accompanying metadata. The focus map 1001 has a lower resolution by the template size (16×16 pixels) for template matching, and one pixel on the focus map corresponds to a 16×16 pixel area of ​​the image. Also, as described with reference to FIG. 3, 303 is the main subject and 304 is the background subject.

[0042] Pixel 1003 on image 401 corresponds to pixel 1002 on focus map 1001. Pixel 1003 indicates main subject 303, but both main subject 303 and the ground, which is the background, are included within the 16 x 16 pixel range of the template. For this reason, the focus map pixel value (defocus value) of pixel 1002 may fall between the defocus values ​​of main subject 303 and the ground. Therefore, for subject contour regions such as pixel 1003, rays are selected broadly, and the sampling range for volume rendering is not limited. In other words, the ray calculation means identifies the subject contour region based on distance distribution information, makes it easier to select rays for the subject contour region, and sets a wide sampling range.

[0043] In this embodiment, distortion due to the lens of the photographing camera is assumed to be very slight, and distortion correction is not performed on the training images. However, this is not limited to this, and training images with distortion may be used. In this case, distortion correction is performed on the training images so that correct light rays can be calculated. Furthermore, distortion correction is also performed on the focus map referenced in pairs with the images, so that correct light rays can be selected. In other words, the acquisition means may process the teacher images and distance distribution information based on the distortion component of the camera's optical system.

[0044] In this embodiment, the pixel values ​​of the focus map stored in the data storage unit 104 are defocus values, but are not limited to this and may be disparity values ​​or distance values. Any format may be used as long as it can be converted into distance values ​​when determining the range of volume rendering. When storing defocus values, the imaging camera may correct focus errors due to lens decentering or tilt of the imaging element beforehand, and the defocus values ​​may be recorded as a focus map and stored in the data storage unit 104. In other words, the distance distribution information may include at least one of a map based on a shift amount representing disparity (disparity map), a map based on a defocus amount (defocus map), or a map based on distance (distance map).

[0045] In this embodiment, as shown in FIG. 3, the range of volume rendering is the hatched area 305 defined by the front depth of field and the rear depth of field, but is not limited thereto. For example, it is possible to determine whether the object surface is in front of or behind the object from the focus map, and further restrict the range of volume rendering. That is, it is possible to determine whether the object surface is in front of (in front of) or behind (behind) the object focus distance based on the distance distribution information (the sign of the focus map), and change the sampling density based on the determination result. For example, for object 303 in FIG. 3, since it is known from the focus map that the object surface is in front, the range of volume rendering may be restricted to only the range of the front depth of field.

[0046] In this embodiment, the volume rendering range is determined based on the focus-object distance, but this is not limiting. For example, a predetermined range may be set as the volume rendering range based on the object surface position. That is, the distance to the object surface may be determined based on the distance distribution information, and the sampling density may be determined based on the distance to the object surface. The depth position of the object surface may be determined using the method described with reference to FIG. 8.

[0047] In this embodiment, volume rendering is performed using only rays within the depth of field during free viewpoint image rendering (step S503), so pixels corresponding to rays outside the depth of field are assigned a fixed pixel value, such as black, but this is not limited to this. Only pixels outside the depth of field, such as the background, may be rendered using a different method. For example, a spherical environmental texture may be applied to the background, or the background may be rendered from a completely different background 3D model.

[0048] In this embodiment, the focus map can be evaluated over the entire screen, but this is not limited to this. A low-reliability region where the focus map cannot be calculated may be defined to prevent problems in post-processing. In such a case, all target rays may be selected for the low-reliability region. That is, the ray calculation means may select all rays for a region (low-reliability region) determined to have low reliability based on the distance distribution information, and set a wide sampling range.

[0049] According to this embodiment, it is possible to appropriately limit the rays and sampling range for volume rendering, thereby speeding up the volume rendering process.

[0050] (Second embodiment) Next, an image processing device according to a second embodiment of the present invention will be described. In this embodiment, the acquisition means further acquires a peripheral teacher image of the peripheral space (a space including a background subject), and the learning parameter calculation means performs machine learning using the teacher image and the peripheral teacher image to calculate learning parameters. The configuration of the image processing device according to this embodiment is the same as the configuration of the personal computer 100 described in the first embodiment with reference to FIG. 1. In addition, the flowcharts for explaining the operation of 3D model learning shown in FIG. 2 and the operation of free viewpoint image rendering shown in FIG. 5, which are executed by the control unit 101 in FIG. 1, are also the same as those in the first embodiment.

[0051] FIG. 11 is an explanatory diagram of the capture of a teacher image and the free-viewpoint camera, showing a bird's-eye view illustrating the relationship between the subject, capture space, capture camera, and free-viewpoint camera. Reference numerals 1101 and 1105 denote the capture cameras, 1103 the free-viewpoint camera, 302 the range of the capture space for the 3D modeling target, 303 the main subject, and 304 the background subject. The capture camera 1101 has a wide-angle focal length and is focused on the background subject 304 within the angle of view and depth of field indicated by the hatched area 1102. The capture camera 1105 has a standard-angle focal length and is focused on the main subject 303 within the angle of view and depth of field indicated by the hatched area 1106.

[0052] In this embodiment, in order to photograph the subject within the photographing space range 302 from various directions, multiple photographing cameras are arranged in addition to the photographing cameras 1101 and 1105. Specifically, multiple photographing cameras having similar focal lengths and shooting distances (focus positions) to the photographing camera 1105 are arranged to photograph the main subject 303. Additionally, multiple photographing cameras having similar focal lengths and shooting distances (focus positions) to the photographing camera 1101 are arranged to photograph subjects other than the main subject 303. For simplicity, only the photographing cameras 1101 and 1105 are shown in FIG. 11.

[0053] 12 and 13 are explanatory diagrams of peripheral teacher images or teacher images and focus maps, showing images 1201 and 1301 captured by photographing cameras 1101 and 1105, and focus maps 1202 and 1302, which are accompanying metadata. The focus maps are calculated using known technology, as in the first embodiment.

[0054] Because the shooting camera 1101 is focused on the background subject 304 at a wide-angle focal length, multiple subjects are captured over a wide range as shown in image 1201, and the background subject 304 shows 50% gray in focus as shown in focus map 1202. On the other hand, because the shooting camera 1105 is focused on the main subject 303 at a standard-angle focal length, the main subject is mainly captured as shown in image 1301, and the main subject 303 shows 50% gray in focus as shown in focus map 1302.

[0055] With the above configuration, when selecting rays for the batch size (S204) in the 3D model learning of Fig. 2, ray 1111 corresponding to 3D point 606 on background subject 304 is not selected because it is outside the depth of field when photographing camera 1105 with a standard angle of view in Fig. 11. On the other hand, ray 1103 of wide-angle photographing camera 1101 is selected as a target ray because 3D point 606 on background subject 304 is within the depth of field, and not only main subject 303 but also background subject 304 can be included in the 3D model.

[0056] 5, it is possible to determine that the corresponding 3D point 607 of the light ray 1110 from the free viewpoint camera 1108 in FIG. 11 is captured within the depth of field of the light ray 1107 of the adjacent photographing camera 1105, and therefore the light ray 1110 is selected as the target light ray. It is possible to determine that the corresponding 3D point 606 of the light ray 1109 from the free viewpoint camera 1108 in FIG. 11 is captured within the depth of field of the light ray 1103 of the adjacent photographing camera 1101, and therefore the light ray 1109 is selected as the target light ray. In this way, it is possible to render free viewpoint images not only for the main subject 303 but also for the background subject 304.

[0057] In this embodiment, the imaging cameras 1101 and 1105 are configured to have different focal lengths and shooting distances so that both the main subject and background subject can be volume rendered, but this is not limiting and only the shooting distances may be different. Increasing the focal length of the imaging camera 1101 for the background subject narrows the angle of view, which may result in an increase in the number of imaging cameras required. In addition, shortening the focal length of the imaging camera 1105 for the main subject widens the angle of view, which requires the camera to move closer to the subject, and there is a possibility that the imaging camera 1105 will be reflected in the imaging camera 1101. On the other hand, matching the focal lengths has advantages in terms of preparation for imaging and camera calibration, making this method suitable for simple imaging.

[0058] In this embodiment, the imaging cameras 1101 and 1105 are configured to have different focal lengths and shooting distances to enable volume rendering of both the main subject and background subjects. However, this is not limited to this; instead, different aperture values ​​may be used. That is, the imaging camera 1105 for the main subject is set to a bright aperture value near full aperture, while the imaging camera 1101 for the background subject is set to a narrow aperture value so that the entire imaging space range 302 is within the depth of field. This allows the imaging camera 1105 for the main subject to select a slow shutter speed, allowing for sharp images even when the main subject is moving. However, the imaging camera 1101 for the background subject may be weak against moving subjects or may have increased imaging sensitivity, resulting in increased noise. On the other hand, in cases where multiple background subjects are scattered within the imaging space range 302, this has the advantage of allowing 3D models to be generated without missing any of the subjects. In this way, the peripheral training images need only differ from the training images in at least one of the focus position (shooting distance), focal length, and aperture value.

[0059] In this embodiment, a focus map is used to speed up volume rendering even when rendering free-viewpoint images, but this is not limited to this. For example, a focus map may be used to speed up 3D model learning, while free-viewpoint image rendering may be performed by spatial sampling across the entire shooting space range as in conventional technology. This eliminates the need for a focus map when rendering free-viewpoint images. This allows renderers based on conventional technology to render free-viewpoint images, thereby increasing the versatility of the renderer.

[0060] In this embodiment, the main subject and background subjects are 3D modeled using the same neural network, but this is not limited to this. Separate neural networks may be configured for the main subject and background subjects. In other words, the learning parameter calculation means may train different machine learning models for the teacher image and the peripheral teacher image. This is expected to improve learning efficiency and the reproducibility of details of the background subject.

[0061] According to this embodiment, it is possible to select the optimal light rays for volume rendering for the main subject and background subjects, and it is possible to quickly generate 3D models including not only the main subject but also the background subjects, and to quickly render free viewpoint images.

[0062] (Third embodiment) Next, an image processing device according to a third embodiment of the present invention will be described. Fig. 14 is a block diagram of a personal computer (image processing device) 1400. The personal computer 1400 of this embodiment differs from the personal computer 100 having the neural network unit 107 described in the first embodiment in that it has a neural network unit 1401. Note that other configurations of the personal computer 1400 are similar to those of the personal computer 100, and therefore description thereof will be omitted.

[0063] The neural network unit 1401 can handle dynamic scenes (videos) where the subject is moving. In such dynamic scenes, points on each ray are sampled, and the color and density of the corresponding points are calculated using a neural network. Volume rendering is then performed across the target space to determine the color of the pixel corresponding to each ray. For learning and inference, the technology disclosed in Non-Patent Document 2, for example, can be used. During learning, the learning weights are converged using backpropagation with the L2 loss between the color calculated by volume rendering and the color of the captured image as a loss function. During inference, a free-viewpoint image is rendered by volume-rendering each ray at a free camera position and orientation.

[0064] FΘ:(x,d,zt)→(c,σ) ···(2) In equation (2), FΘ is a neural network consisting of a multi-layer perceptron, and inputs the three-dimensional coordinates x of the point on the sampled ray, the ray direction vector d, and the latent code zt in the frame at time t. FΘ outputs the RGB color c and density σ for each sampling point.

[0065] The neural network unit 107 of the first embodiment represents only a single 3D model, whereas the neural network unit 1401 of the present embodiment can represent a scene in which the 3D shape changes dynamically from frame to frame by changing the latent code zt for each frame.

[0066] Next, dynamic 3D model learning by the control unit 101 will be described with reference to Fig. 15. Fig. 15 is a flowchart of dynamic 3D model learning. First, in step S201, the shooting camera position and orientation estimation unit 105 estimates the camera position and orientation corresponding to each image used for learning. Because the position of the shooting camera is fixed, the camera position and orientation are estimated using the image acquired in the first frame.

[0067] Next, in step S1501, the control unit 101 determines whether or not processing for all frames has been completed. If processing for all frames has been completed, this flow ends. On the other hand, if processing for all frames has not been completed, the process proceeds to step S1502. In step S1502, the control unit 101 generates a latent code corresponding to each frame.

[0068] Next, in step S1503, the control unit 101 determines whether the iteration process is complete. Learning for each frame is performed by updating the learning weights through the iteration process, so that the learning weights converge to the target learning weights. If the iteration process is complete, the process returns to step S1501. On the other hand, if the iteration process is not complete, the process proceeds to step S1504.

[0069] In step S1504, the control unit 101 first randomly selects rays equal to the batch size in each iteration. Non-Patent Document 2 discloses a technique for efficiently reducing computational costs by performing ray importance sampling, which selects the next ray for learning based on temporal fluctuations in the input video. In this embodiment, ray importance sampling is performed taking into account the depth of field of the input video, thereby achieving further computational efficiency. The operation will be described below with reference to FIGS. 16 to 19.

[0070] FIG. 16 is an explanatory diagram of capturing a teacher image, showing a bird's-eye view illustrating the relationship between the subject, shooting space, and shooting camera in the first frame of the input video. 1601 is the shooting camera, 302 is the range of the shooting space to be 3D modeled, 303 is the main subject, and 304 is the background subject. The shooting camera 1601 is focused on the main subject 303, and the entire shooting space range 302 is in focus within the wide-angle field of view and pan-focus depth of field indicated by the hatched area 1605. Multiple shooting cameras 1601 are positioned to capture the subject within the shooting space range 302 from various directions, but for simplicity, only the shooting camera 1601 is shown in FIG. 16.

[0071] In subsequent frames, the shooting camera 1601 changes the focal length to a standard angle of view and the aperture to a brighter, wider aperture (F-number), thereby changing the camera parameters so that the main subject 303 can be captured with higher resolution. Figure 17 is an explanatory diagram of capturing a teacher image, showing a bird's-eye view illustrating the relationship between the subject, shooting space, and shooting camera after the camera parameters have been changed. The shooting camera 1601 is focused on the main subject 303, and the shooting space range 302 is partially in focus within the standard angle of view and shallow depth of field indicated by the hatched area 1705.

[0072] Fig. 18 is an explanatory diagram of a teacher image and a focus map, showing an image 1801 acquired by the shooting camera 1601 in the first frame of the input video and a focus map 1802, which is associated metadata. In Fig. 18, the degree of focus on the imaging surface is shown in the form of a grayscale map, with white in the foreground, black in the background, and 50% gray indicating in-focus. Fig. 19 is an explanatory diagram of a teacher image and a focus map, showing an image 1901 acquired by the shooting camera 1601 after a camera parameter change, and a focus map 1902, which is associated metadata.

[0073] In the first frame of the input video, rays 1606 and 1607 in Figure 16 are rays to be subjected to volume rendering, and the corresponding 3D points are within the depth of field, so both rays are selected. On the other hand, rays 1706 and 1707 in Figure 17 after the camera parameters are changed are also rays to be subjected to volume rendering, but since there is no object within the depth of field on ray 1706, it is excluded. Whether or not a ray is unnecessary can be determined using focus map 1902 in Figure 19. Pixel 1904 corresponding to ray 1707 is an in-focus pixel that is 50% gray, while pixel 1903 corresponding to ray 1706 is a dark gray pixel outside the depth of field, so it is determined to be a ray to be excluded.

[0074] After selecting rays for the batch size in step S1504 of FIG. 15, the ray calculation unit 106 calculates rays in step S1505. As described above, the ray for volume rendering is defined as r(t) = o + td. Here, o is the principal point of the camera in the world coordinate system, d is the direction vector of the ray expressed in the world coordinate system, and t is the distance from the camera principal point to the sampling point on the ray. The range of distance t sampled for volume rendering is limited within the depth of field indicated by the hatched area 1605 in FIG. 16 and the hatched area 1705 in FIG. 17. In this way, by performing ray importance sampling while taking the depth of field into consideration, the volume rendering process can be accelerated.

[0075] In this embodiment, the sampling range t is limited in accordance with the depth of field even in the case of Fig. 16 where the depth of field is deep, but this is not limiting. In such a case, the effect of limiting the sampling range is low, so a sufficiently wide fixed range may be set to fix the sampling range.

[0076] After calculating the rays in step S1505 in Fig. 15, the neural network unit 1401 updates the learning weights in step S1506. The operations of steps S1503 to S1506 are repeated until the learning weights converge, thereby determining the learning weights. When processing of all frames of the input video is completed, this flow ends (step S1501).

[0077] In this embodiment, ray importance sampling is performed based on the depth of field, but the present invention is not limited to this. Ray importance sampling may also be performed based on both the depth of field and the temporal variation of the input video. In many cases, the input video has a higher resolution than the focus map. Therefore, by observing the temporal variation of the input video, it is possible to remove unnecessary rays with higher resolution and then limit the sampling range based on the focus map, which is expected to further improve computational efficiency.

[0078] Next, free viewpoint image rendering by the control unit 101 will be described with reference to Fig. 20. Fig. 20 is a flowchart of free viewpoint image rendering in this embodiment. First, in step S501, the free camera position and orientation acquisition unit 108 acquires the camera position and orientation of the free viewpoint to be rendered.

[0079] Next, in step S2001, the control unit 101 determines whether processing for all frames has been completed. If processing for all frames has been completed, this flow ends. On the other hand, if processing for all frames has not been completed, the flow proceeds to step S2002, where pixel values ​​(RGB values) are calculated by volume rendering for all pixels of the rendered image while updating the latent code Zt for each frame. That is, in step S2002, the control unit 101 generates a latent code corresponding to each frame. Next, in step S2003, the control unit 101 determines whether calculation of pixel values ​​(RGB values) by volume rendering has been completed for all pixels of the rendered image. If calculation of pixel values ​​for all pixels has been completed, the flow returns to step S2001. On the other hand, if calculation of pixel values ​​for all pixels has not been completed, the flow proceeds to step S2004.

[0080] In step S2004, the ray calculation unit 106 calculates a ray corresponding to each pixel (S2004). As described above, the ray for volume rendering is defined as r(t) = o + td. Here, o is the principal point of the camera in the world coordinate system, d is the direction vector of the ray expressed in the world coordinate system, and t is the distance from the camera principal point to the sampling point on the ray. After the ray is calculated in step S2004, in step S2005, the neural network unit 107 executes volume rendering processing and calculates the corresponding pixel value (RGB value).

[0081] In this embodiment, the focal length and depth of field are changed between the beginning and subsequent portions of the input video, but the present invention is not limited to this. For example, while shooting with a standard angle of view and a shallow depth of field, multiple shots with a wide angle and deep focus may be inserted. This makes it possible to deal with gradual changes in the surrounding environment, such as sunlight, during the course of a long input video.

[0082] In this embodiment, the focal length and depth of field are changed between the beginning and subsequent portions of the input video, but this is not limited to this. It is also possible to control only the aperture and change only the depth of field. This makes it possible to shoot with a prime lens without a zoom mechanism. Alternatively, multiple prime cameras with wide-angle and standard angles of view may be arranged. Furthermore, with a zoom lens, changing the focal length from a standard angle of view to a wide angle while keeping the aperture open will deepen the depth of field, so if this is sufficient, it is also possible to control only the focal length.

[0083] In this embodiment, the focal length and depth of field are changed between the beginning and end of the input video, but this is not limited to this. The focus may be gradually changed from the background to the main subject. This allows not only the main subject but also other subjects to be captured in high definition. Changing the focus also changes the image magnification, so by further correcting the change in image magnification, the 3D model can be trained with higher accuracy.

[0084] As described above, in this embodiment, the teacher image is a video captured at a predetermined frame rate. The learning parameter calculation means calculates, for each frame of the video, a chord feature (latent code) that indicates the characteristics of that frame, and performs machine learning using the chord feature and the teacher image for each frame of the video to calculate learning parameters. In this embodiment, the acquisition means may acquire a time-varying region in the teacher image (a region where the appearance changes significantly over time), and the ray calculation means may focus on selecting ray beams in the region acquired by the acquisition means and output the corresponding ray beams. Also, in this embodiment, the teacher image is a video captured so as to include frames that differ in at least one of focus position, focal length, and aperture value.

[0085] According to this embodiment, volume rendering can be performed efficiently even in dynamic scenes where the subject is moving, making it possible to generate a 3D model at high speed.

[0086] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0087] According to each embodiment, by appropriately limiting the light rays and sampling range for volume rendering, it is possible to measure the detailed shape of an object at high speed and reconstruct a photorealistic image. Therefore, according to each embodiment, it is possible to provide an image processing device, an image processing method, and a program that are capable of performing volume rendering at high speed.

[0088] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0089] 100 Personal Computer (Image Processing Device) 105 Shooting camera position and orientation estimation unit (acquisition means) 106 ray calculation unit (ray calculation means) 107 Neural network unit (learning parameter calculation means, rendering means) 108 Free camera position and orientation acquisition unit (acquisition means)

Claims

1. an acquisition means for acquiring a teacher image and a camera position corresponding to the teacher image; a ray calculation means for calculating a ray corresponding to each pixel of the teacher image using the position of the camera; a learning parameter calculation means for performing machine learning by sampling points on the light ray and using the teacher image, and calculating learning parameters; An image processing device, characterized in that the sampling density of the light rays within the range of the depth of field of the teacher image is higher than the sampling density outside the range of the depth of field.

2. An acquisition means for acquiring the position of the camera; a ray calculation means for calculating a ray corresponding to each pixel of the camera using the position of the camera; a rendering means for rendering an image of the camera by machine learning using learning parameters that have been learned in advance by sampling points on the light ray by a learning parameter calculation means, An image processing device characterized in that the sampling density of the light rays within the range of the depth of field of the teacher image used in the pre-learning is higher than the sampling density outside the range of the depth of field.

3. the acquiring means further acquires distance distribution information corresponding to the teacher image, 3. The image processing apparatus according to claim 1, wherein the ray calculation means determines whether the ray is within the depth of field of the teacher image based on the distance distribution information.

4. It is determined whether the object surface is closer to or further from the object focus distance based on the distance distribution information, 4. The image processing apparatus according to claim 3, wherein the sampling density is changed based on a determination result as to whether the surface of the object is closer to or further from the object focus distance.

5. a distance to a surface of the object is determined based on the distance distribution information; 5. The image processing apparatus according to claim 3, wherein the sampling density is determined based on the distance to the surface of the subject.

6. 6. The image processing apparatus according to claim 3, wherein the distance distribution information has a resolution lower than that of the teacher image.

7. 9. The image processing device according to claim 3, wherein the ray calculation means identifies a subject contour area based on the distance distribution information, makes it easier to select rays for the subject contour area, and sets a wide sampling range.

8. 8. The image processing device according to claim 3, wherein the ray calculation means selects all rays for an area determined to have low reliability based on the distance distribution information, and sets a wide sampling range.

9. 9. The image processing apparatus according to claim 3, wherein the acquisition means processes the teacher image and the distance distribution information based on a distortion component of the optical system of the camera.

10. 10. The image processing device according to claim 3, wherein the distance distribution information includes at least one of a map based on a shift amount representing parallax, a map based on a defocus amount, and a map based on distance.

11. The acquisition means further acquires a peripheral teacher image of the peripheral space, 11. The image processing device according to claim 1, wherein the learning parameter calculation means performs the machine learning by using the teacher image and the peripheral teacher images, and calculates the learning parameters.

12. The image processing apparatus according to claim 11 , wherein the peripheral teacher image is an image that differs from the teacher image in at least one of focus position, focal length, and aperture value.

13. 13. The image processing apparatus according to claim 11, wherein the learning parameter calculation means learns different machine learning models for the teacher image and the peripheral teacher images.

14. the teacher image is a video captured at a predetermined frame rate, The learning parameter calculation means calculating a code feature that indicates a feature of each frame of the video; The image processing device according to any one of claims 1 to 13, characterized in that machine learning is performed by using the chord features and the teacher image for each frame of the video, and the learning parameters are calculated.

15. the acquiring means acquires a region that changes over time in the teacher image, 15. The image processing apparatus according to claim 14, wherein the light ray calculation means selects light rays intensively from the area acquired by the acquisition means, and outputs corresponding light rays.

16. 16. The image processing apparatus according to claim 14, wherein the teacher image is a moving image captured so as to include frames that differ in at least one of focus position, focal length, and aperture value.

17. 17. The image processing device according to claim 1, wherein the permissible circle of confusion diameter for determining the depth of field is determined based on a resolution at the time of rendering using the learning parameters.

18. acquiring a teacher image and a camera position corresponding to the teacher image; calculating a ray corresponding to each pixel of the teacher image using the position of the camera; performing machine learning by sampling points on the ray and using the training image to calculate learning parameters; An image processing method, characterized in that the sampling density of the light rays within the depth of field range of the teacher image is higher than the sampling density outside the depth of field range.

19. obtaining a camera position; calculating a ray corresponding to each pixel of the camera using the position of the camera; and rendering the image of the camera by machine learning using learning parameters pre-trained by sampling points on the light ray by a learning parameter calculation means; An image processing method characterized in that the sampling density of the light rays within the depth of field range of the teacher image used in the pre-learning is higher than the sampling density outside the depth of field range.

20. 20. A program causing a computer to execute the image processing method according to claim 18 or 19.

Images