Image processing device, image processing method, and program

Abstract

The present technology relates to an image processing device, an image processing method, and a program that make it possible to accurately estimate an opacity image. A candidate image generation unit, using an N-viewpoint captured image and shape data indicating the shape of a 3D model of a subject in the captured image, generates a plurality of candidate images that are candidates for RGB texture images of the virtual-viewpoint 3D model. An estimation unit generates an RGB texture image by synthesizing the plurality of candidate images generated by the candidate image generation unit, and estimates an opacity image representing the opacity of each pixel of the RGB texture image from the RGB texture image. The present technology can be applied to, for example, a display device or the like.

Description

Image processing device, image processing method, and program

[0001] The present technology relates to an image processing device, an image processing method, and a program, and more particularly to an image processing device, an image processing method, and a program that enable accurate estimation of an opacity image.

[0002] There is a technology called volumetric capture (volumetric video) that generates images from free virtual viewpoints in all directions (360 degrees) from multi-viewpoint images taken by surrounding a subject (object) with multiple cameras.

[0003] In volumetric capture, for example, a 3D model showing the 3D shape of the subject using a mesh or the like is reconstructed from multi-viewpoint images captured by surrounding the subject with multiple cameras. Then, an image of the subject from a virtual viewpoint is rendered as a free-viewpoint image based on the multi-viewpoint images, parameters of each camera, the 3D model of the subject, and a virtual viewpoint. This free-viewpoint image is used for 2D video viewing, naked-eye stereoscopic viewing (Spatial Reality Display), mobile AR (Augmented Reality), etc.

[0004] One technique for rendering a free-viewpoint image is a free-viewpoint image synthesis technique that generates a final free-viewpoint image by rendering multiple candidate images for the free-viewpoint image and synthesizing them using a machine learning model (see, for example, Non-Patent Document 1). Rendering is performed by, for example, perspectively projecting a 3D model from a virtual viewpoint and applying, to the projection area, an RGB texture image that represents the color and pattern of the 3D model generated from images captured from multiple viewpoints using RGB color information.

[0005] However, because this RGB texture image contains only color information, it is difficult to render semi-transparent areas such as the subject's hair, or to naturally combine (blend) the subject and background assets in a free-viewpoint image. Because the shape of the subject's image (silhouette) in a free-viewpoint image depends on the shape of the 3D model, if an error occurs in the shape of the 3D model, texture mapping will cause the free-viewpoint image to include not only the subject image but also the background image.

[0006] Therefore, it is desirable to improve the image quality of free viewpoint images by performing texture mapping using not only the RGB texture image but also an opacity image that represents opacity information such as the alpha value of each pixel in the RGB texture image.

[0007] On the other hand, there is an alpha matting technique that estimates the alpha value of an input image (see, for example, Non-Patent Document 2). In the technique described in Non-Patent Document 2, the foreground and background are separated based on the estimated alpha value, and only the background is replaced with another image. However, since an RGB texture image is an image of a subject corresponding to a 3D model, i.e., an image of only the foreground, it is difficult to accurately estimate an opacity image using an alpha matting technique that targets an image that has a foreground and a background.

[0008] It is also being considered to employ alpha matting technology in free viewpoint image synthesis that handles images of only the foreground (see, for example, Patent Document 1). The invention described in Patent Document 1 uses alpha matting to separate images captured from multiple viewpoints into three layers, a main layer, a boundary foreground layer, and a boundary background layer, and then performs rendering, thereby achieving high image quality for free viewpoint images.

[0009] However, in the invention described in Patent Document 1, alpha matting is performed on images captured from multiple viewpoints. Therefore, the alpha values ​​of the free-viewpoint image generated by rendering contain errors from the alpha matting in three dimensions. As a result, the image quality of the free-viewpoint image is not sufficiently improved. Furthermore, in the invention described in Patent Document 1, the foreground and background are separated only at the boundary, making it difficult to accurately separate the foreground and background in areas away from the boundary.

[0010] Hedman, Peter & Philip, Julien & Price, True & Frahm, Jan-Michael & Drettakis, George & Gabriel, Jourdan, “Deep blending for free-viewpoint image-based rendering”, ACM Transactions on Graphics Volume 37, 2018, P.1-15Xu, N., Price, B., Cohen, S., & Huang, T., “Deep image matting”, In Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, p. 2970-2979Japanese Patent Application Publication No. 2013-223008

[0011] For these reasons, there is a demand for a method for accurately estimating an opacity image, but this demand has not yet been fully met.

[0012] The present technology has been made in view of such circumstances, and makes it possible to accurately estimate an opacity image.

[0013] An image processing device or program according to one aspect of the present technology is an image processing device or a program for causing a computer to function as an image processing device, including: a candidate image generation unit that generates a plurality of candidate images that are candidates for an RGB texture image of a 3D model of a virtual viewpoint, using captured images from multiple viewpoints and shape data that indicate the shape of a 3D model of a subject in the captured images; and an estimation unit that generates the RGB texture image by combining the plurality of candidate images generated by the candidate image generation unit, and estimates an opacity image that indicates the opacity of each pixel of the RGB texture image from the RGB texture image.

[0014] An image processing method according to one aspect of the present technology is an image processing method that includes: generating a plurality of candidate images that are candidates for an RGB texture image of the 3D model of a virtual viewpoint, using captured images from multiple viewpoints and shape data that indicate the shape of a 3D model of a subject in the captured images; generating the RGB texture image by combining the plurality of candidate images; and estimating an opacity image that indicates the opacity of each pixel of the RGB texture image from the RGB texture image.

[0015] In one aspect of the present technology, multiple candidate images that are candidates for an RGB texture image of the 3D model of a virtual viewpoint are generated using captured images from multiple viewpoints and shape data that indicates the shape of a 3D model of a subject in the captured images, the RGB texture image is generated by combining the multiple candidate images, and an opacity image that indicates the opacity of each pixel of the RGB texture image is estimated from the RGB texture image.

[0016] 1 is a block diagram showing an example configuration of an embodiment of an image processing system to which the present technology is applied. FIG. 2 is a diagram showing an example arrangement of the imaging device of FIG. 1. FIG. 3 is a block diagram showing an example configuration of a rendering unit. FIG. 4 is a diagram explaining generation of candidate images by a candidate image generation unit. FIG. 5 is a block diagram showing an example configuration of a candidate image synthesis unit. FIG. 6 is a diagram explaining an effect of the rendering unit. FIG. 7 is a flowchart explaining 3D data generation processing. FIG. 8 is a flowchart explaining display processing. FIG. 9 is a flowchart explaining details of rendering processing. FIG. 10 is a block diagram showing an example configuration of a learning system. FIG. 11 is a block diagram showing an example configuration of a learning unit. FIG. 12 is a diagram explaining machine learning by a learning unit. FIG. 13 is a flowchart explaining learning processing. FIG. 14 is a block diagram showing an example configuration of computer hardware.

[0017] Hereinafter, modes for carrying out the present technology (hereinafter referred to as embodiments) will be described. The description will be made in the following order: 1. One embodiment (image processing system) 2. Computer 3. Application example

[0018] 1. One embodiment Configuration example of image processing system FIG. 1 is a block diagram showing a configuration example of one embodiment of an image processing system to which the present technology is applied.

[0019] 1 is composed of N (N is a plural number) image capture devices 11-1 to 11-N, a 3D data generation device 12, and a display device 13. The image processing system 10 generates a free viewpoint image using captured images from N viewpoints acquired by the image capture devices 11-1 to 11-N, and displays the free viewpoint image.

[0020] Specifically, the image capturing devices 11-1 to 11-N are disposed at different positions and capture images synchronously. Images captured from N viewpoints obtained as a result of image capturing by the image capturing devices 11-1 to 11-N are input to the 3D data generating device 12. Note that the captured images may be still images captured in units of one frame, or may be moving images captured in units of multiple frames.

[0021] In the following description, unless it is necessary to distinguish between the image capturing devices 11-1 to 11-N, they will be collectively referred to as the image capturing device 11. Image capturing by the N image capturing devices 11 may be performed virtually by physically moving one image capturing device to each of the positions of the N image capturing devices 11. The N image capturing devices 11 may be provided outside the image processing system 10.

[0022] The 3D data generating device 12 is composed of an image acquiring unit 21 , a 3D model generating unit 22 , an encoding unit 23 , and a transmitting unit 24 .

[0023] The image acquisition unit 21 acquires captured images from N viewpoints supplied from N image capture devices 11. The image acquisition unit 21 performs calibration based on the captured images from the N viewpoints and acquires internal parameters and external parameters of each image capture device 11 as camera parameters. The image acquisition unit 21 supplies the captured images from the N viewpoints and the camera parameters of each image capture device 11 to the 3D model generation unit 22. The image acquisition unit 21 generates depth maps for each of the multiple viewpoints based on the captured images from the N viewpoints and supplies them to the 3D model generation unit 22. A depth map for a specific viewpoint is an image in which pixel values ​​are depth values ​​indicating the distance from that viewpoint to a subject.

[0024] The 3D model generation unit 22 generates a 3D model representing the three-dimensional shape of the subject frame by frame based on the captured images from N viewpoints and camera parameters supplied from the image acquisition unit 21. As a result, if the captured images from N viewpoints are still images, a 3D model of the still images, which is a 3D model for one frame, is generated, and if the captured images are moving images, a 3D model of the moving images, which is a 3D model for multiple frames, is generated. Since this 3D model is generated using the captured images, it is a live-action 3D model. For example, visual hulling is an example of a method for generating a 3D model. In visual hulling, a 3D model of the subject is generated by carving out a three-dimensional shape using silhouette images corresponding to the captured images from the N viewpoints.

[0025] The 3D model generation unit 22 deforms the generated 3D model with high precision using the depth map supplied from the image acquisition unit 21. Note that the image acquisition unit 21 does not necessarily have to generate a depth map, and the 3D model generation unit 22 does not necessarily have to deform the 3D model with high precision.

[0026] The 3D model generation unit 22 generates RGB texture images from N viewpoints as RGB texture images of the 3D model in a view-dependent format, in which the color changes depending on the viewpoint position (viewing direction), based on shape data indicating the surface shape of the 3D model and the captured images from N viewpoints.

[0027] Examples of shape data include mesh data called a polygon mesh, which represents a shape by connections between vertices, and point cloud data, which represents a shape using a point cloud. RGB texture image formats include view-dependent formats and view-independent formats, in which colors do not change depending on the viewpoint. The 3D model generation unit 22 supplies the encoding unit 23 with 3D data in which RGB texture images from N viewpoints are associated with the shape data, as well as camera parameters.

[0028] The encoding unit 23 converts the 3D data and camera parameters supplied from the 3D model generation unit 22 into a format suitable for transmission and storage. Specifically, the encoding unit 23 converts shape data from the 3D data into depth maps for one or more viewpoints. The encoding unit 23 encodes (compresses) the depth maps for one or more viewpoints and RGB texture images from N viewpoints in the 3D data, adds camera parameters, and generates a transmission stream, which is a bit stream.

[0029] At this time, the depth map and RGB texture image for each viewpoint may be combined and coded as a single image, or may be coded as separate images. Because the depth map and RGB texture image are two-dimensional images, they can be coded using existing two-dimensional image coding techniques such as AVC (Advanced Video Coding). The coding unit 23 supplies the transmission stream to the transmission unit 24.

[0030] The transmitting unit 24 transmits the transmission stream supplied from the encoding unit 23 to the display device 13 .

[0031] In the 3D data generation device 12 configured as described above, the image acquisition unit 21, the 3D model generation unit 22, and the encoding unit 23 perform processing offline in advance, and the transmission unit 24 can transmit the transmission stream at a predetermined timing. The transmission unit 24 can also transmit the transmission stream obtained as a result of the processing by the image acquisition unit 21, the 3D model generation unit 22, and the encoding unit 23 in real time.

[0032] The display device 13 (image processing device) is composed of a receiving unit 31, a decoding unit 32, a rendering unit 33, and a display unit .

[0033] The receiving unit 31 receives the transmission stream transmitted from the transmitting unit 24 and supplies it to the decoding unit 32 .

[0034] The decoding unit 32 decodes the transmission stream supplied from the receiving unit 31 to obtain a depth map, an RGB texture image, and camera parameters. The decoding unit 32 generates shape data from the depth map, and supplies the shape data, 3D data consisting of the RGB texture image, and camera parameters to the rendering unit 33.

[0035] The rendering unit 33 performs rendering using texture mapping based on the 3D data and camera parameters supplied from the decoding unit 32 and the virtual viewpoint supplied from the display unit 34, and generates an image of the subject from the virtual viewpoint as a free viewpoint image.

[0036] Note that since the RGB texture images included in the 3D data are RGB texture images of N viewpoints in a view-dependent format, the rendering unit 33 performs texture mapping using a view-dependent method that takes the virtual viewpoint into consideration. In view-dependent texture mapping, the RGB texture image to be applied changes depending on the position of the virtual viewpoint, making it possible to generate a free-viewpoint image of higher quality than view-independent texture mapping.

[0037] In contrast, in view-independent texture mapping, which does not take the virtual viewpoint into consideration, the RGB texture image is the same regardless of the position of the virtual viewpoint, so the amount of processing required is less than in view-dependent texture mapping.

[0038] The display unit 34 is configured by a 2D monitor or a 3D monitor of a head-mounted display, a spatial display, a mobile phone, a television receiver, a personal computer (hereinafter referred to as a PC), etc. The display unit 34 performs display based on the free viewpoint image supplied from the rendering unit 33.

[0039] The display unit 34 detects the user's viewing location (Region of Interest) and, based on the viewing location, determines the user's desired viewing viewpoint in all 360-degree directions as a virtual viewpoint, regardless of the position of the image capturing device 11. The display unit 34 supplies the virtual viewpoint to the rendering unit 33. Note that the display unit 34 may be provided outside the display device 13 or the image processing system 10.

[0040] 1 , the 3D data generator 12 transmits a transmission stream and the display device 13 performs display based on the transmission stream, but the 3D data generator 12 and the display device 13 may be integrated. In this case, the image processing system 10 does not need to include the encoding unit 23, the transmitting unit 24, the receiving unit 31, and the decoding unit 32 because a single device such as a PC performs processes from generating 3D data to displaying a free viewpoint image.

[0041] When the image processing system 10 is implemented, it may be implemented entirely by one operator, or it may be implemented separately by multiple operators. For example, when the image processing system 10 is implemented separately by three operators A to C, operator A implements the image acquisition unit 21, the 3D model generation unit 22, and the encoding unit 23 to generate a transmission stream. Operator B, as a platformer, implements the transmission unit 24 to distribute the transmission stream. Operator C implements the display device 13 to generate a free viewpoint image based on the transmission stream and perform a display based on the free viewpoint image.

[0042] <Example of Arrangement of Camera Devices> FIG. 2 is a diagram showing an example of arrangement of the camera devices 11 in FIG.

[0043] In the example of Fig. 2, N is 8. As shown in Fig. 2, the image capturing devices 11-1 to 11-8 are arranged to surround the subject 51. In the example of Fig. 2, the image capturing devices 11 are arranged at equal intervals, but the positions of the image capturing devices 11 can be any positions.

[0044] <Configuration Example of Rendering Unit> FIG. 3 is a block diagram showing a configuration example of the rendering unit 33 in FIG.

[0045] As shown in FIG. 3 , the rendering unit 33 includes a candidate image generating unit 71 , a storage unit 72 , an estimation unit 73 , and a free viewpoint image generating unit 74 .

[0046] The candidate image generation unit 71 acquires a virtual viewpoint supplied from the display unit 34 in Fig. 1. The candidate image generation unit 71 selects one or more viewpoints that are close to the virtual viewpoint from among the N viewpoints, based on the virtual viewpoint and the camera parameters supplied from the decoding unit 32. Then, the candidate image generation unit 71 selects, as a reference image group, the RGB texture image of the selected viewpoint from the RGB texture images of the N viewpoints in the 3D data supplied from the decoding unit 32.

[0047] The candidate image generation unit 71 performs perspective projection of the 3D model from the virtual viewpoint based on the shape data of the 3D data, the camera parameters corresponding to the reference images, and the virtual viewpoint, and performs texture mapping to attach the reference images to the projection area. As a result, the candidate image generation unit 71 renders (generates) multiple candidate images that serve as candidates for the RGB texture image of the 3D model from the virtual viewpoint. These candidate images can also be considered images obtained by warping the viewpoint of the reference images to the virtual viewpoint. The candidate image generation unit 71 supplies the multiple candidate images to the estimation unit 73. The number of candidate images can be set to any number.

[0048] The storage unit 72 stores an alpha value estimation model, which is a machine learning model. The alpha value estimation model is a model that takes a plurality of candidate images as input, synthesizes the plurality of candidate images to generate an RGB texture image of a 3D model of a virtual viewpoint, and estimates an opacity image from the RGB texture image and outputs it. The machine learning of the alpha value estimation model will be described later with reference to FIGS. 11 to 14.

[0049] The estimation unit 73 is composed of a candidate image synthesis unit 81 and an alpha value estimation unit 82. The estimation unit 73 uses the alpha value estimation model stored in the storage unit 72 to synthesize a plurality of candidate images supplied from the candidate image generation unit 71, thereby generating an RGB texture image of a 3D model of the virtual viewpoint and estimating an opacity image.

[0050] Specifically, the candidate image synthesis unit 81 synthesizes the multiple candidate images supplied from the candidate image generation unit 71 in accordance with the alpha value estimation model to generate an RGB texture image of the 3D model of the virtual viewpoint. The candidate image synthesis unit 81 supplies this RGB texture image to the alpha value estimation unit 82 and the free viewpoint image generation unit 74.

[0051] The alpha value estimation unit 82 estimates an opacity image from the RGB texture image supplied from the candidate image synthesis unit 81 in accordance with the alpha value estimation model, and supplies the estimated opacity image to the free viewpoint image generation unit 74 .

[0052] In this way, the estimation unit 73 supplies the free viewpoint image generation unit 74 with an RGBA texture image consisting of an RGB texture image and an opacity image of the 3D model of the virtual viewpoint, i.e., an RGBA texture image having color information and opacity information of the 3D model of the virtual viewpoint.

[0053] The free viewpoint image generation unit 74 performs texture mapping to render the image of the subject at the virtual viewpoint supplied from the display unit 34, using the RGBA texture image supplied from the estimation unit 73 and the shape data of the 3D data supplied from the decoding unit 32. In this way, the free viewpoint image generation unit 74 generates the image of the subject at the virtual viewpoint as a free viewpoint image.

[0054] The free viewpoint image generation unit 74 may prepare background CG (Computer Graphics) that is different from the background image in the captured image and render the image of the subject at the virtual viewpoint using the background CG as the background image. In this case, the free viewpoint image generation unit 74 generates a free viewpoint image in which the image of the subject at the virtual viewpoint and the background CG are combined. The free viewpoint image generation unit 74 supplies the free viewpoint image to the display unit 34.

[0055] <Description of Generation of Candidate Images> FIG. 4 is a diagram for explaining generation of candidate images by the candidate image generating unit 71 in FIG.

[0056] In the example of FIG. 4, the candidate image generation unit 71 generates a candidate image from a viewpoint c that is close to the virtual viewpoint v based on the camera parameters of the image capturing device 11 and the virtual viewpoint v. ０ and c １ The two RGB texture images are selected as the reference image group. This also applies to FIG. 5, which will be described later.

[0057] The candidate image generation unit 71 perspectively projects each predetermined region p of the 3D model from the virtual viewpoint v onto the region p, and determines color information for each pixel of each of the plurality of candidate images corresponding to the projection region by referring to the reference image group. One method for determining color information for each pixel of each of the plurality of candidate images is to determine, for each pixel of the candidate image, color information for the pixel of the corresponding reference image group in descending order of score.

[0058] This score is calculated for each pixel in each reference image group at the viewpoint c ０ (c １ ) and the distance between the virtual viewpoint v and the viewpoint c for the area p corresponding to that pixel ０ (c １ ) and the difference θ between the shooting angle of the virtual viewpoint v ０ (θ １) or the like. A known image-based rendering method such as the Unstructured Lumigraph rendering method can be used to calculate this score. The Unstructured Lumigraph rendering method is described, for example, in Gortler, Steven & Buehler, Chris & Bosse, Michael & Mcmillan, Leonard & Cohen, Michael, "Unstructured Lumigraph Rendering", Proceedings of SIGGRAPH, 2001, pp. 425-432. The score can be calculated, for example, based on the viewpoint c ０ (c １ ) and the virtual viewpoint v, the smaller the difference θ ０ (θ １ ) is calculated so that the smaller it is, the higher the value will be.

[0059] <Configuration Example of Candidate Image> FIG. 5 is a diagram showing a configuration example of a candidate image.

[0060] In Fig. 5, for the sake of simplicity, only 4 × 3 pixels of the candidate image are shown. In Fig. 5, black circles represent the pixels of the candidate image at the viewpoint c ０ c has the color information of the pixels of the captured image ０ The white circle represents the pixel, and the white circle represents the viewpoint c １ c has the color information of the pixels of the captured image １ represents a pixel.

[0061] As shown in FIG. 5, each pixel of the ith (i is an integer equal to or greater than 1) candidate image generated by the candidate image generating unit 71 has color information of the pixel in the reference image group having the ith highest score corresponding to that pixel. ０ pixel or c １ It is a pixel.

[0062] <Configuration Example of Candidate Image Combining Unit> FIG. 6 is a block diagram showing a configuration example of the candidate image combining unit 81 in FIG.

[0063] The candidate image synthesis unit 81 in FIG. 6 is composed of a ratio setting unit 111 and a synthesis unit 112 .

[0064] The ratio setting unit 111 determines a blending ratio from the plurality of candidate images supplied from the candidate image generating unit 71 in accordance with the alpha value estimation model stored in the storage unit 72 , and supplies the blending ratio to the blending unit 112 .

[0065] The synthesis unit 112 synthesizes the plurality of candidate images supplied from the candidate image generation unit 71 based on the synthesis ratio supplied from the synthesis unit 112, and generates an RGB texture image of the 3D model of the virtual viewpoint.

[0066] For example, when four candidate images I1 to I4 are supplied, the synthesis unit 112 performs weighted addition of the four candidate images based on the synthesis ratio using the following equation (1), and generates an RGB texture image I of the 3D model of the virtual viewpoint: fg Generate.

[0067] I fg =W1I1+W2I2+W3I3+W4I4...(1)

[0068] In equation (1), W1 to W4 are weights and composition ratios for the candidate images I1 to I4, respectively.

[0069] As described above, the candidate image synthesis unit 81 synthesizes multiple candidate images to generate a final RGB texture image of the 3D model of the virtual viewpoint, thereby improving the image quality of the RGB texture image of the 3D model of the virtual viewpoint. The candidate image synthesis unit 81 supplies the generated RGB texture image of the 3D model of the virtual viewpoint to the free viewpoint image generation unit 74 and the alpha value estimation unit 82 in FIG. 3 .

[0070] <Explanation of Effects Produced by Rendering Unit> FIG. 7 is a diagram for explaining the effects produced by the rendering unit 33 in FIG.

[0071] 7A, an error occurs in the shape of the 3D model generated by the 3D model generation unit 22, causing the shape of the 3D model to be larger than the shape of the actual subject. In this case, an image of the subject as a foreground image in the captured image is pasted as an RGB texture image into a projection area 132 of the actual subject within a projection area 131 of the 3D model in the free viewpoint image. However, a background image in the captured image is pasted as an RGB texture image into an area 133 extending outside the projection area 132.

[0072] At this time, the rendering unit 33 generates the free viewpoint image using the RGBA texture image instead of the RGB texture image. Therefore, when an opacity image is generated so that the opacity information of the region 133 is low, the background image is pasted onto the region 133 in the free viewpoint image so as to be transparent, as shown in B of Fig. 7 .

[0073] In contrast, when a free-viewpoint image is generated using an RGB texture image, the background image is pasted onto the area 133 so as not to be transparent, as shown in C of Fig. 7. Therefore, the free-viewpoint image includes an opaque background image. As a result, the image quality of the free-viewpoint image deteriorates.

[0074] Although not shown in the figures, when the free-viewpoint image generation unit 74 performs rendering on the background CG, opacity information of the boundary region of the 3D model is set to an intermediate value, thereby enabling natural synthesis of the background CG and the subject in the free-viewpoint image. The free-viewpoint image generation unit 74 performs rendering using an RGBA texture image, allowing it to express semi-transparent parts such as the subject's hair in the free-viewpoint image.

[0075] <Description of Transmission Process> Fig. 8 is a flowchart illustrating the 3D data generation process by the 3D data generation device 12 of Fig. 1. This 3D data generation process is performed, for example, when captured images from N viewpoints in frame units are input from N image capture devices 11.

[0076] 8 , the image acquisition unit 21 acquires captured images from N viewpoints supplied from N image capture devices 11. The image acquisition unit 21 performs calibration based on the captured images from the N viewpoints and acquires camera parameters. The image acquisition unit 21 generates depth maps for each of the multiple viewpoints based on the captured images from the N viewpoints. The image acquisition unit 21 supplies the captured images from the N viewpoints, the camera parameters of each image capture device 11, and the depth maps for each of the multiple viewpoints to the 3D model generation unit 22.

[0077] In step S12, the 3D model generation unit 22 generates a 3D model and RGB texture images from N viewpoints based on the captured images from N viewpoints, camera parameters, and depth maps for each of the multiple viewpoints supplied from the image acquisition unit 21. The 3D model generation unit 22 supplies the 3D data in which the RGB texture images from N viewpoints are associated with shape data of the 3D model and the camera parameters to the encoding unit 23.

[0078] In step S13, the encoding unit 23 converts the formats of the 3D data and camera parameters supplied from the 3D model generation unit 22 to generate a transmission stream. The encoding unit 23 supplies the transmission stream to the transmission unit 24.

[0079] In step S14, the transmission unit 24 transmits the transmission stream generated by the processing in step S13 to the display device 13. Then, the 3D data generation processing ends.

[0080] <Explanation of Display Processing> Fig. 9 is a flowchart illustrating the display processing by the display device 13 of Fig. 1. This display processing is started, for example, when a transmission stream is transmitted by the processing of step S14 of Fig. 8.

[0081] In step S31 of FIG. 9, the receiving unit 31 receives the transport stream transmitted by the process of step S14 of FIG.

[0082] In step S32, the decoding unit 32 decodes the transmission stream received in step S31 to obtain a depth map, RGB texture images from N viewpoints, and camera parameters. The decoding unit 32 generates shape data from the depth map and supplies the shape data, 3D data consisting of the RGB texture images from N viewpoints, and camera parameters to the rendering unit 33.

[0083] In step S33, the rendering unit 33 performs rendering processing to generate a free viewpoint image based on the 3D data and camera parameters supplied from the decoding unit 32 and the virtual viewpoint supplied from the display unit 34. Details of this rendering processing will be described later with reference to FIG. 10. The rendering unit 33 supplies the free viewpoint image to the display unit 34.

[0084] In step S34, the display unit 34 performs display based on the free viewpoint image generated in the process of step S33, and the display process then ends.

[0085] <Details of Rendering Process> FIG. 10 is a flowchart illustrating details of the rendering process in step S33 of FIG.

[0086] 10 , the candidate image generation unit 71 and the free viewpoint image generation unit 74 acquire a virtual viewpoint supplied from the display unit 34. In step S52, the candidate image generation unit 71 selects a group of reference images from among the RGB texture images of N viewpoints supplied from the decoding unit 32, based on the virtual viewpoint acquired by the processing of step S51 and the camera parameters supplied from the decoding unit 32.

[0087] In step S52, the candidate image generating unit 71 calculates a score for each pixel in the reference image group based on the camera parameters, shape data, and the virtual viewpoint acquired by the processing in step S51.

[0088] In step S54, the candidate image generation unit 71 generates a plurality of candidate images based on the score calculated in step S53, the shape data, and the group of reference images. The candidate image generation unit 71 supplies the plurality of candidate images to the candidate image synthesis unit 81.

[0089] In step S55, the candidate image synthesis unit 81 synthesizes the multiple candidate images generated in step S54 in accordance with the alpha value estimation model to generate an RGB texture image of the 3D model of the virtual viewpoint. The candidate image synthesis unit 81 supplies this RGB texture image to the free viewpoint image generation unit 74 and the alpha value estimation unit 82.

[0090] In step S56 , the alpha value estimation unit 82 estimates an opacity image from the RGB texture image supplied from the candidate image synthesis unit 81 in accordance with the alpha value estimation model, and supplies the estimated opacity image to the free viewpoint image generation unit 74 .

[0091] In step S57 , the free viewpoint image generation unit 74 acquires an RGBA image composed of the RGB texture image supplied from the candidate image synthesis unit 81 and the opacity image supplied from the alpha value estimation unit 82 .

[0092] In step S58, the free viewpoint image generation unit 74 generates a free viewpoint image based on the RGBA texture image acquired in the process of step S57, the shape data supplied from the decoding unit 32, and the virtual viewpoint acquired in step S51. Then, the rendering process ends, and the process returns to step S33 in Fig. 9 and proceeds to step S34.

[0093] <Configuration Example of Learning System> FIG. 11 is a block diagram showing a configuration example of a learning system that performs machine learning to learn an alpha value estimation model.

[0094] 11 is configured with N (N is a plural number) image capture devices 311-1 to 311-N and a learning device 312. The learning system 300 performs machine learning to generate an alpha value estimation model using images captured from N viewpoints by the image capture devices 311-1 to 311-N.

[0095] Specifically, the image capture devices 311-1 to 311-N are configured in the same manner as the image capture devices 11-1 to 11-N, and therefore a description thereof will be omitted. Hereinafter, unless there is a need to particularly distinguish between the image capture devices 311-1 to 311-N, they will be collectively referred to as image capture devices 311. The N image capture devices 311 capture images from N viewpoints that include a subject, and supply these images to the learning device 312 as the N viewpoints for machine learning. The N image capture devices 311 also capture background images from N viewpoints that do not include a subject, and supply these images to the learning device 312.

[0096] The learning device 312 is composed of an image acquisition unit 321 , a 3D model generation unit 322 , and a learning unit 323 .

[0097] The image acquisition unit 321 acquires captured images from N viewpoints for machine learning supplied from the N image capture devices 311. Similar to the image acquisition unit 21 in FIG. 1 , the image acquisition unit 321 generates camera parameters and a depth map for each image capture device 311 based on the captured images from the N viewpoints, and supplies the captured images from the N viewpoints, camera parameters, and depth map to the 3D model generation unit 322. The image acquisition unit 321 also supplies background images from the N viewpoints supplied from the N image capture devices 311 to the learning unit 323.

[0098] Similar to the 3D model generation unit 22, the 3D model generation unit 322 generates 3D data based on the captured images from N viewpoints, camera parameters, and depth maps supplied from the image acquisition unit 321. The 3D model generation unit 322 supplies the 3D data and camera parameters to the learning unit 323.

[0099] The learning unit 323 performs machine learning to generate an alpha value estimation model based on the 3D data and camera parameters supplied from the 3D model generation unit 322 and the background image supplied from the image acquisition unit 321. As the architecture of the alpha value estimation model, a network (model) such as U-Net in which the input image and output image sizes match, a known alpha matting network, or the like can be used.

[0100] <Configuration Example of Learning Unit> FIG. 12 is a block diagram showing a configuration example of the learning unit 323 in FIG.

[0101] As shown in FIG. 12 , the learning unit 323 includes a candidate image generating unit 341 , a storage unit 342 , an estimation unit 343 , a free viewpoint image generating unit 344 , a calculation unit 345 , and an update unit 346 .

[0102] The candidate image generation unit 341 generates a plurality of candidate images to be input for machine learning of the alpha value estimation model. Specifically, the candidate image generation unit 341 sequentially determines each of the N viewpoints as a virtual viewpoint. In this embodiment, the candidate image generation unit 341 sequentially determines all of the N viewpoints as virtual viewpoints, but it may also sequentially determine only some of the viewpoints as virtual viewpoints.

[0103] The candidate image generation unit 341 supplies the calculation unit 345 with an RGB texture image of the virtual viewpoint from the 3D data supplied from the 3D model generation unit 322 in FIG. 11 . Similar to the candidate image generation unit 71 in FIG. 3 , the candidate image generation unit 341 selects one or more image capture devices 311 of viewpoints close to the virtual viewpoint from among the N viewpoints based on the virtual viewpoint and the camera parameters supplied from the 3D model generation unit 322. The candidate image generation unit 341 then selects, as a reference image group, the RGB texture images of the selected viewpoints from the RGB texture images of the N viewpoints. Similar to the candidate image generation unit 71, the candidate image generation unit 341 generates multiple candidate images of the RGB texture image of the 3D model of the virtual viewpoint based on the shape data from the 3D data, the camera parameters corresponding to the reference image group, and the virtual viewpoint. The candidate image generation unit 341 supplies the multiple candidate images to the estimation unit 343.

[0104] The candidate image generation unit 341 supplies the background image of the virtual viewpoint and the virtual viewpoint out of the background images of the N viewpoints supplied from the image acquisition unit 321 to the free viewpoint image generation unit 344 .

[0105] The storage unit 342 stores the alpha value estimation model currently being trained.

[0106] The estimation unit 343 is composed of a candidate image synthesis unit 351 and an alpha value estimation unit 352. Similar to the estimation unit 73, the estimation unit 343 synthesizes a plurality of candidate images supplied from the candidate image generation unit 341 in accordance with the alpha value estimation model stored in the storage unit 342, thereby generating an RGB texture image of a 3D model of a virtual viewpoint and estimating an opacity image.

[0107] The free viewpoint image generation unit 344 acquires an RGBA texture image composed of an RGB texture image and an opacity image, supplied from the estimation unit 343. The free viewpoint image generation unit 344 renders an image of the subject at the virtual viewpoint in the background image of the virtual viewpoint, based on the RGBA texture image, the shape data supplied from the image acquisition unit 321, and the virtual viewpoint. As a result, the free viewpoint image generation unit 344 generates a free viewpoint image corresponding to the captured image of the virtual viewpoint, and supplies the generated image to the calculation unit 345.

[0108] The calculation unit 345 calculates a loss function by comparing the captured image from the virtual viewpoint supplied from the candidate image generation unit 341 as an ideal value of the output of the alpha value estimation model with the free viewpoint image supplied from the free viewpoint image generation unit 344 as an actual value of the output. Here, since the free viewpoint image includes not only an image of the subject but also a background image, the difference between the captured image from the virtual viewpoint and the free viewpoint image can be calculated using the loss function. The calculation unit 345 supplies the calculated loss function to the update unit 346.

[0109] The update unit 346 updates the alpha value estimation model stored in the storage unit 342 by backpropagating the loss function supplied from the calculation unit 345 to the alpha value estimation model. As a result, the alpha value estimation model learns, as an image captured from a virtual viewpoint, the ideal value of a composite image of an output when a plurality of candidate images are input and a background image of a virtual viewpoint. As a result, when a plurality of candidate images are input, the alpha value estimation model is trained to output a free viewpoint image that, when combined with the background image of the virtual viewpoint, has image quality similar to that of the image captured from the virtual viewpoint.

[0110] <Explanation of Machine Learning by Learning Unit> FIG. 13 is a diagram illustrating machine learning by the learning unit 323 in FIG.

[0111] 13, m (m is a plural number) candidate images 401-1 to 401-m are input to the estimation unit 343 from the candidate image generation unit 341. The estimation unit 343 generates an RGB texture image 402 of a 3D model of a virtual viewpoint from the m candidate images 401-1 to 401-m, and estimates an opacity image 403 of the virtual viewpoint.

[0112] The free viewpoint image generation unit 344 renders an image of the subject at the virtual viewpoint in a background image 404 of the virtual viewpoint based on the RGBA texture image made up of the RGB texture image 402 and the opacity image 403, the shape data, and the virtual viewpoint. In this way, the free viewpoint image generation unit 344 generates a free viewpoint image 405 corresponding to the captured image at the virtual viewpoint.

[0113] The free viewpoint image 405 is an image in which an RGB texture image 402 and a background image 404, which are rendered for each pixel based on shape data and a virtual viewpoint, are synthesized based on an opacity image 403, as shown in the following equation (2).

[0114] I pred = αI fg_pred +(1-α)I bg_gt ... (2)

[0115] In formula (2), I pred is the pixel value of the pixel to be processed in the free viewpoint image 405, and α is the pixel value of the pixel to be processed in the opacity image 403, and is a decimal number ranging from 0 to 1. fg_pred is the pixel value of the pixel to be processed when the RGB texture image 402 is rendered, and I bg_gt is the pixel value of the pixel to be processed in the background image 404.

[0116] The calculation unit 345 calculates a loss function by comparing the captured image 406 from the virtual viewpoint supplied from the candidate image generation unit 341 as an ideal value of the output of the alpha value estimation model with the free viewpoint image 405 as an actual value of the output. The loss function may be any function that can calculate the difference between the captured image 406 and the free viewpoint image 405. Examples of loss functions include L1 loss and L2 loss. The update unit 346 learns the alpha value estimation model by backpropagating this loss function to the alpha value estimation model.

[0117] Note that the calculation unit 345 may calculate the loss function not by using the free viewpoint image 405 (synthetic image) and the captured image 406, but by using the opacity image 403 and the opacity image 407 corresponding to the captured image 406. Since the captured image 406 is an image that includes a subject as the foreground and a background, the opacity image 407 is estimated from the captured image 406 by, for example, a known alpha matting technique.

[0118] The calculation unit 345 may calculate a loss function using the free viewpoint image 405 and the captured image 406, and may also calculate a loss function using the opacity image 403 and the opacity image 407. In this case, the update unit 346 updates the alpha value estimation model by backpropagating both loss functions to the alpha value estimation model.

[0119] <Explanation of Learning Process> Fig. 14 is a flowchart illustrating the learning process by the learning unit 323 in Fig. 12. This learning process starts, for example, when a background image is supplied from the image acquisition unit 321 in Fig. 11 and 3D data and camera parameters are supplied from the 3D model generation unit 322.

[0120] 14 , the candidate image generation unit 341 determines one of the N viewpoints that has not yet been determined as a virtual viewpoint as a virtual viewpoint. The candidate image generation unit 341 supplies the virtual viewpoint and a background image of the virtual viewpoint from among the background images of the N viewpoints supplied from the image acquisition unit 321 to the free viewpoint image generation unit 344. The candidate image generation unit 341 supplies an RGB texture image of the virtual viewpoint from the 3D data supplied from the 3D model generation unit 322 to the calculation unit 345. The candidate image generation unit 341 selects a group of reference images from the RGB texture images of the N viewpoints based on the virtual viewpoint and the camera parameters supplied from the 3D model generation unit 322.

[0121] In step S101, the candidate image generation unit 341 generates a plurality of candidate images of an RGB texture image of a 3D model of a virtual viewpoint based on shape data of the 3D data, camera parameters corresponding to a group of reference images, and a virtual viewpoint. The candidate image generation unit 341 supplies the plurality of candidate images to the estimation unit 343.

[0122] In step S102, the candidate image generation unit 341 determines whether or not there is any viewpoint among the N viewpoints that has not yet been determined as a virtual viewpoint by the processing in step S100. If it is determined in step S102 that there is any viewpoint that has not yet been determined as a virtual viewpoint, the processing returns to step S100, and the processing in steps S101 to S102 is repeated.

[0123] On the other hand, if it is determined in step S102 that there are no viewpoints that have not been determined as virtual viewpoints, that is, if all of the N viewpoints have been determined as virtual viewpoints, the process proceeds to step S103. The processes of steps S103 to S107 are performed for each virtual viewpoint.

[0124] In step S103, the candidate image synthesis unit 351 generates an RGB texture image of the 3D model of the virtual viewpoint from the multiple candidate images generated by the processing of step S101 in accordance with the alpha value estimation model. The candidate image synthesis unit 351 supplies this RGB texture image to the alpha value estimation unit 352 and the free viewpoint image generation unit 344.

[0125] In step S104, the alpha value estimation unit 352 estimates an opacity image from the RGB texture image generated by the processing of step S103 in accordance with the alpha value estimation model. The alpha value estimation unit 352 supplies the opacity image to the free viewpoint image generation unit 344.

[0126] In step S105, the free-viewpoint image generation unit 344 generates a free-viewpoint image corresponding to the captured image of the virtual viewpoint based on the RGBA texture image, shape data, virtual viewpoint, and background image of the virtual viewpoint. This RGBA texture image is composed of the RGB texture image generated by the processing of step S103 and the opacity image estimated by the processing of step S104. The free-viewpoint image generation unit 344 supplies the generated free-viewpoint image to the calculation unit 345.

[0127] In step S106, the calculation unit 345 calculates a loss function by comparing the image captured from the virtual viewpoint as an ideal value of the output of the alpha value estimation model with the free viewpoint image generated by the processing of step S105 as an actual value of the output. The calculation unit 345 supplies this loss function to the update unit 346.

[0128] In step S107, the update unit 346 back-propagates the loss function calculated in the processing of step S106 to the alpha value estimation model stored in the storage unit 342, thereby updating the alpha value estimation model.

[0129] In step S108, the learning unit 323 determines whether the number of times of processing in step S107 has reached a predetermined number of times of learning. If it is determined in step S108 that the number of times of learning has not reached the predetermined number of times, the process proceeds to step S109.

[0130] In step S109, the learning unit 323 determines whether background images from N viewpoints, 3D data, and camera parameters have been input as new learning data. If it is determined in step S109 that new learning data has been input, the process returns to step S100, and the subsequent processes are repeated.

[0131] On the other hand, if it is determined in step S109 that new learning data has not been input, the process returns to step S108, and the subsequent processes are repeated.

[0132] If it is determined in step S108 that the predetermined number of learning times has been reached, the learning process ends.

[0133] As described above, in the display device 13, the candidate image generation unit 71 generates multiple candidate images using 3D data. The estimation unit 73 generates an RGB texture image by combining the multiple candidate images and estimates an opacity image from the RGB texture image. Therefore, the opacity image can be accurately estimated using the rendering results based on the 3D data. As a result, it becomes possible to accurately represent translucent objects such as hair and to separate the foreground and background with high accuracy in an RGBA texture image composed of an RGB texture image and an opacity image.

[0134] Therefore, for example, the free-viewpoint image generator 74 can use the RGBA texture image and shape data to absorb shape errors of the 3D model and generate a high-quality free-viewpoint image that accurately represents a translucent object. Because it is difficult to accurately represent a detailed shape, such as the shape of a single hair of a subject, in a 3D model, it is important to absorb shape errors of the 3D model in the free-viewpoint image.

[0135] The alpha value estimation model used to estimate the opacity image is generated by machine learning using 3D data, so by using this alpha value estimation model, the opacity image can be estimated using an opacity information estimation method suitable for images of only the foreground.

[0136] The alpha value estimation model may be a machine learning model that directly estimates an opacity image from multiple candidate images without generating an RGB texture image of a 3D model of a virtual viewpoint. The alpha value estimation model may use, as input, not only the multiple candidate images but also score information corresponding to each pixel of the candidate images.

[0137] The learning of the alpha value estimation model may be completed before the display processing, or may be performed during processing by the image processing system 10 using images captured from N viewpoints by N imaging devices 11.

[0138] 2. Computer The above-described series of processes can be executed by hardware or software. When the series of processes are executed by software, the programs that make up the software are installed on a computer. Here, the term "computer" includes computers built into dedicated hardware, and general-purpose personal computers, for example, that can execute various functions by installing various programs.

[0139] FIG. 15 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0140] In the computer, a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903 are interconnected by a bus 904.

[0141] An input / output interface 905 is also connected to the bus 904. An input unit 906, an output unit 907, a storage unit 908, a communication unit 909, and a drive 910 are connected to the input / output interface 905.

[0142] The input unit 906 includes a keyboard, mouse, microphone, touch panel, input terminal, etc. The output unit 907 includes a display, speaker, output terminal, etc. The storage unit 908 includes a hard disk, RAM disk, non-volatile memory, etc. The communication unit 909 includes a network interface, etc. The drive 910 drives removable media 911 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0143] In a computer configured as described above, the CPU 901 performs the above-described series of processes by, for example, loading a program stored in the storage unit 908 into the RAM 903 via the input / output interface 905 and the bus 904 and executing the program. The RAM 903 also stores data necessary for the CPU 901 to execute various processes as appropriate.

[0144] The program executed by the computer (CPU 901) can be provided by being recorded on a removable medium 911 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0145] In a computer, the program can be installed in the storage unit 908 via the input / output interface 905 by inserting the removable medium 911 into the drive 910. The program can also be received by the communication unit 909 via a wired or wireless transmission medium and installed in the storage unit 908. Alternatively, the program can be installed in the ROM 902 or the storage unit 908 in advance.

[0146] The program executed by the computer may be a program in which processing is performed in chronological order according to the order described in this specification, or may be a program in which processing is performed in parallel or at a required timing, such as when called. In other words, as long as no contradiction occurs, the processing of each step may be performed in an order different from the order described above. Furthermore, the processing of the steps describing this program may be performed in parallel with the processing of another program, or may be performed in combination with the processing of another program.

[0147] The device that executes the above-described series of processes by a program may be a device other than the computer shown in Fig. 15. In that case, the device has the necessary functional blocks and is configured to be able to obtain the necessary information.

[0148] 3. Application Examples Application to Video Content Production The technology according to the present disclosure can be applied to various products and services.

[0149] For example, new video content may be produced by combining a 3D model of a subject generated in this embodiment with a 3D model managed by another server. Also, by combining a background image captured by a photographing device such as Lidar with a 3D model of a subject generated in this embodiment, video content can be produced in which the subject appears to be located in the location indicated by the background image.

[0150] The 3D model of the subject generated in this embodiment is, for example, a 3D model generated by the 3D model generation unit 22 or a 3D model reconstructed by the rendering unit 33. The video content to be produced may be three-dimensional video content or two-dimensional video content converted into two dimensions.

[0151] <Application to Virtual Space Experiences> For example, in a virtual space where a user communicates as an avatar, a subject generated in this embodiment can be placed as, for example, a performer. In this case, the user can view a live-action subject in the virtual space as an avatar.

[0152] <Application to Communication with Remote Locations> For example, the transmitter 24 transmits a transmission stream of the 3D model of the subject generated by the 3D model generator 22 to the display device 13 at a remote location, allowing a user at the remote location to view the 3D model of the subject through the display device 13. For example, the transmitter 24 transmits a transmission stream of the 3D model of the subject in real time, allowing the subject and a user at a remote location to communicate in real time. Examples of such communication include communication between a subject who is a teacher and a user who is a student, or communication between a subject who is a doctor and a user who is a patient.

[0153] <Others> For example, free viewpoint images of a sports match or the like can be generated based on the 3D models of multiple subjects generated in this embodiment, or individuals can distribute their own 3D models generated in this embodiment on a distribution platform.

[0154] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0155] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present technology.

[0156] For example, the multiple technologies related to the present technology can be implemented independently and independently as long as no contradiction occurs. Of course, any multiple technologies can be implemented in combination. For example, any part or all of the above-mentioned technologies can be implemented in combination with other technologies not described above.

[0157] For example, the present technology may be configured as a cloud computing system in which a single function is shared and processed collaboratively by multiple devices via a network. For example, the rendering unit 33 may be implemented in a server external to the display device 13. In this case, information is exchanged between the display device 13 and the server.

[0158] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0159] Furthermore, when one step includes multiple processes, the processes included in that one step can be executed by one device or can be shared and executed by multiple devices. In other words, the processes included in one step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as one step.

[0160] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0161] The present technology may have the following configurations. (1) An image processing device comprising: a candidate image generation unit that generates a plurality of candidate images as candidates for an RGB texture image of a 3D model of a virtual viewpoint, using images captured from multiple viewpoints and shape data indicating the shape of a 3D model of a subject in the captured images; and an estimation unit that generates the RGB texture image by combining the plurality of candidate images generated by the candidate image generation unit and estimates an opacity image indicating the opacity of each pixel of the RGB texture image from the RGB texture image. (2) The image processing device described in (1), further comprising: a free viewpoint image generation unit that generates an image of the subject of the virtual viewpoint, using the shape data and an RGBA texture image composed of the RGB texture image and the opacity image generated by the estimation unit. (3) The image processing device described in (1) or (2), further comprising: a 3D model generation unit that generates the 3D model using the images captured from multiple viewpoints. (4) The image processing device described in any of (1) to (3), wherein the estimation unit is configured to generate the RGB texture image and estimate the opacity image using a machine learning model. (5) The image processing device according to (4), wherein the machine learning model is configured to learn an ideal value of a synthesized image, which is an image taken from the predetermined viewpoint among the multi-viewpoint images for machine learning, as an input, of an output when a plurality of candidate images for machine learning are input and are candidates for an RGB texture image of the machine learning 3D model from the predetermined viewpoint, which is generated using machine learning captured images that are images taken from a viewpoint other than a predetermined viewpoint among the multi-viewpoint images for machine learning, and machine learning shape data that indicates the shape of the machine learning 3D model that is a 3D model of a subject in the machine learning captured images, and a background image from the predetermined viewpoint, as a teacher image that is an image taken from the predetermined viewpoint among the multi-viewpoint images for machine learning. (6) The image processing device according to (5), wherein a loss function is used for training the machine learning model. (7) The image processing device according to (6), wherein the loss function is calculated using the synthesized image and the teacher image.(8) The image processing device according to (6) or (7), wherein the loss function is calculated using an opacity image of the predetermined viewpoint from the output and an opacity image estimated from the teacher image. (9) An image processing method including: generating a plurality of candidate images that are candidates for an RGB texture image of the 3D model of a virtual viewpoint, using captured images from multiple viewpoints and shape data that indicates a shape of a 3D model of a subject in the captured images, generating the RGB texture image by synthesizing the plurality of candidate images, and estimating an opacity image that indicates the opacity of each pixel of the RGB texture image from the RGB texture image. (10) A program that causes a computer to function as an image processing device, comprising: a candidate image generation unit that generates a plurality of candidate images that are candidates for the RGB texture image of the 3D model of a virtual viewpoint, using captured images from multiple viewpoints and shape data that indicates the shape of a 3D model of a subject in the captured images, and an estimation unit that generates the RGB texture image by synthesizing the plurality of candidate images generated by the candidate image generation unit, and estimating an opacity image that indicates the opacity of each pixel of the RGB texture image from the RGB texture image.

[0162] 10 Image processing system, 13 Display device, 71 Candidate image generation unit, 73 Estimation unit, 74 Free viewpoint image generation unit, 22 3D model generation unit

Claims

1. An image processing device comprising: a candidate image generation unit that generates a plurality of candidate images that are candidates for an RGB texture image of a 3D model of a virtual viewpoint, using captured images from multiple viewpoints and shape data that indicates the shape of a 3D model of a subject in the captured images; and an estimation unit that generates the RGB texture image by synthesizing the plurality of candidate images generated by the candidate image generation unit, and estimates an opacity image that indicates the opacity of each pixel of the RGB texture image from the RGB texture image.

2. The image processing device according to claim 1, further comprising a free viewpoint image generation unit that generates an image of the subject at the virtual viewpoint using an RGBA texture image consisting of the RGB texture image and the opacity image generated by the estimation unit and the shape data.

3. The image processing device according to claim 1, further comprising: a 3D model generation unit that generates the 3D model using the multi-viewpoint captured images.

4. The image processing device according to claim 1, wherein the estimation unit is configured to generate the RGB texture image and estimate the opacity image using a machine learning model.

5. The image processing device according to claim 4, wherein the machine learning model is configured to learn an ideal value of a composite image of an output when a plurality of candidate machine learning images, which are candidates for an RGB texture image of the machine learning 3D model from a predetermined viewpoint, generated using machine learning images, which are images taken from a multi-viewpoint among images taken from multiple viewpoints for machine learning, other than a predetermined viewpoint, and machine learning shape data indicating the shape of the machine learning 3D model, which is a 3D model of a subject in the machine learning images, are input, and a background image from the predetermined viewpoint, as a teacher image, which is an image taken from the multi-viewpoint among images taken from multiple viewpoints for machine learning.

6. The image processing device according to claim 5, configured to use a loss function for training the machine learning model.

7. The image processing device according to claim 6, configured to calculate the loss function using the synthetic image and the training image.

8. An image processing device according to claim 6, configured to calculate the loss function using an opacity image of the predetermined viewpoint among the outputs and an opacity image estimated from the teacher image.

9. An image processing method comprising: generating a plurality of candidate images that are candidates for an RGB texture image of the 3D model of a virtual viewpoint, using captured images from multiple viewpoints and shape data that indicates the shape of a 3D model of a subject in the captured images; generating the RGB texture image by synthesizing the plurality of candidate images; and estimating an opacity image that indicates the opacity of each pixel of the RGB texture image from the RGB texture image.

10. A program for causing a computer to function as an image processing device comprising: a candidate image generation unit that generates multiple candidate images that are candidates for an RGB texture image of a 3D model of a virtual viewpoint, using images captured from multiple viewpoints and shape data that indicate the shape of a 3D model of a subject in the captured images; and an estimation unit that generates the RGB texture image by combining the multiple candidate images generated by the candidate image generation unit, and estimates an opacity image that indicates the opacity of each pixel of the RGB texture image from the RGB texture image.

Images