Image processing device, image processing method, and program

Abstract

The present technology relates to an image processing device, an image processing method, and a program that enable an appropriate tracking period to be set by part for mesh tracking of part units. For each final part of a 3D model in imaged frames, a dividing unit divides the imaged frames into one or more tracking periods on the basis of movement information indicating movement of the final part between frames and / or topology change information indicating a detection region of a change in the topology of the 3D model between frames. A block tracking unit performs mesh tracking on the 3D model for each final part and for each tracking period, and generates shape data indicating the shape of the final part. The present technology can be applied to a 3D data generating device, for example.

Description

Image processing device, image processing method, and program

[0001] The present technology relates to an image processing device, an image processing method, and a program, and in particular to an image processing device, an image processing method, and a program that enable setting an appropriate tracking period for each part in mesh tracking on a part-by-part basis.

[0002] There is a technology called volumetric capture (volumetric video) that generates images from free virtual viewpoints in all directions (360 degrees) from multi-viewpoint images taken by surrounding a subject (object) with multiple cameras.

[0003] In volumetric capture, for example, a 3D model showing the 3D shape of the subject using a mesh or the like is reconstructed from multiple-viewpoint images captured by surrounding the subject using multiple cameras, and a texture image to be applied to the 3D model is generated. Note that a texture image is an image that represents the color and pattern of the 3D model using RGB color information. Then, based on this texture image and the 3D model of the subject, an image of the subject from a virtual viewpoint is rendered as a free-viewpoint image. This free-viewpoint image is used for 2D video viewing, naked-eye stereoscopic viewing (Spatial Reality Display), mobile AR (Augmented Reality), and the like.

[0004] In volumetric capture, mesh tracking is a technique for tracking the movement of a 3D model of a subject in a captured image, which is a moving image (see, for example, Non-Patent Documents 1 and 2). In mesh tracking, a 3D model in a frame called a key frame is used as a template, and in frames other than the key frame, the template is used as a non-rigid body for fitting, thereby tracking the movement of the 3D model.

[0005] Mesh tracking is a four-dimensional sequence consisting of three spatial dimensions and time, so it is possible to separate the shape and motion of a 3D model. Therefore, by performing smoothing filtering and stabilization processing only on the motion of the 3D model, it is possible to suppress flickering in free-viewpoint images generated using the 3D model without losing the detailed shape of the subject.

[0006] On the other hand, there is a technology called UV texture mapping, which divides a 3D model into parts, develops it on a plane on UV coordinates, and then places the texture images of the corresponding parts on that plane to generate a UV map.The parts include the face, head, left arm, right arm, chest, back, left leg, right leg, left foot (left shoe), and right foot (right shoe).

[0007] When UV texture mapping is performed on a 3D model with mesh tracking, the connection information (structure) of the triangular mesh of the 3D model is the same between frames. Therefore, the position of each mesh vertex on the UV coordinate system is the same between frames, improving the compression efficiency when compressing the UV map as a moving image. In addition, the material information of each mesh vertex can be associated between frames. This improves the transmission efficiency of UV maps, material information, etc.

[0008] In contrast, when UV texture mapping is performed on a 3D model without mesh tracking, the mesh connectivity information of the 3D model differs between frames. As a result, the UV coordinate position of each mesh vertex differs between frames, resulting in poor UV map compression efficiency. It is also difficult to associate the material information of each mesh vertex between frames. This results in poor transmission efficiency for UV maps and material information.

[0009] In mesh tracking, if the topology of at least a part of the object changes or if artifacts that are visible in the free viewpoint image occur due to the accumulation of fitting errors, it is necessary to update the keyframes. However, if mesh tracking is performed on an object-by-object basis, it is necessary to update the keyframes for the entire object even if the topology changes or artifacts occur only in a part of the object.

[0010] Therefore, in parts of the subject where there is almost no movement and fitting is performed with small errors, keyframe updates, i.e., changes in the shape of the template, may be performed unnecessarily, resulting in noticeable flickering in the free-viewpoint image. Furthermore, when keyframes are updated, the template mesh connectivity information changes, which in turn changes the UV coordinate positions corresponding to each vertex of the mesh. This results in significant changes in the UV map image, which reduces compression efficiency. Therefore, it is desirable to avoid unnecessary keyframe updates.

[0011] Here, a mesh tracking method has been devised that includes a first pass in which mesh tracking is performed on a fixed part-by-part basis, and a second pass in which mesh tracking is performed on the entire integrated shape by stitching the parts together after mesh tracking (see, for example, Patent Document 1).

[0012] In the mesh tracking described in Patent Document 1, mesh tracking is performed in fixed region units in the first pass, but keyframes are updated for the entire shape in the second pass. As a result, keyframes are updated even for regions where keyframe updates are not required, which can cause noticeable flicker in the free viewpoint image.

[0013] Alvaro Collet, Ming Chuang, Pat Sweeney, Don Gillett, Dennis Evseev, David Calabrese, Huges Hoppe, Adam Kirk, Steven Sullivan, “High-quality streamable free-viewpoint video”, ACM Transaction Graphics(SIGGRAPH) Volume34 Issue 4, 2015, p.1-13Hao Li, “Robust single-View geometry and motion reconstruction”, ACM Transaction Graphics Volume28 Issue 5, 2009, p.1-10 Special Publication No. 2020-518080

[0014] Therefore, in mesh tracking in units of parts, it is desirable to prevent key frames from being updated in parts that do not need to be updated by appropriately setting the update timing of key frames for each part. However, a method for appropriately setting the update timing of key frames for each part, i.e., the tracking period during which mesh tracking is performed using the same key frames, has not been devised.

[0015] The present technology has been made in view of such circumstances, and makes it possible to set an appropriate tracking period for each part in mesh tracking in units of parts.

[0016] An image processing device or program according to one aspect of the present technology is an image processing device or a program for causing a computer to function as an image processing device, which includes: a division unit that divides, for each part of a 3D model in multiple frames, the multiple frames into one or more tracking periods based on at least one of motion information indicating the movement of the part between the frames and topology change information indicating a detection area of ​​a change in the topology of the 3D model between the frames; and a tracking unit that performs mesh tracking on the 3D model for each part and for each tracking period, and generates shape data indicating the shape of the part.

[0017] An image processing method according to one aspect of the present technology is an image processing method that includes: dividing, for each part of a 3D model in a plurality of frames, the plurality of frames into one or more tracking periods based on at least one of motion information indicating the movement of the part between the frames and topology change information indicating a detection area for a change in the topology of the 3D model between the frames; and performing mesh tracking on the 3D model for each part and for each tracking period to generate shape data indicating the shape of the part.

[0018] In one aspect of the present technology, for each part of a 3D model in multiple frames, the multiple frames are divided into one or more tracking periods based on at least one of motion information indicating the movement of that part between the frames and topology change information indicating a detection area of ​​a change in the topology of the 3D model between the frames, and mesh tracking is performed on the 3D model for each part and for each tracking period, and shape data indicating the shape of the part is generated.

[0019] The image processing device may be a standalone device or a module incorporated into another device.

[0020] 18 is a block diagram showing an example configuration of a first embodiment of an image processing system to which the present technology is applied. FIG. 19 is a diagram showing an example arrangement of the imaging device of FIG. 1. FIG. 20 is a block diagram showing an example configuration of the 3D model generation unit of FIG. 1. FIG. 21 is a block diagram showing an example configuration of the tracking unit of FIG. 3. FIG. 30 is a diagram explaining a first example of determining a tracking period. FIG. 31 is a diagram explaining a first example of tracking blocks. FIG. 32 is a diagram explaining a second example of determining a tracking period. FIG. 33 is a diagram explaining an example of determining a final part. FIG. 34 is a diagram explaining a second example of tracking blocks. FIG. 35 is a diagram explaining a first example of selecting key frames. FIG. 36 is a diagram explaining a second example of selecting key frames. FIG. 37 is a diagram showing an example of a UV map. FIG. 38 is a diagram showing an example of a UV map in which regions are not divided for each part. FIG. 39 is a flowchart explaining 3D data generation processing. FIG. 39 is a flowchart explaining display processing. FIG. 40 is a block diagram showing an example configuration of a second embodiment of an image processing system to which the present technology is applied. FIG. 41 is a block diagram showing an example configuration of the 3D model generation unit 222 of FIG. 17. FIG. 42 is a block diagram showing an example configuration of the tracking unit 253 of FIG. 18. FIG. 43 is a flowchart explaining 3D data generation processing. FIG. 44 is a block diagram showing an example configuration of the hardware of a computer.

[0021] Hereinafter, modes for carrying out the present technology (hereinafter referred to as embodiments) will be described. The description will be made in the following order: 1. First embodiment (image processing system for dividing initial body parts based on user input) 2. Second embodiment (image processing system for dividing initial body parts using a human body parametric model) 3. Computer 4. Application example

[0022] 1. First Embodiment Configuration Example of Image Processing System FIG. 1 is a block diagram showing a configuration example of a first embodiment of an image processing system to which the present technology is applied.

[0023] 1 is composed of N (N is plural) image capture devices 11-1 to 11-N, a 3D data generation device 12, and a display device 13. The image processing system 10 generates free viewpoint images, which are moving images, using captured images, which are moving images of multiple frames from N viewpoints, acquired by the image capture devices 11-1 to 11-N, and displays the free viewpoint images.

[0024] Specifically, the image capturing devices 11-1 to 11-N are disposed at different positions and capture images synchronously. Images captured from N viewpoints by the image capturing devices 11-1 to 11-N are input to the 3D data generating device 12.

[0025] In the following description, unless it is necessary to distinguish between the image capturing devices 11-1 to 11-N, they will be collectively referred to as the image capturing device 11. Image capturing by the N image capturing devices 11 may be performed virtually by physically moving one image capturing device to each of the positions of the N image capturing devices 11. The N image capturing devices 11 may be provided outside the image processing system 10.

[0026] The 3D data generating device 12 (image processing device) is composed of an image acquiring unit 21 , a 3D model generating unit 22 , a touch panel 23 , an encoding unit 24 , and a transmitting unit 25 .

[0027] The image acquisition unit 21 acquires captured images from N viewpoints supplied from N image capture devices 11. The image acquisition unit 21 performs calibration based on the captured images from the N viewpoints and acquires internal parameters and external parameters of each image capture device 11 as camera parameters. The image acquisition unit 21 supplies the captured images from the N viewpoints and the camera parameters of each image capture device 11 to the 3D model generation unit 22. The image acquisition unit 21 generates depth maps for each of the multiple viewpoints based on the captured images from the N viewpoints and supplies them to the 3D model generation unit 22. A depth map for a specific viewpoint is an image in which pixel values ​​are depth values ​​indicating the distance from that viewpoint to a subject.

[0028] The 3D model generation unit 22 generates a 3D model that represents the three-dimensional shape of the subject in each frame of the captured image based on the captured images from N viewpoints and camera parameters supplied from the image acquisition unit 21. Methods for generating 3D models include Visual Hull, Shape from Silhouette, and Multi-view Stereo. When Visual Hull is used as the 3D model generation method, the 3D model generation unit 22 generates a 3D model of the subject by carving out the three-dimensional shape of the subject using silhouette images corresponding to the captured images from the N viewpoints. Since this 3D model is generated using the captured images, it is a real-life 3D model.

[0029] The 3D model generation unit 22 deforms the generated 3D model with high precision using the depth map supplied from the image acquisition unit 21. Note that the image acquisition unit 21 does not necessarily have to generate a depth map, and the 3D model generation unit 22 does not necessarily have to deform the 3D model with high precision.

[0030] The 3D model generation unit 22 supplies the 3D model to the touch panel 23 for display. The 3D model generation unit 22 acquires area information indicating the area of ​​each part of the 3D model, which is supplied from the touch panel 23 in response to a user's touch operation on the touch panel 23 displaying the 3D model. The 3D model generation unit 22 divides the 3D model into parts based on the area information. The 3D model generation unit 22 performs mesh tracking on the 3D model for all frames corresponding to the captured image (hereinafter referred to as captured frames) for each part, and generates shape data indicating the surface shape of each part of the captured frame.

[0031] The shape data for each part of the shooting frame is composed of, for example, mesh data of the shape in the key frame of that part and data representing the movement of that shape in each frame other than the key frame. Mesh data is data called a polygon mesh that represents the shape by the connections between vertices.

[0032] The 3D model generation unit 22 generates a UV map for each frame based on the shape data of each part in the photographed frame and the photographed images from N viewpoints, in which a texture image of that part is arranged in an area assigned to that part. Texture image formats include a view-dependent format, in which the color changes depending on the viewpoint position (view direction), and a view-independent format, in which the color does not change depending on the viewpoint position. The 3D model generation unit 22 supplies the encoding unit 24 with 3D data in which the shape data of each part in the photographed frame and the UV map are associated with each other.

[0033] The touch panel 23 displays the 3D model supplied from the 3D model generation unit 22, for example, on a three-dimensional CG (Computer Graphics) tool screen. The user performs a touch operation to specify each part of the 3D model displayed on the touch panel 23. The touch panel 23 generates area information in response to the touch operation and supplies the area information to the 3D model generation unit 22.

[0034] The encoding unit 24 converts the 3D data supplied from the 3D model generation unit 22 into a format suitable for transmission and storage. Specifically, the encoding unit 24 converts shape data of each part of the captured frame in the 3D data into a depth map for each frame of the captured frame from one or more viewpoints. The encoding unit 24 encodes (compresses) the depth map and UV map for each frame of the captured frame to generate a transmission stream, which is a bit stream.

[0035] In this case, the depth map and UV map of one frame may be combined and encoded as one frame image, or may be encoded as separate images. Since the depth map and UV map are two-dimensional images, they can be encoded using existing two-dimensional image encoding technology such as AVC (Advanced Video Coding). The encoding unit 24 supplies the transmission stream to the transmitting unit 25.

[0036] The transmitting unit 25 transmits the transmission stream supplied from the encoding unit 24 to the display device 13 .

[0037] In the 3D data generation device 12 configured as described above, for example, the image acquisition unit 21, the 3D model generation unit 22, the touch panel 23, and the encoding unit 24 can perform processing offline in advance, and the transmission unit 25 can transmit the transmission stream at a predetermined timing. The transmission unit 25 can also transmit the transmission stream obtained as a result of the processing by the image acquisition unit 21, the 3D model generation unit 22, the touch panel 23, and the encoding unit 24 in real time.

[0038] The display device 13 includes a receiving unit 31, a decoding unit 32, a rendering unit 33, and a display unit .

[0039] The receiving unit 31 receives the transmission stream transmitted from the transmitting unit 25 and supplies it to the decoding unit 32 .

[0040] The decoding unit 32 decodes the transmission stream supplied from the receiving unit 31 to obtain a depth map and a UV map for each frame. The decoding unit 32 generates shape data for each part of the captured frame from the depth map for each frame of the captured frame, and supplies 3D data in which the shape data is associated with the UV map of the captured frame to the rendering unit 33.

[0041] The rendering unit 33 performs rendering by projecting a mesh of the 3D model at the virtual viewpoint for each frame and performing texture mapping using a UV map based on the 3D data supplied from the decoding unit 32 and the virtual viewpoint supplied from the display unit 34. In this way, the rendering unit 33 generates an image of the subject in the captured frame of the virtual viewpoint as a free viewpoint image.

[0042] There are two types of texture mapping: a view-dependent method that takes the virtual viewpoint into account, and a view-independent method that does not. With view-dependent texture mapping, the texture image to be applied changes depending on the position of the virtual viewpoint, so it is possible to generate a higher quality free-viewpoint image than with view-independent texture mapping. In contrast, with view-independent texture mapping, the texture image to be applied is the same regardless of the position of the virtual viewpoint, so it requires less processing than view-dependent texture mapping.

[0043] The display unit 34 is configured by a 2D monitor or a 3D monitor of a head-mounted display, a spatial display, a mobile phone, a television receiver, a personal computer (hereinafter referred to as a PC), etc. The display unit 34 performs display based on the free viewpoint image supplied from the rendering unit 33.

[0044] The display unit 34 detects the user's viewing location (Region of Interest) and, based on the viewing location, determines the user's desired viewing viewpoint in all 360-degree directions as a virtual viewpoint, regardless of the position of the image capturing device 11. The display unit 34 supplies the virtual viewpoint to the rendering unit 33. Note that the display unit 34 may be provided outside the display device 13 or the image processing system 10.

[0045] 1, the 3D data generator 12 transmits a transmission stream and the display device 13 performs display based on the transmission stream, but the 3D data generator 12 and the display device 13 may be integrated. In this case, the image processing system 10 does not need to include the encoding unit 24, the transmitting unit 25, the receiving unit 31, and the decoding unit 32 because a single device such as a PC performs processes from generating 3D data to displaying a free viewpoint image.

[0046] When the image processing system 10 is implemented, it may be implemented entirely by one operator, or it may be implemented separately by multiple operators. For example, when the image processing system 10 is implemented separately by three operators A to C, operator A implements the image acquisition unit 21, 3D model generation unit 22, touch panel 23, and encoding unit 24 to generate a transmission stream. Operator B, as a platformer, implements the transmission unit 25 to distribute the transmission stream. Operator C implements the display device 13 to generate a free viewpoint image based on the transmission stream and perform a display based on the free viewpoint image.

[0047] <Example of Arrangement of Camera Devices> FIG. 2 is a diagram showing an example of arrangement of the camera devices 11 in FIG.

[0048] In the example of Fig. 2, N is 8. As shown in Fig. 2, the image capturing devices 11-1 to 11-8 are arranged to surround the subject 41. In the example of Fig. 2, the image capturing devices 11 are arranged at equal intervals, but the positions of the image capturing devices 11 can be any positions.

[0049] <Configuration Example of 3D Model Generator> FIG. 3 is a block diagram showing a configuration example of the 3D model generator 22 in FIG.

[0050] The 3D model generation unit 22 in FIG. 3 is made up of a 3D shape restoration unit 51, an initial part division unit 52, a tracking unit 53, a UV development unit 54, and an overlap processing unit 55.

[0051] The 3D shape restoration unit 51 generates a 3D model of the subject in the captured image of each frame based on the captured images from N viewpoints and camera parameters supplied from the image acquisition unit 21 in Fig. 1. At this time, the 3D shape restoration unit 51 deforms the generated 3D model, as necessary, using the depth map supplied from the image acquisition unit 21. The 3D shape restoration unit 51 supplies the 3D model of each frame to the initial part division unit 52.

[0052] The initial part division unit 52 determines a predetermined frame (e.g., the first frame) as an initial key frame, which is the initial value of the key frame. The initial part division unit 52 supplies the 3D model of the initial key frame, among the 3D models of each frame supplied from the 3D shape restoration unit 51, to the touch panel 23 of FIG. 1 for display. The initial part division unit 52 acquires, from the touch panel 23, area information corresponding to a user's touch operation on the 3D model. The initial part division unit 52 (area division unit) divides an initial template, which is the 3D model of the initial key frame, into one or more initial parts, based on the 3D model of the initial key frame and the area information. The initial part division unit 52 supplies the initial template of each initial part and the 3D model of each frame to the tracking unit 53.

[0053] The tracking unit 53 determines tracking blocks based on the initial templates of each initial region supplied from the initial region division unit 52 and the 3D model of each frame. A tracking block is a block obtained by dividing the mesh tracking target temporally and spatially. Specifically, a tracking block is the final region for which mesh tracking is performed using the same key frame within a tracking period in which mesh tracking is performed using the same key frame. This final region may be the initial region itself, or a region obtained by dividing or integrating the initial region.

[0054] The tracking unit 53 performs mesh tracking on the 3D model of each frame in units of tracking blocks. The tracking unit 53 supplies the resulting shape data of each final portion of the captured frame and block information representing each tracking block to the UV development unit 54. The tracking unit 53 also supplies the shape data of each final portion of the captured frame to the overlap processing unit 55.

[0055] The UV development unit 54 divides the entire area of ​​the UV map into initial part areas, which are rectangular areas to be allocated to each initial part, based on the block information supplied from the tracking unit 53. The UV development unit 54 generates final part areas, which are rectangular areas to be allocated to each final part, by leaving each initial part area of ​​the UV map for each frame as is, or by dividing or integrating it, based on the block information.

[0056] The UV development unit 54 performs planar development (UV development) of each final part onto its final part area for each frame based on the captured images from the N viewpoints and the shape data of each final part supplied from the image acquisition unit 21, and maps the texture image. As a result, the UV development unit 54 generates a UV map (texture map) for each frame in which the texture image of the final part is planar developed and arranged (repacked) in the final part area of ​​each final part. The UV development unit 54 supplies the UV map for each frame to the overlap processing unit 55.

[0057] The overlap processing unit 55 performs non-rigid registration on the shape data of the overlapping portion of the shape data of each final portion supplied from the tracking unit 53 .

[0058] Specifically, when a 3D model is generated based on the shape data of each final part obtained as a result of mesh tracking using different templates for each final part, inconsistencies occur at the boundaries of the final parts. Because the positions of the boundaries of the final parts change depending on the movement of the final parts, mesh tracking for each final part by the tracking unit 53 is performed by providing overlapping portions with adjacent final parts for each final part. However, even when overlapping portions are provided for each final part, gaps occur at the boundaries between each final part due to subtle differences in shape between the final parts. As a result, when adding shading to the 3D model using lighting, for example, shadows occur in the gaps. Therefore, the overlap processing unit 55 prevents gaps from occurring at the boundaries between each final part by performing non-rigid registration on the shape data of the overlapping portions of the shape data for each final part.

[0059] Details of non-rigid registration techniques are described, for example, in Bailin Deng, Yuxin Yao, Roberto M. Dyke, Juyong Zhang, “A Survey of Non-Rigid 3D Registration”, Computer Graphics Forum Volume 41 Issue 2, 2022, pp. 559-589.

[0060] The overlap processing unit 55 supplies 3D data in which the UV map of each frame supplied from the UV development unit 54 is associated with the shape data of each final part for which non-rigid registration has been performed to the encoding unit 24 in Fig. 1. The overlap processing unit 55 may be provided between the tracking unit 53 and the UV development unit 54.

[0061] The UV unwrapping unit 54 may use an alpha channel that represents the opacity of the UV map to set the alpha value of the texture image in the overlapping portion to a relatively high transparency value, which can suppress flickering in the free viewpoint image due to changes in the shape of the overlapping portion.

[0062] <Configuration Example of Tracking Unit> FIG. 4 is a block diagram showing a configuration example of the tracking unit 53 in FIG.

[0063] The tracking unit 53 in FIG. 4 is made up of an analysis unit 71 , a division unit 72 , a key frame selection unit 73 , and a block tracking unit 74 .

[0064] The analysis unit 71 (estimation unit) estimates motion information indicating the inter-frame motion of each initial part of the 3D model based on the initial template of each initial part supplied from the initial part division unit 52 in Figure 3 and the 3D model of each frame.

[0065] As a method for estimating the motion information of each initial region, for example, a non-rigid registration method based on a deformation graph can be adopted. The non-rigid registration method based on a deformation graph is described, for example, in Hao Li, Robert Sumner, and Mark Pauly, “GLOBAL CORRESPONDENCE OPTIMIZATION FOR NON-RIGID REGISTRATION OF DEPTH SCANS,” Computer Graphics Forum 27(5), Proceedings of the Sixth Eurographics Symposium on Geometry Processing 2008. The estimation of the motion information of each initial region may be performed in parallel for each frame. The analysis unit 71 supplies the motion information of each initial region for each frame to the division unit 72, and supplies fitting error information indicating the fitting error for each frame calculated when estimating the motion information to the key frame selection unit 73.

[0066] The analysis unit 71 also performs mesh tracking based on the initial template of each initial portion and the 3D model of each frame. The analysis unit 71 then analyzes the fitting error and bidirectional fitting results in the mesh tracking to detect topology changes in the 3D model between frames that make mesh tracking difficult. This analysis method is described, for example, in K. Zampogiannis, C. Fermuller, and Y. Aloimonos, “Topology-Aware Non-Rigid Point Cloud Registration,” IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), vol. 43, no. 3, pp. 1056-1069, March 1, 2021. The analysis unit 71 supplies topology change information indicating a detection area of ​​a topology change in the 3D model between each frame to the division unit 72. Note that if a topology change is not detected, the topology change information is information that does not indicate any area as a detection area.

[0067] The dividing unit 72 determines the final part of the 3D model based on at least one of the motion information and topology change information of each initial part supplied from the analysis unit 71. The dividing unit 72 divides the captured frames into one or more tracking periods for each final part based on at least one of the motion information and topology change information of each initial part, thereby determining the update frequency and update timing of the key frames for each final part. In this manner, the dividing unit 72 sets tracking blocks. The dividing unit 72 supplies block information of these tracking blocks to the key frame selection unit 73.

[0068] For each tracking block indicated by the block information, the key frame selection unit 73 selects one frame within the tracking period corresponding to that tracking block as a key frame. For example, the key frame selection unit 73 selects a frame at a predetermined position within the tracking period, such as the first frame or the center frame, as a key frame. Alternatively, the key frame selection unit 73 selects a predetermined frame within the tracking period as a key frame based on the fitting error information supplied from the analysis unit 71. The key frame selection unit 73 supplies the block information and key frame information indicating the key frame of each tracking block to the block tracking unit 74.

[0069] The block tracking unit 74 performs mesh tracking in parallel for each tracking block on the 3D model of each frame supplied from the initial part division unit 52 based on the block information and key frame information. Specifically, for each tracking block, the block tracking unit 74 performs fitting processing on the entire 3D model in parallel using the corresponding final part in the key frame corresponding to that tracking block as a template. The block tracking unit 74 supplies shape data of each final part of the captured frame obtained as a result of mesh tracking in tracking block units to the UV development unit 54 and overlap processing unit 55 in FIG. 3. The block tracking unit 74 supplies the block information to the UV development unit 54.

[0070] <Explanation of a First Example of Determining a Tracking Period> FIG. 5 is a diagram illustrating a first example of determining a tracking period by the dividing unit 72 of FIG.

[0071] In the example of Fig. 5, there are five captured frames, from frame 1 to frame 5. Fig. 5A shows 3D models 80-1 to 80-5 generated using captured images from N viewpoints for each of frames 1 to 5. Fig. 5B shows 3D models 90-1 to 90-5 corresponding to the shape data for frames 1 to 5 obtained as a result of performing mesh tracking on 3D models 80-1 to 80-5 on a 3D model-by-3D model basis using 3D model 80-1 as a template.

[0072] In the example of Figure 5, the topology does not change in the 3D models 80-1 to 80-5. However, in the example of Figure 5, the movement of the left torso arms 81-1 to 81-5, which are the torso and left arm regions of the 3D models 80-1 to 80-5, is large. Therefore, the accumulation of fitting errors with the templates is large in the left torso arms 91-1 to 91-5 of the 3D models 90-1 to 90-5, which correspond to the left torso arms 81-1 to 81-5. Therefore, it is necessary to frequently update the key frames corresponding to the templates in the left torso arms 81-1 to 81-5.

[0073] On the other hand, the movement of both legs 82-1 to 82-5 of the 3D models 80-1 to 80-5 is small. Therefore, for both legs 92-1 to 92-5 of the 3D models 90-1 to 90-5 corresponding to both legs 82-1 to 82-5, the accumulation of fitting errors with the templates is small. Therefore, for both legs 82-1 to 82-5, it is not necessary to frequently update the key frames corresponding to the templates.

[0074] As described above, the appropriate key frame update frequency differs depending on the magnitude of movement of each part of the 3D model. Therefore, the dividing unit 72 determines the tracking period for each final part based on the movement information of each initial part so that the tracking period becomes shorter as the movement becomes greater.

[0075] In addition, if the analysis unit 71 can estimate information indicating the complexity of the movement between frames of each initial part, the division unit 72 may determine the tracking period based on that information so that the more complex the movement, the shorter the tracking period.

[0076] <First Example of Tracking Blocks> FIG. 6 is a diagram showing an example of tracking blocks determined by the dividing unit 72 for the 3D models 80-1 to 80-5 in FIG.

[0077] In the table of Fig. 6, the first to fifth frames corresponding to the 3D models 80-1 to 80-5 are listed as items in each column, and the name of each final part is listed as an item in each row in the first column from the left. In the second and subsequent rows from the top and columns from the left in the table of Fig. 6, the tracking blocks corresponding to those rows and columns are listed. The same applies to Figs. 10 and 11, which will be described later.

[0078] As described above, the topology of 3D models 80-1 to 80-5 does not change, but the movement of the left arms of the torso 81-1 to 81-5 is large and the movement of both legs 82-1 to 82-5 is small. Therefore, in the example of Fig. 6, the dividing unit 72 determines the final parts of 3D models 80-1 to 80-5 to be the same as the initial parts: the face, torso, left arm, right arm, and legs, based on the movement information and topology change information of each initial part.

[0079] The dividing unit 72 also determines a tracking period for each final part based on the motion information and topology change information for each part. Specifically, the dividing unit 72 determines the tracking period for the torso and left arm parts corresponding to the torso left arm parts 81-1 to 81-5, which have large motion, to be one frame period. On the other hand, the dividing unit 72 determines the tracking period for the legs corresponding to both legs 82-1 to 82-5, which have small motion, to be the period from the first frame to the fifth frame, i.e., the entire period of the imaging frames.

[0080] In this way, the dividing unit 72 sets tracking blocks with the entire period of the imaging frames as the tracking period and the body part as the face or the legs. The dividing unit 72 sets tracking blocks with each frame of the imaging frames as the tracking period and the body part as the left arm or the torso. The dividing unit 72 sets a tracking block with the first to fourth frames as the tracking period and the body part as the right arm, and a tracking block with the fifth frame as the tracking period and the body part as the right arm.

[0081] <Explanation of a Second Example of Determining a Tracking Period> FIG. 7 is a diagram illustrating a second example of determining a tracking period by the dividing unit 72. In FIG.

[0082] In the example of Figure 7, there are four captured frames, the first to fourth frames, and the 3D shape restoration unit 51 generates 3D models 111-1 to 111-4 for the first to fourth frames. At this time, the initial part division unit 52 supplies, for example, the 3D model 111-1 to the touch panel 23 as the 3D model of the initial key frame and displays it. The user performs a touch operation to paint each part of the 3D model 111-1 displayed on the touch panel 23 in a different color. As a result, for example, lines 121-1 to 121-6 of different colors are drawn on the head 120-1, torso 120-2, left arm 120-3, right arm 120-4, left leg 120-5, and right leg 120-6 of the 3D model 111-1 being displayed.

[0083] Touch panel 23 supplies area information of head 120-1, torso 120-2, left arm 120-3, right arm 120-4, left leg 120-5, and right leg 120-6 corresponding to lines 121-1 to 121-6 to initial part division section 52. Based on this area information, initial part division section 52 then divides 3D model 111-1 into head 120-1, torso 120-2, left arm 120-3, right arm 120-4, left leg 120-5, and right leg 120-6 as initial parts.

[0084] In the example of Figure 7, the right arm of 3D model 111-1 is spread out, while the right arm of 3D model 111-2 is closed with the right hand resting on the hip. The left hand pose of 3D model 111-1 is a rock-paper-scissors symbol, while the left hand pose of 3D model 111-2 is a scissors-paper symbol. Therefore, the analysis unit 71 detects a topology change in the left arm 120-3 and the right arm 120-4 between the first and second frames. As a result, the division unit 72 determines the tracking periods of the left arm 120-3 and the right arm 120-4 to be the first frame, respectively.

[0085] In 3D model 111-3, the left hand pose becomes rock-paper-scissors again, and in 3D model 111-4, the left hand pose remains rock-paper-scissors. Therefore, the analysis unit 71 detects a topology change in the left arm 120-3 between the second and third frames, but does not detect a topology change in the left arm 120-3 between the third and fourth frames. As a result, the division unit 72 determines the tracking period of the left arm 120-3 to be one frame period of the second frame and two frame periods of the third and fourth frames.

[0086] On the other hand, in the 3D models 111-2 to 111-4, the right arm 120-4 remains closed with its right hand on its hip, so the analysis unit 71 does not detect a topology change between the second and fourth frames. Furthermore, the movement of the right arm 120-4 in the 3D models 111-2 to 111-4 is small. Based on the above, the division unit 72 determines the tracking period for the right arm 120-4 to be a three-frame period from the second to the fourth frames.

[0087] <Explanation of an Example of Determining the Final Region> FIG. 8 is a diagram illustrating an example of determining the final region by the dividing unit 72. In FIG.

[0088] 8, there are four captured frames, the first to fourth frames, and 3D models 141-1 to 141-4 for the first to fourth frames are generated by the 3D shape restoration unit 51. The initial parts are a face 150-1, a torso 150-2, a left arm 150-3, a right arm 150-4, a left leg 150-5, and a right leg 150-6 of the 3D model 141-1.

[0089] 8, the hand pose of the left hand of the 3D model 141-1 is a rock-paper-scissors symbol, while the hand pose of the left hand of the 3D model 141-2 is a rock-paper-scissors symbol. Therefore, the analysis unit 71 detects a change in the topology of the left hand of the left arm 150-3 between the first frame and the second frame. Therefore, the division unit 72 divides the left arm 150-3 into a left upper arm 150-3a and a left forearm peripheral portion 150-3b including the left side other than the left upper arm, and determines these as the final portions of the 3D model 141-2.

[0090] The left hand pose of the 3D model 141-3 is hang-loose, but the topology of the left upper arm 150-3a does not change from the topology of the left upper arm 150-3a of the 3D model 141-2. Therefore, the division unit 72 determines the final parts of the 3D model 141-3 to be the left upper arm 150-3a and the left forearm peripheral part 150-3b.

[0091] The left hand pose of the 3D model 141-4 again becomes the paper pose in rock-paper-scissors, and the topology of the left upper arm 150-3a changes from the topology of the left upper arm 150-3a of the 3D model 141-3. Therefore, the division unit 72 determines the final region corresponding to the left arm 150-3 of the 3D model 141-4 to be the entire left arm 150-3, which is the same as the initial region.

[0092] <Second Example of Tracking Blocks> FIG. 9 is a diagram showing an example of tracking blocks determined by the dividing unit 72 for the 3D models 141-1 to 141-4 in FIG.

[0093] In the table of Fig. 9, the first to fourth frames corresponding to the 3D models 141-1 to 141-4 are listed as items in each column, and the name of each final part is listed as an item in each row in the first column from the left. In the rows from the second row from the top and in the columns from the second column from the left in the table of Fig. 9, the tracking blocks corresponding to the row and column are listed.

[0094] The dividing unit 72 first determines the final parts of the 3D model 141-1 of the first frame to be the same as the initial parts: face 150-1, torso 150-2, left arm 150-3, right arm 150-4, left leg 150-5, and right leg 150-6.

[0095] As described in Figure 8, the division unit 72 divides the left arm portion 150-3 of the initial parts of the 3D model 141-2 of the second frame and the 3D model 141-3 of the third frame into a left upper arm portion 150-3a and a left forearm peripheral portion 150-3b, respectively, as the final parts.

[0096] As described above, only the topology of the left hand is changed in the left arm portion 150-3 of the 3D models 141-2 and 141-3. Therefore, the dividing unit 72 determines the tracking period of the left upper arm portion 150-3a of the 3D models 141-2 and 141-3 to be two frame periods, the second and third frames. The dividing unit 72 then determines the tracking period of the left forearm peripheral portion 150-3b of the 3D models 141-2 and 141-3 to be one frame period of the second frame and one frame period of the third frame.

[0097] As described with reference to FIG. 8, the division unit 72 determines the final region corresponding to the left arm 150-3 of the 3D model 141-4 of the fourth frame to be the entire left arm 150-3, which is the same as the initial region.

[0098] As a result, as shown in FIG. 9 , there are five tracking blocks corresponding to the left arm 150-3. Specifically, the first tracking block is a block whose tracking period is the first frame and whose body part is the left arm 150-3. The second tracking block is a block whose tracking period is the second and third frames and whose body part is the left upper arm 150-3a. The third and fourth tracking blocks are blocks whose tracking periods are the second and third frames, respectively, and whose body part is the left forearm periphery 150-3b. The fifth tracking block is a tracking block whose tracking period is the fourth frame and whose body part is the left arm 150-3.

[0099] On the other hand, no topology change or large movement occurs in the face 150-1, torso 150-2, right arm 150-4, left leg 150-5, and right leg 150-6 of the 3D models 141-1 to 141-4, except for the left arm 150-3. Therefore, in the example of Figure 9, the division unit 72 determines the tracking period corresponding to the face 150-1, torso 150-2, right arm 150-4, left leg 150-5, and right leg 150-6 to be the period from the first frame to the fourth frame, i.e., the entire period of the captured frames. As a result, as shown in Figure 9, the number of tracking blocks corresponding to the face 150-1, torso 150-2, right arm 150-4, left leg 150-5, and right leg 150-6 is one each.

[0100] 8 and 9, cases where the presence or absence of topology change within the initial part is described. However, when the magnitude of movement within the initial part is different, the initial part is similarly divided to determine the final part. In this case, when the magnitude of movement within the initial part is similar, the final part is integrated into that initial part.

[0101] <First Example of Key Frame Selection> FIG. 10 is a diagram showing a first example of key frame selection by the key frame selection unit 73 of FIG.

[0102] In the example of Fig. 10, the tracking blocks of Fig. 6 are set. In the example of Fig. 10, the key frame selection unit 73 selects, for each tracking block, the first frame in the tracking period corresponding to that tracking block as the key frame. Note that in Fig. 10, the hatched cells represent key frames in each tracking block.

[0103] Although not shown in the figures, the key frame selection unit 73 can also select, for each tracking block, the center frame within the tracking period corresponding to that tracking block as the key frame. In this case, the key frame selection unit 73 can minimize the accumulated error of fitting in mesh tracking of each tracking block.

[0104] <Second Example of Key Frame Selection> FIG. 11 is a diagram showing a second example of key frame selection by the key frame selection unit 73 of FIG.

[0105] In the example of Fig. 11, the tracking blocks of Fig. 6 are set. In the example of Fig. 11, the key frame selection unit 73 selects key frames based on, for example, fitting error information so as to minimize the accumulated fitting error in mesh tracking of each tracking block. Specifically, the key frame selection unit 73 uses the fitting error indicated by the fitting error information as an approximation of the fitting error in mesh tracking, and selects key frames that minimize the accumulation of the fitting error.

[0106] 11, the hatched cells represent the key frames in each tracking block. The key frame selection method can be, for example, the method described in Non-Patent Document 1.

[0107] <Example of UV Map> FIG. 12 is a diagram showing an example of a UV map generated by the UV development unit 54 in FIG.

[0108] In the example of Fig. 12, the tracking blocks of Fig. 9 are set. In this case, the UV development section 54 divides the entire area of ​​each of the UV maps 170-1 to 170-4 of the first and fourth frames into initial part areas based on the block information.

[0109] Specifically, the initial parts are a face 150-1, a torso 150-2, a left arm 150-3, a right arm 150-4, a left leg 150-5, and a right leg 150-6. Therefore, as shown in Figure 12, the entire area of ​​the UV maps 170-1 to 170-4 is divided into initial part areas 171-1 to 171-6 for the face 150-1, the torso 150-2, the left arm 150-3, the right arm 150-4, the left leg 150-5, and the right leg 150-6, respectively.

[0110] Based on the block information, the UV development unit 54 sets the initial part regions 171-1 to 171-6 of the UV map 170-1 of the first frame and the UV map 170-4 of the fourth frame as the final part regions as they are.

[0111] The UV development unit 54 also sets the initial part regions 171-1, 171-2, and 171-4 to 171-6 of the UV map 170-2 of the second frame and the UV map 170-3 of the third frame as the final part regions based on the block information.

[0112] Based on the block information, the UV development unit 54 divides the initial part region 171-3 of the UV maps 170-2 and 170-3 into final part regions 171-3a and 171-3b. Specifically, in the final parts of the second and third frames, the left arm 150-3 is divided into a left upper arm 150-3a and a left forearm peripheral region 150-3b. Therefore, in the UV maps 170-2 and 170-3, the initial part region 171-3 of the left arm 150-3 is further divided into a final part region 171-3a of the left upper arm 150-3a and a final part region 171-3b of the left forearm peripheral region 150-3b.

[0113] The texture images of the assigned final parts are laid out in a planar manner in initial part areas 171-1 to 171-6, or in initial part areas 171-1, 171-2, and 171-4 to 171-6 and final part areas 171-3a and 171-3b. Note that in Figure 12, the shapes of the texture images shown in initial part areas 171-1 to 171-6 and final part areas 171-3a and 171-3b are simplified and differ from the actual shapes.

[0114] The tracking blocks corresponding to the left forearm peripheral area 150-3b in the second frame and the left forearm peripheral area 150-3b in the third frame are different. That is, the key frames for the left forearm peripheral area 150-3b in the second frame and the left forearm peripheral area 150-3b in the third frame are different. Therefore, as shown in FIG. 12 , the layout and shape of the texture images in the final part area 171-3b of the UV map 170-2 and the final part area 171-3b of the UV map 170-3 are different.

[0115] As described above, in the UV maps 170-2 and 170-3, the final part regions 171-3a and 171-3b are generated by dividing the initial part region 171-3. Therefore, the difference in image between the UV maps 170-1 and 170-4, in which the initial part region 171-3 is the final part region, and the UV maps 170-2 and 170-3 is reduced. As a result, the compression efficiency of the UV maps 170-1 to 170-4 can be improved.

[0116] Although not shown in the drawings, when initial regions are integrated to form a final region, the initial region regions are integrated to generate a final region region.

[0117] Because the UV map is rectangular, it is desirable that the shape of the area allocated to each final part be rectangular, but it does not have to be rectangular. The UV development unit 54 may determine the size of the initial part area (final part area) based on the tracking period or visual importance corresponding to the initial part (final part) allocated to the initial part area (final part area). In this case, for example, the size of the initial part area (final part area) is determined so that it becomes larger the longer the tracking period or the higher the visual importance.

[0118] <Example of UV Map in Which Regions Are Not Divided for Each Part> FIG. 13 is a diagram showing an example of a UV map in which regions are not divided for each part.

[0119] Fig. 13A shows UV maps of multiple frames of a 3D model where regions are not divided by body part, and mesh tracking is not performed. Fig. 13B shows UV maps of multiple frames of a 3D model where mesh tracking is performed, and regions are not divided by body part. In Fig. 13, the horizontal direction is the time direction.

[0120] In a 3D model without mesh tracking, the mesh connection information of the 3D model differs between frames, so the arrangement and shape of the texture images of each part in the UV map differs from frame to frame, as shown in Figure 13A.

[0121] On the other hand, in a 3D model where mesh tracking is performed, the mesh connection information of the 3D model remains the same throughout the tracking period, so the arrangement and shape of the texture images of each part in the UV map differ for each tracking period, as shown in Figure 13B.

[0122] As shown in Figures 13A and 13B, if the UV map area is not divided for each part, the layout and shape of the texture image for each part in the UV map will vary across the entire area of ​​the UV map, resulting in poor UV map compression efficiency.

[0123] <Description of Transmission Process> Fig. 14 is a flowchart illustrating the 3D data generation process by the 3D data generation device 12 of Fig. 1. This 3D data generation process is performed, for example, when captured images from N viewpoints are input from N image capture devices 11.

[0124] 14 , the image acquisition unit 21 acquires captured images from N viewpoints supplied from N image capture devices 11. The image acquisition unit 21 performs calibration based on the captured images from the N viewpoints and acquires camera parameters. The image acquisition unit 21 generates depth maps for each of the multiple viewpoints based on the captured images from the N viewpoints. The image acquisition unit 21 supplies the captured images from the N viewpoints, the camera parameters of each image capture device 11, and the depth maps for each of the multiple viewpoints to the 3D model generation unit 22.

[0125] In step S12, the 3D model generation unit 22 performs a 3D data generation process to generate 3D data based on the captured images from N viewpoints, camera parameters, and depth maps for each of the multiple viewpoints supplied from the image acquisition unit 21. Details of this 3D data generation process will be described later with reference to FIG. 15. The 3D model generation unit 22 supplies the 3D data to the encoding unit 24.

[0126] In step S13, the encoding unit 24 converts the format of the 3D data supplied from the 3D model generation unit 22 to generate a transmission stream. The encoding unit 24 supplies the transmission stream to the transmission unit 25.

[0127] In step S14, the transmission unit 25 transmits the transmission stream generated in the process of step S13 to the display device 13. Then, the 3D data generation process ends.

[0128] <Description of 3D Data Generation Processing> FIG. 15 is a flowchart illustrating the 3D data generation processing in step S12 of FIG.

[0129] 15 , the 3D shape restoration unit 51 generates a 3D model of the subject in the captured image of each frame based on the captured images from N viewpoints and camera parameters supplied from the image acquisition unit 21. At this time, the 3D shape restoration unit 51 deforms the generated 3D model as necessary using the depth map supplied from the image acquisition unit 21. The 3D shape restoration unit 51 supplies the 3D model of each frame to the initial part division unit 52.

[0130] In step S22, the initial part division unit 52 determines an initial key frame, and supplies the 3D model of the initial key frame from among the 3D models generated by the processing in step S21 to the touch panel 23 for display.

[0131] In step S23, initial part division unit 52 acquires area information corresponding to a user's touch operation on the displayed 3D model from touch panel 23. In step S24, initial part division unit 52 divides (segments) the initial template into initial parts based on the 3D model of the initial key frame and the area information acquired by the processing of step S23. Initial part division unit 52 supplies the initial template of each initial part and the 3D model of each frame to tracking unit 53.

[0132] In step S25, analysis unit 71 estimates motion information of each initial part for each frame and detects topology changes in the 3D model based on the initial template of each initial part and the 3D model of each frame supplied from initial part division unit 52. Then, analysis unit 71 supplies the motion information and topology change information of each initial part for each frame to division unit 72, and supplies fitting error information to key frame selection unit 73.

[0133] In step S26, the dividing unit 72 determines a tracking block based on at least one of the motion information and topology change information of each initial portion of each frame supplied from the analyzing unit 71. The dividing unit 72 supplies the block information of this tracking block to the key frame selecting unit 73.

[0134] In step S27, the key frame selection unit 73 selects, for each tracking block indicated by the block information, one frame within the tracking period corresponding to that tracking block as a key frame. At this time, the key frame selection unit 73 refers to the fitting error information supplied from the analysis unit 71 as necessary. The key frame selection unit 73 supplies the block information and the key frame information of each tracking block to the block tracking unit 74.

[0135] In step S28, the block tracking unit 74 performs mesh tracking in parallel for each tracking block based on the block information and key frame information, and generates shape data for each final portion of the photographed frame. The block tracking unit 74 supplies the shape data for each final portion of the photographed frame to the UV development unit 54 and the overlap processing unit 55, and supplies the block information to the UV development unit 54.

[0136] In step S29, the UV development unit 54 generates a UV map for each frame based on the block information and shape data of each final portion supplied from the block tracking unit 74, and the captured images from the N viewpoints supplied from the image acquisition unit 21. The UV development unit 54 supplies the UV map for each frame to the overlap processing unit 55.

[0137] In step S30, the overlap processing unit 55 performs non-rigid registration on the shape data of the overlapping portions of the shape data of each final portion supplied from the block tracking unit 74. The overlap processing unit 55 supplies the encoding unit 24 with 3D data in which the UV map generated by the processing of step S29 is associated with the shape data of each final portion for which non-rigid registration has been performed. Then, the processing returns to step S12 in Fig. 14 and proceeds to step S13.

[0138] <Explanation of Display Processing> Fig. 16 is a flowchart illustrating the display processing by the display device 13 of Fig. 1. This display processing is started, for example, when a transmission stream is transmitted by the processing of step S14 of Fig. 14.

[0139] In step S31 of FIG. 16, the receiving unit 31 receives the transport stream transmitted by the process of step S14 of FIG.

[0140] In step S32, the decoding unit 32 decodes the transmission stream received in step S31 to obtain a depth map and a UV map for each frame. The decoding unit 32 generates shape data for each part of the captured frame from the depth map for each frame of the captured frame, and supplies 3D data in which the shape data is associated with the UV map for the captured frame to the rendering unit 33.

[0141] In step S33, the rendering unit 33 performs rendering based on the 3D data supplied from the decoding unit 32 and the virtual viewpoint supplied from the display unit 34, and generates a free viewpoint image. The rendering unit 33 supplies the free viewpoint image to the display unit 34.

[0142] In step S34, the display unit 34 performs display based on the free viewpoint image generated in the process of step S33, and the display process then ends.

[0143] As described above, in the 3D data generator 12, the dividing unit 72 divides the imaging frame into one or more tracking periods for each final portion based on at least one of the motion information and topology change information of that final portion. The block tracking unit 74 performs mesh tracking on the 3D model for each tracking block corresponding to that tracking period and final portion, and generates shape data for each final portion of the imaging frame.

[0144] Therefore, in mesh tracking for each final region, an appropriate tracking period can be set for each final region. As a result, key frames can be updated at an appropriate frequency for each final region. Therefore, unnecessary changes in the UV map image due to key frame updates are not generated, and 3D data transmission efficiency can be improved. In addition, flickering of the free viewpoint image caused by unnecessary updates of key frames for regions with small changes in movement or no topology changes can be prevented, and the visual stability of the free viewpoint image in the time direction can be improved.

[0145] The UV development unit 54 generates a UV map for each frame, in which a texture image of each final part is placed in the final part area of ​​each final part, based on the shape data of each final part and the captured images from N viewpoints. Therefore, the image of the UV map is updated for each tracking block only in the final part area of ​​the final part corresponding to that tracking block. This improves the compression efficiency of the UV map and the transmission efficiency of 3D data.

[0146] The block tracking unit 74 performs mesh tracking in parallel for each tracking block, thereby reducing the processing time for mesh tracking.

[0147] The block tracking unit 74 performs mesh tracking for each tracking block, so that parameters such as the rigidity and smoothness of mesh tracking can be easily adjusted for each final portion, thereby optimizing mesh tracking.

[0148] Specifically, mesh tracking is achieved by, for example, iteratively performing non-rigid registration based on nonlinear optimization. At this time, mesh tracking for each part can be optimized by adjusting parameters for that part. For example, mesh tracking for a part wearing clothing can be optimized by reducing the rigidity of that part. Therefore, the block tracking unit 74 adjusts mesh tracking parameters for each final part to optimize mesh tracking.

[0149] On the other hand, even when mesh tracking is performed on the entire 3D model, it is possible to adjust parameters for each part by adjusting the parameters for each vertex or graph node. However, this has side effects such as slowing down the convergence of the entire 3D model as the energy balance tends to be biased towards the stronger weighting of the parameters. Therefore, it is not easy to adjust parameters for each part.

[0150] 2. Second Embodiment Configuration Example of Image Processing System FIG. 17 is a block diagram showing a configuration example of a second embodiment of an image processing system to which the present technology is applied.

[0151] In the image processing system 210 in Fig. 17, parts corresponding to those in the image processing system 10 in Fig. 1 are assigned the same reference numerals. Therefore, the description of those parts will be omitted as appropriate, and the description will focus on parts that differ from the image processing system 10. The image processing system 210 differs from the image processing system 10 in that the 3D data generation device 12 is replaced by a 3D data generation device 212, but is otherwise configured in the same way as the image processing system 10. The 3D data generation device 212 divides the 3D model into initial parts using a human body parametric model rather than user input.

[0152] Specifically, the 3D data generating device 212 differs from the 3D data generating device 12 in that it has a 3D model generating unit 222 instead of the 3D model generating unit 22, and in that it does not have a touch panel 23, but is otherwise configured in the same way as the 3D data generating device 12.

[0153] Similar to the 3D model generation unit 22, the 3D model generation unit 222 generates a 3D model of the subject in the captured image of each frame based on the captured images from N viewpoints, camera parameters, and depth map supplied from the image acquisition unit 21.

[0154] The 3D model generation unit 222 divides the 3D model of each frame into each initial part using a human body parametric model. As the human body parametric model, for example, the SMPL (Skinned Multi-Person Linear) model described in Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, Michael J. Black, “SMPL: a skinned multi-person linear model,” ACM Transactions on Graphics, Volume 34, Issue 6, 2015, pp. 1-16 can be used.

[0155] The 3D model generation unit 222, like the 3D model generation unit 22, generates shape data for each final portion of the captured frame by performing mesh tracking in units of tracking blocks corresponding to the final portion generated from the initial portion of each frame. Similar to the 3D model generation unit 22, the 3D model generation unit 222 generates a UV map for each frame based on the shape data for each final portion of the captured frame and the captured images from N viewpoints. The 3D model generation unit 222 supplies 3D data in which the shape data for each portion of the captured frame and the UV map are associated with each other to the encoding unit 24.

[0156] <Configuration Example of 3D Model Generator> FIG. 18 is a block diagram showing a configuration example of the 3D model generator 222 in FIG.

[0157] In 3D model generation unit 222 in Fig. 18, parts corresponding to those in 3D model generation unit 22 in Fig. 3 are assigned the same reference numerals. Therefore, explanation of those parts will be omitted as appropriate, and the following explanation will focus on parts that differ from 3D model generation unit 22. 3D model generation unit 222 differs from 3D model generation unit 22 in that it includes an initial part division unit 252 and a tracking unit 253 instead of initial part division unit 52 and tracking unit 53, but is otherwise configured similarly to 3D model generation unit 22.

[0158] The initial part division unit 252 uses a human body parametric model to divide the 3D model of each frame supplied from the 3D shape reconstruction unit 51 into one or more initial parts. The initial part division unit 252 supplies each initial part of the 3D model of each frame to the tracking unit 253.

[0159] The tracking unit 253 determines tracking blocks based on each initial part of each frame supplied from the initial part division unit 252. The tracking unit 253 performs mesh tracking on the 3D model of each frame in tracking block units. The tracking unit 253 supplies the resulting shape data of each final part of the captured frame and block information of each tracking block to the UV development unit 54. The tracking unit 253 also supplies the shape data of each final part of the captured frame to the overlap processing unit 55.

[0160] <Configuration Example of Tracking Unit> FIG. 19 is a block diagram showing a configuration example of the tracking unit 253 in FIG.

[0161] In the tracking unit 253 in Fig. 19, parts corresponding to those in the tracking unit 53 in Fig. 4 are given the same reference numerals. Therefore, the description of those parts will be omitted as appropriate, and the following description will focus on the parts that differ from the tracking unit 53. The tracking unit 253 differs from the tracking unit 53 in that it has an analysis unit 271 instead of the analysis unit 71, but is otherwise configured in the same way as the tracking unit 53.

[0162] The analysis unit 271 (estimation unit) estimates motion information of each initial part for each frame based on the initial part of the 3D model for each frame and the human body parametric model supplied from the initial part division unit 252 in FIG. 18 . Specifically, the analysis unit 271 estimates motion information of each initial part for each frame by fitting each initial part of the 3D model for each frame to the human body parametric model. Note that the analysis unit 271 may also estimate motion information based on the results of motion information estimation performed in the same manner as the analysis unit 71. The estimation of motion information of each initial part may be performed in parallel for each frame. The analysis unit 271 supplies the motion information of each initial part for each frame to the division unit 72 and supplies fitting error information for each frame calculated when estimating this motion information to the key frame selection unit 73.

[0163] Similarly to the analysis unit 71 , the analysis unit 271 also detects a topology change of the 3D model between frames based on each initial portion of the 3D model in each frame, and supplies the topology change information to the division unit 72 .

[0164] <Explanation of 3D Data Generation Processing> FIG. 20 is a flowchart illustrating the 3D data generation processing by the 3D data generation device 212 of FIG.

[0165] The process of step S121 in FIG. 20 is similar to the process of step S21 in FIG. 15, and therefore a description thereof will be omitted.

[0166] In step S122, the initial part division unit 252 uses a human body parametric model to divide the 3D model of each frame generated by the processing of step S121 into one or more initial parts. The initial part division unit 252 supplies each initial part of the 3D model of each frame to the tracking unit 253.

[0167] In step S123, the analysis unit 271 estimates motion information of each initial part for each frame based on each initial part divided by the processing of step S122 and the human body parametric model, and detects a topology change for each frame based on each initial part. The analysis unit 271 supplies the motion information and topology change information of each initial part for each frame to the division unit 72, and supplies fitting error information for each frame calculated when estimating this motion information to the key frame selection unit 73.

[0168] The processing in steps S124 to S128 is similar to the processing in steps S26 to S30 in FIG. 15, and therefore a description thereof will be omitted.

[0169] As described above, in the image processing system 210, the initial part division unit 252 divides the 3D model into one or more initial parts using a human body parametric model. Therefore, the user does not need to perform an operation such as a touch operation to divide the 3D model into initial parts. The initial part division unit 252 can divide the 3D model into initial parts with high accuracy for a subject whose shape is similar to the human body parametric model, such as a subject wearing tight-fitting clothing.

[0170] 3. Computer The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs that make up the software are installed on a computer. Here, the term "computer" includes computers built into dedicated hardware, and general-purpose personal computers, for example, that can execute various functions by installing various programs.

[0171] FIG. 21 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0172] In the computer, a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903 are interconnected by a bus 904.

[0173] An input / output interface 905 is also connected to the bus 904. An input unit 906, an output unit 907, a storage unit 908, a communication unit 909, and a drive 910 are connected to the input / output interface 905.

[0174] The input unit 906 includes a keyboard, mouse, microphone, touch panel, input terminal, etc. The output unit 907 includes a display, speaker, output terminal, etc. The storage unit 908 includes a hard disk, RAM disk, non-volatile memory, etc. The communication unit 909 includes a network interface, etc. The drive 910 drives removable media 911 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0175] In a computer configured as described above, the CPU 901 performs the above-described series of processes by, for example, loading a program stored in the storage unit 908 into the RAM 903 via the input / output interface 905 and the bus 904 and executing the program. The RAM 903 also stores data necessary for the CPU 901 to execute various processes as appropriate.

[0176] The program executed by the computer (CPU 901) can be provided by being recorded on a removable medium 911 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0177] In a computer, the program can be installed in the storage unit 908 via the input / output interface 905 by inserting the removable medium 911 into the drive 910. The program can also be received by the communication unit 909 via a wired or wireless transmission medium and installed in the storage unit 908. Alternatively, the program can be installed in the ROM 902 or the storage unit 908 in advance.

[0178] The program executed by the computer may be a program in which processing is performed in chronological order according to the order described in this specification, or may be a program in which processing is performed in parallel or at a required timing, such as when called. In other words, as long as no contradiction occurs, the processing of each step may be performed in an order different from the order described above. Furthermore, the processing of the steps describing this program may be performed in parallel with the processing of another program, or may be performed in combination with the processing of another program.

[0179] The device that executes the above-described series of processes by a program may be a device other than the computer shown in Fig. 15. In that case, the device has the necessary functional blocks and is configured to be able to obtain the necessary information.

[0180] 4. Application Examples Application to Video Content Production The technology according to the present disclosure can be applied to various products and services.

[0181] For example, new video content may be produced by combining a 3D model of a subject generated in this embodiment with a 3D model managed by another server. Also, by combining a background image captured by a photographing device such as Lidar with a 3D model of a subject generated in this embodiment, video content can be produced in which the subject appears to be located in the location indicated by the background image.

[0182] The 3D model of the subject generated in this embodiment is, for example, a 3D model generated by the 3D model generation unit 22 or a 3D model reconstructed by the rendering unit 33. The video content to be produced may be three-dimensional video content or two-dimensional video content converted into two dimensions.

[0183] <Application to Virtual Space Experiences> For example, in a virtual space where a user communicates as an avatar, a subject generated in this embodiment can be placed as, for example, a performer. In this case, the user can view a live-action subject in the virtual space as an avatar.

[0184] <Application to Communication with Remote Locations> For example, the transmitter 25 transmits a transmission stream of the 3D model of the subject generated by the 3D model generator 22 to the display device 13 at a remote location, allowing a user at the remote location to view the 3D model of the subject through the display device 13. For example, the transmitter 25 transmits a transmission stream of the 3D model of the subject in real time, allowing the subject and a user at a remote location to communicate in real time. Examples of such communication include communication between a subject who is a teacher and a user who is a student, or communication between a subject who is a doctor and a user who is a patient.

[0185] <Others> For example, free viewpoint images of a sports match or the like can be generated based on the 3D models of multiple subjects generated in this embodiment, or individuals can distribute their own 3D models generated in this embodiment on a distribution platform.

[0186] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0187] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present technology.

[0188] For example, multiple technologies related to the present technology can be implemented independently and independently, as long as no contradiction occurs. Of course, any multiple technologies can also be implemented in combination. For example, part or all of the present technology described in any embodiment can be implemented in combination with part or all of the present technology described in another embodiment. Furthermore, part or all of any of the above-mentioned technologies can be implemented in combination with other technologies not described above.

[0189] For example, the present technology may be configured as a cloud computing system in which a single function is shared and processed collaboratively by multiple devices via a network. For example, the rendering unit 33 may be implemented in a server external to the display device 13. In this case, information is exchanged between the display device 13 and the server.

[0190] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0191] Furthermore, when one step includes multiple processes, the processes included in that one step can be executed by one device or can be shared and executed by multiple devices. In other words, the processes included in one step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as one step.

[0192] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0193] The present technology may have the following configurations: (1) An image processing device comprising: a division unit that divides, for each part of a 3D model of a plurality of frames, the plurality of frames into one or more tracking periods based on at least one of motion information indicating the movement of the part between the frames and topology change information indicating a detection area of ​​a change in the topology of the 3D model between the frames; and a tracking unit that performs mesh tracking on the 3D model for each part and for each tracking period to generate shape data indicating a shape of the part. (2) The image processing device described in (1) above further comprises: a key frame selection unit that selects, for each part and for each tracking period, one of the frames within the tracking period as a key frame for the mesh tracking. (3) The image processing device described in (2) above, wherein the key frame selection unit is configured to select the frame at a predetermined position within the tracking period as the key frame. (4) The image processing device described in (2) above, wherein the key frame selection unit is configured to select the key frame based on a fitting error calculated when estimating the motion information for each part. (5) The image processing device according to any one of (1) to (4), further comprising: a 3D shape restoration unit that generates the 3D model of the subject in each of the captured images of the plurality of frames using the multi-viewpoint captured images of the plurality of frames; and an expansion unit that generates, for each of the frames, a texture map in which a texture image of the part is placed in a part area that is an area assigned to each of the parts, based on the shape data for each of the parts and the multi-viewpoint captured images.(6) The image processing device according to (5), wherein the dividing unit is configured to divide or integrate at least some of the parts of the 3D model in at least some of the frames based on at least one of the motion information and the topology change information to generate final parts, and divide the at least some of the frames into one or more tracking periods for each final part, and the unfolding unit divides or integrates the part regions of the at least some of the parts to generate the texture map in which the texture image of the final part is arranged in a final part region that is an area assigned to the final part. (7) The image processing device according to any of (1) to (6), further comprising a part dividing unit that divides the 3D model into one or more parts. (8) The image processing device according to (7), wherein the part dividing unit divides the 3D model into one or more parts based on user input. (9) The image processing device according to (7), wherein the part dividing unit divides the 3D model into one or more parts using a parametric model. (10) The image processing device according to any one of (1) to (9), further comprising an estimation unit that estimates the movement information for each of the body parts based on the 3D model of the plurality of frames. (11) The image processing device according to (10), further comprising: an overlap processing unit that performs non-rigid registration on shape data of an overlapping portion with another body part included in the shape data for each of the body parts generated by the tracking unit. (13) The image processing device according to any one of (1) to (12), further comprising: an overlap processing unit that performs non-rigid registration on shape data of an overlapping portion with another body part included in the shape data for each of the body parts generated by the tracking unit.(14) An image processing method comprising: dividing, for each part of a 3D model of a plurality of frames, the plurality of frames into one or more tracking periods based on at least one of motion information indicating the movement of the part between the frames and topology change information indicating a detection area of ​​a change in the topology of the 3D model between the frames; and tracking: performing mesh tracking on the 3D model for each part and for each tracking period to generate shape data indicating a shape of the part. (15) A program for causing a computer to function as an image processing device comprising: a dividing unit: dividing, for each part of a 3D model of a plurality of frames, the plurality of frames into one or more tracking periods based on at least one of motion information indicating the movement of the part between the frames and topology change information indicating a detection area of ​​a change in the topology of the 3D model between the frames; and a tracking unit: performing mesh tracking on the 3D model for each part and for each tracking period to generate shape data indicating the shape of the part.

[0194] 12 3D data generating device, 51 3D shape restoration unit, 52 initial part division unit, 54 UV development unit, 55 overlap processing unit, 71 analysis unit, 72 division unit, 73 key frame selection unit, 74 block tracking unit, 212 3D data generating device, 252 initial part division unit, 271 analysis unit

Claims

1. An image processing device comprising: a division unit that divides, for each part of a 3D model in multiple frames, the multiple frames into one or more tracking periods based on at least one of motion information indicating the movement of that part between the frames and topology change information indicating a detection area for a change in the topology of the 3D model between the frames; and a tracking unit that performs mesh tracking on the 3D model for each part and for each tracking period, and generates shape data indicating the shape of the part.

2. The image processing device according to claim 1, further comprising a key frame selection unit that selects, for each of the body parts and each of the tracking periods, one of the frames within that tracking period as a key frame for the mesh tracking.

3. The image processing device according to claim 2, wherein the key frame selection unit is configured to select the frame at a predetermined position within the tracking period as the key frame.

4. The image processing device according to claim 2, wherein the key frame selection unit is configured to select the key frame based on a fitting error calculated when estimating the movement information for each of the body parts.

5. The image processing device according to claim 1, further comprising: a 3D shape restoration unit that generates the 3D model of the subject in the captured image of each of the plurality of frames using the multi-viewpoint captured images of the plurality of frames; and an expansion unit that generates, for each frame, a texture map in which a texture image of the part is placed in a part area that is an area assigned to each part, based on the shape data for each part and the multi-viewpoint captured images.

6. The image processing device according to claim 5, wherein the dividing unit divides or integrates at least some of the parts of the 3D models of at least some of the frames of the plurality of frames based on at least one of the motion information and the topology change information to generate final parts, and divides the at least some of the frames into one or more tracking periods for each final part, and the unfolding unit divides or integrates the part areas of the at least some of the parts to generate the texture map in which the texture image of the final part is placed in a final part area that is an area assigned to the final part.

7. The image processing device according to claim 1, further comprising a region dividing unit that divides the 3D model into one or more of the regions.

8. The image processing device according to claim 7, wherein the part dividing unit divides the 3D model into one or more parts based on a user input.

9. The image processing device according to claim 7, wherein the region dividing unit divides the 3D model into one or more regions using a parametric model.

10. The image processing device according to claim 1, further comprising an estimation unit that estimates the movement information for each of the body parts based on the 3D models of the plurality of frames.

11. The image processing device according to claim 10, wherein the estimation unit is configured to estimate the movement information by fitting the 3D model of the plurality of frames to a parametric model for each of the body parts.

12. The image processing device according to claim 1, further comprising an overlap processing unit that performs non-rigid registration on shape data of overlapping portions with other parts included in the shape data for each part generated by the tracking unit.

13. The image processing device according to claim 1, wherein the tracking unit is configured to perform the mesh tracking in parallel for each of the parts and for each of the tracking periods.

14. An image processing method comprising: dividing, for each part of a 3D model in a plurality of frames, the plurality of frames into one or more tracking periods based on at least one of motion information indicating the movement of the part between the frames and topology change information indicating a detection area for a change in the topology of the 3D model between the frames; and tracking, performing mesh tracking on the 3D model for each part and for each tracking period, to generate shape data indicating the shape of the part.

15. A program for causing a computer to function as an image processing device comprising: a division unit that divides, for each part of a 3D model in multiple frames, the multiple frames into one or more tracking periods based on at least one of motion information indicating the movement of that part between the frames and topology change information indicating a detection area for a change in the topology of the 3D model between the frames; and a tracking unit that performs mesh tracking on the 3D model for each part and for each tracking period, and generates shape data indicating the shape of the part.

Images