Information processing system, information processing method, and information processing program

Abstract

To provide an arrangement for reproducing a 3D model of a person with a simpler arrangement.SOLUTION: An information processing system according to the present invention has a face texture reconstruction unit for reconstructing a face texture from a 2D image of a person captured by a camera, a face shape reconstruction unit for reconstructing a 3D shape of a face from the 2D image of the person, a posing estimation unit for estimating a pose of the person from the 2D image of the person, a shape integration unit for reconstructing the 3D shape of a body corresponding to the pose estimated based on the 3D shape data and for reconstructing the 3D shape of the person by integrating the reconstructed 3D shape of the body and the reconstructed 3D shape of the face, a texture reconstruction unit for reconstructing a texture image of the person by blending the texture image of the reconstructed face into the texture image included in the texture data, and a model generation unit for generating the 3D model of the person based on the 3D shape of the person captured by the camera and the texture image of the person captured by the camera.SELECTED DRAWING: Figure 8

Description

Technical Field

[0001] The present invention relates to an information processing system, an information processing method, and an information processing program for reproducing a 3D model.

Background Art

[0002] There has been proposed a technology for providing remote communication that reconstructs a 3D model representing a person in a more realistic form, transmits the reconstructed 3D model to a remote location, and shares a 3D space using XR (VR / AR / MR) technology.

[0003] For example, Non-Patent Document 1 discloses a system that acquires the 3D shape and texture of a person using a depth sensor, transmits the acquired 3D shape and texture to a remote location, and enables communication in a state where the 3D model of the person is superimposed on the real space using an MR (Mixed Reality) headset. Also, the 3D shape of a person may be acquired using a plurality of cameras.

Prior Art Documents

Non-Patent Documents

[0004]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] The aforementioned prior art requires a depth sensor or multiple cameras, which can complicate the device configuration. Therefore, there is a need for a method that can acquire the 3D shape and texture of a person from a 2D image captured by a single camera, and then transmit and reproduce the 3D model of that person at a remote location.

[0006] One objective of the present invention is to provide a configuration that can reproduce a 3D model of a person with a simpler setup. [Means for solving the problem]

[0007] An information processing system according to one embodiment includes a camera, a storage unit that stores pre-created 3D shape data showing the 3D shape of a body and texture data showing the texture of the body, a face texture reconstruction unit that reconstructs the texture of a face from a 2D image of a person captured by the camera, a face shape reconstruction unit that reconstructs the 3D shape of a face from a 2D image of a person captured by the camera, a pose estimation unit that estimates the pose of a person from a 2D image of a person captured by the camera, a shape integration unit that reconstructs the 3D shape of a body corresponding to the pose estimated based on the 3D shape data, and integrates the reconstructed 3D shape of the body and the reconstructed 3D shape of the face to reconstruct the 3D shape of a person captured by the camera, a texture reconstruction unit that blends the reconstructed face texture image with the texture image included in the texture data to reconstruct the texture image of a person captured by the camera, and a model generation unit that generates a 3D model of a person captured by the camera based on the 3D shape of a person captured by the camera and the texture image of a person captured by the camera.

[0008] The texture data may include a texture image corresponding to the 3D shape of the body to be reconstructed, a texture image corresponding to the 3D shape of the face to be reconstructed, a texture map corresponding to the 3D shape of the body to be reconstructed, and a texture map corresponding to the 3D shape of the face to be reconstructed.

[0009] The information processing system may further include a body shape reconstruction unit that reconstructs the 3D shape of the body from multiple 2D images of a person captured by a camera, a head shape reconstruction unit that reconstructs the 3D shape of the head from multiple 2D images of a person captured by a camera, and a texture integration unit that determines the correspondence between the reconstructed 3D shape of the body and the reconstructed 3D shape of the head, determines the correspondence between the texture map corresponding to the 3D shape of the body and the texture map corresponding to the 3D shape of the head based on the determined correspondence between the 3D shapes, and generates a texture image corresponding to the 3D shape of the head from the texture image corresponding to the 3D shape of the body based on the determined correspondence between the texture maps.

[0010] The shape integration unit may integrate the reconstructed 3D shape of the body and the reconstructed 3D shape of the face based on the texture map included in the texture data.

[0011] The model generation unit may integrate the 3D shape of the person with the texture image of the person based on the texture map included in the texture data.

[0012] The texture reconstruction unit may also superimpose the result of masking the texture image of the person captured by the camera onto the texture image included in the texture data.

[0013] The mask may be configured so that its transparency changes continuously.

[0014] A partial image corresponding to a window set in a 2D image of a person captured by a camera may be input to the face texture reconstruction unit and the face shape reconstruction unit. The information processing system may further include a stabilization unit that temporally smooths the position of the person in the 2D image and sets the window.

[0015] The texture integration unit may generate texture data by integrating a texture image corresponding to the 3D shape of the body with a texture image corresponding to the 3D shape of the head, as well as integrating a texture map corresponding to the 3D shape of the body with a texture map corresponding to the 3D shape of the head.

[0016] An information processing method according to another embodiment includes the steps of: reconstructing a facial texture from a 2D image of a person captured by a camera; reconstructing a 3D shape of the face from a 2D image of a person captured by a camera; estimating the pose of a person from a 2D image of a person captured by a camera; reconstructing a 3D shape of the body corresponding to the estimated pose based on pre-created 3D shape data representing the 3D shape of the body; integrating the reconstructed 3D shape of the body and the reconstructed 3D shape of the face to reconstruct the 3D shape of the person captured by the camera; blending the reconstructed facial texture image with a texture image included in pre-created texture data representing the texture of the body to reconstruct a texture image of the person captured by the camera; and generating a 3D model of the person captured by the camera based on the 3D shape of the person captured by the camera and the texture image of the person captured by the camera.

[0017] In yet another embodiment, an information processing program is provided for causing a computer to perform the above method. [Effects of the Invention]

[0018] According to the present invention, a 3D model of a person can be reproduced with a simpler configuration. [Brief explanation of the drawing]

[0019] [Figure 1] This is a schematic diagram showing an example of the system configuration of an information processing system according to this embodiment. [Figure 2] This is a schematic diagram showing an example of the hardware configuration of an information processing device that constitutes an information processing system according to this embodiment. [Figure 3] This is a flowchart showing the processing procedure in the initial model construction stage of the information processing system according to this embodiment. [Figure 4] This is a flowchart showing the processing procedure in the 3D model reproduction stage of the information processing system according to this embodiment. [Figure 5] This is a schematic diagram showing an example of a functional configuration for realizing the initial model construction stage of the information processing system according to this embodiment. [Figure 6] This is a diagram showing an example of data generated in the initial model construction stage of the information processing system according to this embodiment. [Figure 7] This is a schematic diagram for explaining the texture integration process in the initial model construction stage of the information processing system according to this embodiment. [Figure 8] This is a schematic diagram showing an example of a functional configuration for realizing the 3D model reproduction stage of the information processing system according to this embodiment. [Figure 9] This is a diagram showing an example of data generated in the 3D model reproduction stage of the information processing system according to this embodiment. [Figure 10] This is a schematic diagram for explaining the blending process in the information processing system according to this embodiment. [Figure 11] This is a schematic diagram showing another example of the system configuration of the information processing system according to this embodiment. [Embodiments for Carrying Out the Invention]

[0020] Embodiments of the present invention will be described in detail with reference to the drawings. For the same or corresponding parts in the drawings, the same reference numerals are given and their descriptions will not be repeated.

[0021] In this specification, three-dimension or solid is abbreviated as "3D", and two-dimension or plane is abbreviated as "2D".

[0022] [A. System Configuration] Figure 1 is a schematic diagram showing an example of the system configuration of an information processing system 1 according to this embodiment. Figure 1 shows an example configuration in which, for example, information processing devices 100-1, 100-2 (hereinafter sometimes collectively referred to as "information processing device 100") and information processing device 200 are connected via a network 2. Camera 140-1 is connected to information processing device 100-1, and camera 140-2 is connected to information processing device 100-2.

[0023] The information processing device 100 has previously acquired an initial model of person 10. The information processing device 100 reproduces a 3D model of person 10 by continuously capturing images of person 10 with the camera 140. The reproduced 3D model changes in real time, reflecting the movements and facial expressions of person 10 being captured. The reproduced 3D model of person 10 is also called a 3D avatar, or simply an avatar.

[0024] In the example shown in Figure 1, person 10-1 is within the field of view of camera 140-1, and person 10-2 is within the field of view of camera 140-2. Information processing device 100-1 captures an image of person 10-1 and reproduces a 3D model 20-1 of person 10-1 on the screen of information processing device 200. Similarly, information processing device 100-2 captures an image of person 10-2 and reproduces a 3D model 20-2 of person 10-2 on the screen of information processing device 200. The 3D models 20-1 and 20-2 reproduced on the screen of information processing device 200 can be placed in any 3D space.

[0025] [B. Hardware Configuration Example] Figure 2 is a schematic diagram showing an example of the hardware configuration of an information processing device 100 that constitutes an information processing system 1 according to this embodiment. Typically, the information processing device 100 can be implemented using a general-purpose computer.

[0026] Referring to Figure 2, the information processing device 100 includes, as its main hardware components, a CPU 102, a GPU 104, a main memory 106, a display 108, a network interface (I / F) 110, an input device 112, an optical drive 114, a camera interface (I / F) 118, and storage 120.

[0027] The CPU 102 and / or GPU 104 are processors that perform an information processing method according to this embodiment. Multiple CPUs 102 and GPUs 104 may be arranged, and they may have multiple cores.

[0028] The main memory 106 is a memory area where the processor (CPU 102 and / or GPU 104) temporarily stores (or caches) program code, work data, etc., when executing processing. It consists of volatile memory devices such as DRAM (Dynamic Random Access Memory) and SRAM (Static Random Access Memory).

[0029] The display 108 is a display unit that outputs the user interface related to processing and processing results, and is composed of, for example, an LCD (liquid crystal display) or an organic EL (electroluminescence) display.

[0030] The network interface 110 exchanges data with any information processing device or other device connected to network 2.

[0031] The input device 112 is a device that receives instructions and operations from the user, and is composed of, for example, a keyboard, mouse, touch panel, pen, etc.

[0032] The optical drive 114 reads information stored on an optical disc 116, such as a CD-ROM (compact disc read-only memory) or DVD (digital versatile disc), and outputs it to other components. The optical disc 116 is an example of a non-transitory recording medium, and it circulates with arbitrary programs stored on it in a non-volatile state. When the optical drive 114 reads a program from the optical disc 116 and installs it on a storage device 120 or the like, the computer becomes able to function as an information processing device 100. Therefore, the subject of the present invention may be the program itself installed on the storage device 120 or the like, or a recording medium such as an optical disc 116 that stores a program for realizing functions and processes according to this embodiment.

[0033] Figure 2 shows an optical recording medium such as an optical disk 116 as an example of a non-transient recording medium, but it is not limited to this, and semiconductor recording media such as flash memory, magnetic recording media such as hard disks or storage tapes, and magneto-optical recording media such as MO (magneto-optical disk) may also be used.

[0034] The camera interface 118 acquires the video captured by the camera 140 and also provides the camera 140 with commands related to image capture.

[0035] The storage 120 stores the programs and data necessary for the computer to function as an information processing device 100. For example, it consists of non-volatile storage devices such as hard disks and SSDs (solid state drives).

[0036] More specifically, the storage 120 stores an operating system (OS) (not shown), an initial model construction program 122 that implements the initial model construction process (initial model construction stage), and a 3D model reproduction program 124 that implements the 3D model generation process (3D model reproduction stage). These information processing programs cause an information processing device 100, which is an example of a computer, to execute various processes according to this embodiment.

[0037] Furthermore, initial 3D shape data 162 and initial texture data 168 generated during the initial model construction stage may be stored in the storage 120. In other words, the storage 120 corresponds to a storage unit that stores pre-created 3D shape data 126 representing the 3D shape of the body and initial texture data 168 (texture data) representing the texture of the body.

[0038] Figure 2 shows an example of configuring the information processing device 100 using a single computer, but the system is not limited to this. Multiple computers connected via a computer network may cooperate explicitly or implicitly to realize the information processing method according to this embodiment.

[0039] All or part of the functions realized by the processor (CPU102 and / or GPU104) executing a program may be realized using hard-wired circuits such as integrated circuits. For example, they may be realized using ASICs (application-specific integrated circuits) or FPGAs (field-programmable gate arrays).

[0040] Those skilled in the art will be able to realize the information processing device 100 according to this embodiment by appropriately using technology appropriate to the era in which the present invention is implemented.

[0041] Furthermore, the hardware configuration of the information processing device 200, which constitutes the information processing system 1, is the same as in Figure 2, so a detailed explanation will not be repeated.

[0042] [C. Processing Procedures for 3D Model Reproduction] To reproduce a 3D model, typically, an initial model construction process (initial model construction phase) and a process to generate the 3D model (3D model reproduction phase) are performed.

[0043] In this specification, "texture data" is a general term encompassing both texture images and texture maps.

[0044] (c1: Initial model building stage) Figure 3 is a flowchart showing the processing procedure during the initial model construction phase of the information processing system 1 according to this embodiment. Each process shown in Figure 3 is typically realized by the processor of the information processing device 100 executing a program (the initial model construction program 122 shown in Figure 2).

[0045] Referring to Figure 3, the information processing device 100 acquires a 2D image (one frame) captured by the camera 140 (step S100). The information processing device 100 determines whether or not a predetermined number of 2D images have been acquired (step S102). If a predetermined number of 2D images have not been acquired (NO in step S102), the process from step S100 onward is repeated.

[0046] The information processing device 100 may start imaging with the camera 140 upon explicit instruction from the user, or it may repeat imaging at predetermined intervals.

[0047] Next, the information processing device 100 reconstructs body 3D shape data 160, which shows the 3D shape of the captured body, based on the acquired multiple 2D images (multiple viewpoint images 144) (step S104). Then, the information processing device 100 flattens the region corresponding to the face region in the displacement map included in the body 3D shape data 160 (step S106). Finally, the shape parameters and the displacement map after flattening are output as initial 3D shape data 162 (step S108).

[0048] Furthermore, the information processing device 100 reconstructs body texture data (body texture image 1642 and body texture map 1644) that shows the texture of the body based on the acquired multiple 2D images (multiple viewpoint images 144) (step S110).

[0049] Furthermore, the information processing device 100 reconstructs head 3D shape data 167, which shows the 3D shape of the captured head, based on the acquired multiple 2D images (multiple viewpoint images 144) (step S112).

[0050] The information processing device 100, specifically the texture integration unit 158, integrates the body texture data 164 and the face texture data 166 to reconstruct the initial texture data 168 (initial texture image 1682 and initial texture map 1684) (step S114).

[0051] The execution order of steps S104-S108 and steps S110-S114 does not matter. Alternatively, these processes may be executed in parallel.

[0052] Finally, the information processing device 100 stores the initial 3D shape data 162 and initial texture data 168 of the person as the initial model (step S116).

[0053] (c2: 3D model reproduction stage) Figure 4 is a flowchart showing the processing procedure in the 3D model reproduction stage of the information processing system 1 according to this embodiment. Each process shown in Figure 4 is typically realized by the processor of the information processing device 100 executing a program (the 3D model reproduction program 124 shown in Figure 2).

[0054] Referring to Figure 4, the information processing device 100 acquires the 2D image (1 frame) captured by the camera 140 (step S200).

[0055] The information processing device 100 detects the face region included in the acquired 2D video (one frame) (step S202), and determines the position and size of the current window based on the results of past face region detections (step S204).

[0056] The information processing device 100 reconstructs a face texture image 1666 representing the captured face image based on the 2D image of the portion corresponding to the determined window (step S206). In other words, the information processing device 100 reconstructs the face texture from the 2D image of the person captured by the camera 140.

[0057] Next, the information processing device 100 blends the face texture image 1666 with the initial texture image 1682 (initial face texture image 1686) that was reconstructed in the initial model construction stage to reconstruct the blended face texture image 1824 (step S208). In other words, the information processing device 100 blends the reconstructed face texture image (face texture image 1666) with the texture image (initial face texture image 1686) included in the texture data representing the texture of the body that was created in advance to reconstruct the texture image of the person captured by the camera 140 (blended face texture image 1824).

[0058] Furthermore, the information processing device 100 reconstructs parameters (face expression parameters 184) that represent facial expression, movement, and 3D shape, respectively, based on the 2D image of the portion corresponding to the determined window (step S210). In other words, the information processing device 100 reconstructs the 3D shape of the face from the 2D image of the person captured by the camera 140.

[0059] Furthermore, the information processing device 100 estimates the body pose (posture) for each frame from the 2D image (one frame) (step S212). That is, the information processing device 100 estimates the pose of the person from the 2D image of the person captured by the camera 140. The estimated pose is output as body pose data 186 for each frame.

[0060] The processes in step S210 and step S212 may be executed in parallel or in serial order. The order in which the processes are executed does not matter.

[0061] The information processing device 100 inputs body pose data 186 and facial expression parameters 184 into the initial 3D shape data 162 reconstructed during the initial model construction stage, thereby reconstructing integrated 3D shape data 188 that represents a 3D shape that integrates the 3D shape of the body and the 3D shape of the face (step S214). More specifically, the information processing device 100 reconstructs the 3D shape of the body (integrated 3D shape data 188) corresponding to the pose estimated based on the 3D shape data (initial 3D shape data 162) that represents the 3D shape of the body created in advance. The information processing device 100 also integrates the reconstructed 3D shape of the body and the reconstructed 3D shape of the face to reconstruct the 3D shape of the person captured by the camera 140 (integrated 3D shape data 188).

[0062] The processes in steps S202-S208 and steps S210-S214 may be executed in parallel or sequentially. The order in which the processes are executed does not matter.

[0063] The information processing device 100 integrates the integrated 3D shape data 188 and the blended face texture image 1824 (step S216) and outputs a 3D model viewed from a specified viewpoint (step S218). That is, the information processing device 100 generates a 3D model 190 of the person captured by the camera 140 based on the 3D shape of the person captured by the camera 140 and the texture image of the person captured by the camera 140.

[0064] The processing in steps S200 to S218 is repeated every frame.

[0065] [D. Details of the process during the initial model building phase] In the initial model construction stage of the information processing system 1 according to this embodiment, an initial model for reproducing a 3D model is constructed by capturing images of a person. The constructed initial model reflects information about the person's body and face, respectively.

[0066] Figure 5 is a schematic diagram showing an example of a functional configuration for realizing the initial model construction stage of the information processing system 1 according to this embodiment. Figure 6 is a diagram showing an example of data generated during the initial model construction stage of the information processing system 1 according to this embodiment.

[0067] Each function shown in Figure 5 is typically realized by the processor of the information processing device 100 executing a program (the initial model construction program 122 shown in Figure 2). Referring to Figure 5, the information processing device 100 includes an image acquisition unit 142, a body 3D shape reconstruction unit 150, a 3D shape modification unit 152, a body texture reconstruction unit 154, a face texture reconstruction unit 156, a head 3D shape reconstruction unit 157, and a texture integration unit 158.

[0068] (d1: Video acquisition unit 142) The video acquisition unit 142 acquires 2D images captured by the camera 140. At this time, the video acquisition unit 142 acquires multiple 2D images (multi-viewpoint images 144) of the person who is the target of the 3D model reproduction, captured from multiple viewpoints. The camera 140 may be positioned differently relative to the person to capture from multiple viewpoints, or the camera 140 may be fixed while the person rotates their body to achieve capture from multiple viewpoints. Alternatively, multiple cameras 140 may be prepared, and multiple 2D images may be acquired by capturing the person with each camera 140. Figure 6(A) shows an example of a multi-viewpoint image 144 in which the person is captured from eight viewpoints.

[0069] Furthermore, the multi-viewpoint video 144 used for reconstructing the initial model is preferably 2D video consisting of 5 to 10 frames.

[0070] (d2:Body 3D shape reconstruction part 150) The 3D body shape reconstruction unit 150 reconstructs the 3D shape of the body based on the multiple viewpoint images 144. That is, the 3D body shape reconstruction unit 150 reconstructs the 3D shape of the body from multiple 2D images of the person captured by the camera 140 and outputs 3D body shape data 160 that shows the captured 3D shape of the body. Figure 6(B) shows an example of a visual representation of the reconstructed 3D body shape data 160.

[0071] More specifically, the 3D body shape reconstruction unit 150 reconstructs a model representing the 3D shape of a person's body from a 2D image. For reconstructing such 3D shape data, a publicly known algorithm such as "Tex2Shape" (Alldieck, T.; Pons-Moll, G.; Theobalt, C.; Magnor, M. Tex2Shape: Detailed Full Human Body Geometry From a Single Image. In 2019 IEEE / CVF International Conference on Computer Vision (ICCV); 2019; pp 2293-2303. https: / / doi.org / 10.1109 / ICCV.2019.00238.) can be used.

[0072] "Tex2Shape" outputs shape parameters (principal component features β that indicate the shape) and a displacement map. If "Tex2Shape" outputs a model in SMPL format, it may be further converted to the SMPL-X format, which has four times the resolution of the standard SMPL format.

[0073] The 3D body shape reconstruction unit 150 outputs 3D body shape data 160 as information representing the 3D shape of a person's body. The 3D body shape data 160 typically consists of data in mesh format.

[0074] (d3:3D shape correction section 152) The 3D shape modification unit 152 flattens the face region of the body 3D shape data 160 reconstructed by the body 3D shape reconstruction unit 150. Since a separate model is used to reproduce the person's face during the 3D model reproduction stage, it is preferable not to alter the face region of the reconstructed 3D shape.

[0075] Therefore, the 3D shape correction unit 152 corrects the area in the displacement map corresponding to the estimated face area to a flat area. In other words, the 3D shape correction unit 152 corrects the face area to a flat area without any undulations. This flattening makes the process of reproducing the human head during the 3D model reproduction stage more efficient.

[0076] More specifically, the 3D shape correction unit 152 extracts the person contained in the 2D video used to reconstruct the 3D body shape data 160, and estimates the human body regions (body parts) of the extracted person. For example, regions corresponding to the person's face, hands, feet, etc., are estimated. For such estimation of human body regions, a known algorithm such as "DensePose" (Gueler, RA; Neverova, N.; Kokkinos, I. DensePose: Dense Human Pose Estimation in the Wild. In 2018 IEEE / CVF Conference on Computer Vision and Pattern Recognition; 2018; pp 7297-7306. https: / / doi.org / 10.1109 / CVPR.2018.00762.) can be used.

[0077] The 3D shape correction unit 152 then updates the values ​​of the regions in the displacement map corresponding to the estimated face region to values ​​indicating a flat region.

[0078] Furthermore, since areas such as a person's fingers are easily modeled as variant regions, it is preferable to correct them to flat regions.

[0079] Finally, the 3D shape correction unit 152 outputs initial 3D shape data 162 showing a 3D shape with the face region flattened. Figure 6(C) shows an example of a visual representation of the initial 3D shape data 162.

[0080] (d4: Body Texture Reconstruction Unit 154) The body texture reconstruction unit 154 reconstructs the body texture from multiple 2D images (multiple viewpoint images 144) of a person captured by the camera 140. More specifically, the body texture reconstruction unit 154 reconstructs a body texture image 1642 and a body texture map 1644. The body texture image 1642 and the body texture map 1644 are sometimes collectively referred to as "body texture data 164".

[0081] Figure 6(D) shows an example of a body texture image 1642 and a body texture map 1644 (body texture data 164).

[0082] The body texture reconstruction unit 154 reconstructs the body texture data 164 according to the following process.

[0083] First, the body texture reconstruction unit 154 detects key points of a person from the 2D images contained in the multi-viewpoint video 144. For detecting such key points, a well-known algorithm such as "OpenPose" (Cao, Z.; Hidalgo, G.; Simon, T.; Wei, S.-E.; Sheikh, Y. OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields. IEEE Transactions on Pattern Analysis and Machine Intelligence 2021, 43 (1), 172-186. https: / / doi.org / 10.1109 / TPAMI.2019.2929257.) can be used.

[0084] Next, the body texture reconstruction unit 154 uses the detected keypoints to perform semantic segmentation on the 2D image to estimate the human body regions (body parts). For such semantic segmentation, a known algorithm such as "PGN" (Gong, K.; Liang, X.; Li, Y.; Chen, Y.; Yang, M.; Lin, L. Instance-Level Human Parsing via Part Grouping Network. In Computer Vision - ECCV 2018; Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y., Eds.; Lecture Notes in Computer Science; Springer International Publishing: Cham, 2018; pp 805-822. https: / / doi.org / 10.1007 / 978-3-030-01225-0_47.) can be used.

[0085] Finally, the body texture reconstruction unit 154 uses the estimated human body region to reconstruct texture data (body texture image 1642 and body texture map 1644) from multiple 2D images (multiple viewpoint images 144). For such texture data reconstruction, a known algorithm such as "Semantic Human Texture Stitching" (Alldieck, T.; Magnor, M.; Xu, W.; Theobalt, C.; Pons-Moll, G. Detailed Human Avatars from Monocular Video. In 2018 International Conference on 3D Vision (3DV); 2018; pp 98-109. https: / / doi.org / 10.1109 / 3DV.2018.00022.) can be used.

[0086] "Semantic Human Texture Stitching" can output texture data in either the SMPL or SMPL-X format. As mentioned above, when 3D body shape data 160 conforming to the SMPL-X format is used, texture data also conforming to the SMPL-X format is used.

[0087] Here, the SMPL format / SMPL-X format uses the same format as the texture map (UV mapping) included in the texture data.

[0088] (d5: Face Texture Reconstruction Unit 156) The face texture reconstruction unit 156 reconstructs the face texture from the 2D image of the person captured by the camera 140. In the initial model construction stage, the face texture reconstruction unit 156 reconstructs the face texture based on the 2D image contained in the multi-viewpoint image 144. More specifically, the face texture reconstruction unit 156 reconstructs the face texture image 1662 and the face texture map 1664. The face texture image 1662 and the face texture map 1664 are sometimes collectively referred to as "face texture data 166". As will be described later, the face texture image is reconstructed by the texture integration unit 158, so the face texture image 1662 reconstructed by the face texture reconstruction unit 156 may be discarded.

[0089] The face texture reconstruction unit 156 reconstructs the face texture data 166 according to the following process. That is, it can use a known algorithm such as "DECA" (Feng, Y.; Feng, H.; Black, MJ; Bolkart, T. Learning an Animatable Detailed 3D Face Model from In-the-Wild Images. ACM Trans. Graph. 2021, 40 (4), 88:1-88:13. https: / / doi.org / 10.1145 / 3450626.3459936.).

[0090] "DECA" outputs FLAME model parameters (representing the shape and expression of the face) and texture data conforming to the FLAME format for reproducing a person's face. In this way, face texture data 166 conforming to the FLAME format is output from the 2D image of the person captured by the camera. As will be described later, the texture integration unit 158 ​​integrates the face texture data 166 conforming to the FLAME format with the body texture data 164 by applying it to the body texture data 164.

[0091] Furthermore, the face texture reconstruction unit 156 also reconstructs the face texture data 166 frame by frame during the 3D model reproduction stage.

[0092] (d6: Head 3D shape reconstruction part 157) The head 3D shape reconstruction unit 157 reconstructs the 3D shape of the head from multiple 2D images (multiple viewpoint images 144) of the person captured by the camera 140. In other words, the head 3D shape reconstruction unit 157 reconstructs head 3D shape data 167 that shows the 3D shape of the captured head.

[0093] The head 3D shape reconstruction unit 157 uses the same algorithm as the body 3D shape reconstruction unit 150 to reconstruct a model showing the 3D shape of a person's head from 2D images. The head 3D shape reconstruction unit 157 outputs head 3D shape data 167 as information showing the 3D shape of the head. The head 3D shape data 167 typically consists of data in mesh format.

[0094] (d7: Texture Integration Unit 158) The texture integration unit 158 ​​integrates the body texture data 164 and the face texture data 166 to reconstruct the initial texture data 168 (initial texture image 1682 and initial texture map 1684). The texture integration unit 158 ​​integrates the body texture data 164 and the face texture data 166 based on the correspondence between the body 3D shape data 160 and the head 3D shape data 167.

[0095] As shown in Figure 6(E), the initial texture image 1682 and the initial texture map 1684 each consist of a portion for the head, including the face, and a portion for the body excluding the head.

[0096] More specifically, the initial texture image 1682 consists of an initial face texture image 1686, which is reconstructed by a process described later, and a modified body texture image 1642A, which is obtained by disabling the head portion image 1642H corresponding to the head from the body texture image 1642.

[0097] The initial texture map 1684 consists of the face texture map 1664 and a modified body texture map 1644A, which is a body texture map 1644 with the head sub-map 1644H (corresponding to the head) disabled.

[0098] As shown in Figure 6(E), the initial texture data 168 (texture data) includes a texture image corresponding to the 3D shape of the body to be reconstructed (modified body texture image 1642A), a texture image corresponding to the 3D shape of the face to be reconstructed (initial face texture image 1686), a texture map corresponding to the 3D shape of the body to be reconstructed (modified body texture map 1644A), and a texture map corresponding to the 3D shape of the face to be reconstructed (face texture map 1664).

[0099] Note that Figure 6(E) shows an example of the head portion image 1642H and head portion map 1644H being disabled by being deleted. However, they do not necessarily need to be deleted; they can simply be set not to be used in processing.

[0100] Figure 7 is a schematic diagram illustrating the texture integration process in the initial model construction phase of the information processing system 1 according to this embodiment. The texture integration unit 158 ​​performs the following five processes.

[0101] (1) Alignment of body 3D shape data 160 and head 3D shape data 167 The texture integration unit 158 ​​aligns the two shape data by mapping the body 3D shape data 160 and the head 3D shape data 167 into a common 3D space. Here, since the body 3D shape data 160 and the head 3D shape data 167 represent 3D shapes reconstructed from the same person, they are considered to have substantially the same topology.

[0102] The texture integration unit 158 ​​focuses on common facial features (such as eyes and nose) and maps their respective shape data to a common 3D space so that the focused parts have the same coordinates. To achieve this alignment, a coordinate transformation matrix is ​​used, which includes operations such as translation (transition), rotation, and scaling.

[0103] (2) Determining the correspondence between meshes Next, the texture integration unit 158 ​​determines the correspondence between meshes in the two aligned shape data. Specifically, the texture integration unit 158 ​​determines the correspondence between the mesh contained in the body 3D shape data 160 (for example, a set of triangles defined by three vertices) and the mesh contained in the head 3D shape data 167.

[0104] More specifically, the texture integration unit 158 ​​searches for the closest mesh among the meshes included in the aligned head 3D shape data 167 for each mesh included in the aligned body 3D shape data 160. Finally, the texture integration unit 158 ​​determines the correspondence between the meshes (for example, an array showing the correspondence between the index representing each mesh included in the body 3D shape data 160 and the index representing each mesh included in the head 3D shape data 167).

[0105] In this way, the texture integration unit 158 ​​determines the correspondence between the reconstructed 3D shape of the body (body 3D shape data 160) and the reconstructed 3D shape of the head (head 3D shape data 167).

[0106] (3) Determining the correspondence between texture maps Next, the texture integration unit 158 ​​determines the correspondence between the body texture map 1644 and the face texture map 1664.

[0107] The one-to-one correspondence between the body 3D shape data 160 and the body texture map 1644 is known, and similarly, the one-to-one correspondence between the head 3D shape data 167 and the face texture map 1664 is also known. As a result of the above process, the one-to-one correspondence between the body 3D shape data 160 and the head 3D shape data 167 is determined, and the texture integration unit 158 ​​uses this correspondence of shape data to determine the correspondence between texture maps.

[0108] In this way, the texture integration unit 158 ​​determines the correspondence between the texture map (body texture map 1644) corresponding to the 3D shape of the body (body 3D shape data 160) and the texture map (face texture map 1664) corresponding to the 3D shape of the head (head 3D shape data 167), based on the determined correspondence between the 3D shapes.

[0109] (4) Generation of initial face texture image Next, the texture integration unit 158 ​​generates an initial face texture image 1686 based on the correspondence between the body texture map 1644 and the face texture map 1664.

[0110] More specifically, the texture integration unit 158 ​​determines the coordinates of the body texture map 1644 corresponding to each coordinate of the face texture map 1664, and applies the pixel values ​​of the body texture image 1642 at the determined coordinates of the body texture map 1644 as new pixel values ​​for the face texture image. In other words, based on the correspondence between the body texture map 1644 and the face texture map 1664, the body texture image 1642 is mapped to generate a new face texture image, which is the initial face texture image 1686.

[0111] In this way, the texture integration unit 158 ​​generates a texture image (initial face texture image 1686) corresponding to the 3D shape of the head (head 3D shape data 167) from a texture image (body texture image 1642) corresponding to the 3D shape of the body (body 3D shape data 160) based on the determined correspondence of texture maps.

[0112] (5) Data joining Finally, the texture integration unit 158 ​​reconstructs the initial texture data 168 (initial texture image 1682 and initial texture map 1684).

[0113] More specifically, the texture integration unit 158 ​​disables the head portion image 1642H corresponding to the head from the body texture image 1642, and then combines it with the generated initial face texture image 1686. The initial texture image 1682 corresponds to the modified body texture map 1644A and the initial face texture image 1686 being adjusted to the same scale and placed adjacent to each other.

[0114] Furthermore, the texture integration unit 158 ​​disables the head portion map 1644H corresponding to the head from the body texture map 1644, and then merges it with the face texture map 1664. The initial texture map 1684 corresponds to the modified body texture image 1642A and the face texture map 1664 being adjusted to the same scale and placed adjacent to each other.

[0115] Furthermore, texture data conforming to the SMPL-X format can be reformatted to the FLAME format by applying a predetermined scaling. In other words, there is a one-to-one correspondence between texture maps conforming to the SMPL-X format and texture maps conforming to the FLAME format, so the scaling factor when decompressing a texture image can be uniquely determined based on the correspondence between the formats.

[0116] In this way, the texture integration unit 158 ​​integrates a texture image (body texture image 1642) corresponding to the 3D shape of the body (body 3D shape data 160) and a texture image (initial face texture image 1686) corresponding to the 3D shape of the head (head 3D shape data 167), and also integrates a texture map (modified body texture map 1644A) corresponding to the 3D shape of the body and a texture map (face texture map 1664) corresponding to the 3D shape of the head, thereby generating initial texture data 168.

[0117] The initial texture data 168 (initial texture image 1682 and initial texture map 1684) consists of a part for the head, including the face, and the part of the body excluding the head. By preparing more textures for the head, including the face, it is possible to improve the reproduction of facial expressions and movements (gestures) even when imaging with a single camera.

[0118] The initial model construction process is completed through the above steps.

[0119] [E. Details of the processing during the 3D model reproduction stage] In the 3D model reproduction stage of the information processing system 1 according to this embodiment, a 3D model is reproduced from a 2D image (one frame) of a person captured by a single camera 140. By updating the 3D model frame by frame of the 2D image, the person's movements and changes in facial expressions can be reproduced as a video.

[0120] Figure 8 is a schematic diagram showing an example of a functional configuration for realizing the 3D model reproduction stage of the information processing system 1 according to this embodiment. Figure 9 is a diagram showing an example of data generated in the 3D model reproduction stage of the information processing system 1 according to this embodiment.

[0121] Each function shown in Figure 8 is typically realized by the processor of the information processing device 100 executing a program (the 3D model reproduction program 124 shown in Figure 2). Some processing may be handled by the information processing device 200.

[0122] Referring to Figure 8, the information processing device 100 includes a stabilization unit 170, a face texture reconstruction unit 156, a texture image blending unit 172, a face shape reconstruction unit 174, a pose estimation unit 176, a shape integration unit 178, and a 3D model generation unit 180.

[0123] (e1: Stabilization part 170) The stabilization unit 170 detects the face region included in the 2D image captured by the camera 140 and stabilizes the detected face region in time. The stabilization unit 170 outputs the partial image corresponding to the time-stabilized face region to the face texture reconstruction unit 156 and the face shape reconstruction unit 174. In other words, the partial image corresponding to the window set in the 2D image of the person captured by the camera 140 is input to the face texture reconstruction unit 156 and the face shape reconstruction unit 174.

[0124] The stabilization unit 170 smooths the position and size of the face region 163 (window) extracted from the 2D image 146 over time. Figure 9(A) shows an example of the process for extracting face regions 163A and 163B from the 2D image 146. The ranges of face regions 163A and 163B can be determined by known image recognition processing.

[0125] Let's consider a scenario where a known algorithm such as "DECA," as described above, is used to reproduce a person's face frame by frame. While "DECA" can reproduce a face frame by frame, if the size and position of the face region 163 are determined each frame individually, fluctuations and discontinuities may occur in the reproduced face when viewed between frames.

[0126] Generally, the position of key points on a face (e.g., eyes) detected from a 2D image (one frame) can change from frame to frame, and therefore the position and size of the window determined based on the detected key points can also change from frame to frame.

[0127] Therefore, the stabilization unit 170 stabilizes the reproduced face by temporally smoothing the position and size of the window. In other words, the stabilization unit 170 sets the window by temporally smoothing the position of the person in the 2D image.

[0128] More specifically, the stabilization unit 170 employs a window of a certain size that can cover the entire face of a person, and sets the window at a position based on a specific key point. For example, the window can be set centered on the tip of the nose.

[0129] For example, if a person moves within the window set in the previous frame in the next frame, the stabilization unit 170 sets the position of the window in the next frame based on the average position of specific keypoints detected from the previous n frames. Also, if a person moves closer to or further away from the camera 140 in the next frame, the stabilization unit 170 adjusts the size of the window accordingly based on the moving average of the window size in the previous n frames.

[0130] This processing reduces the degree of discontinuity between frames when making the window follow the movement of a character in 2D video 146.

[0131] If the character moves too quickly and moves outside the window, the window's size and position will be reset and set again. In this case, discontinuities may occur in the reproduced face, so additional processing may be performed to reduce the sense of incongruity.

[0132] By employing the processing described above, the position and size of the sequentially extracted face regions 163 (windows) do not change significantly between frames, thus reducing discontinuities in the reconstructed face shape.

[0133] (e2: Face Texture Reconstruction Section 156) The face texture reconstruction unit 156 reconstructs the face texture based on the image of the face region 163 extracted from the 2D image 146. More specifically, the face texture reconstruction unit 156 reconstructs the face texture image 1666. The face texture reconstruction unit 156 is substantially the same as the face texture reconstruction unit 156 shown in Figure 5, so a detailed explanation will not be repeated. Figure 9(B) shows an example of the reconstructed face texture image 1666.

[0134] The face texture reconstruction unit 156 also reconstructs the face texture map, but this is not necessarily required in the texture image blending unit 172, so it may be discarded.

[0135] (e3: Texture image blending section 172) The texture image blending unit 172 blends the initial texture image 1682, which was reconstructed during the initial model construction stage, with the face texture image 1666, which was reconstructed by the face texture reconstruction unit 156, to reconstruct the blended face texture image 1824. In other words, the texture image blending unit 172 blends the reconstructed face texture image (initial texture image 1682) with the texture image (initial texture image 1682) included in the texture data (initial texture data 168) to reconstruct the texture image (blended face texture image 1824) of the person captured by the camera 140.

[0136] Figure 9(C) shows an example of the reconstructed blended face texture image 1824.

[0137] Figure 10 is a schematic diagram illustrating the blending process in the information processing system 1 according to this embodiment. Referring to Figure 10, the texture image blending unit 172 generates a modified face texture image 1686A by blending the face texture image 1666 with the initial face texture image 1686 of the initial texture image 1682 (initial face texture image 1686 and modified body texture image 1642A) using the mask 1826.

[0138] In other words, a blended face texture image 1824 is generated by blending the initial texture image 1682 with the initial face texture image 1686. At this time, the texture image blending unit 172 superimposes the result of making the mask 1826 transparent from the face texture image 1666 onto the initial face texture image 1686. In this way, the texture image blending unit 172 superimposes the result of making the mask transparent from the texture image of the person captured by the camera 140 (blended face texture image 1824) onto the initial face texture image 1686 included in the initial texture data 168.

[0139] The mask 1826 may also be generated, for example, by assigning the confidence level of each pixel of the face texture data 166 reconstructed by the face texture reconstruction unit 156 as its intensity (transparency).

[0140] Alternatively, a mask 1826 may be generated based on the face texture image 1666. More specifically, pixels in the face texture image 1666 whose pixel values ​​exceed a predetermined threshold are assigned "1" (transparent), and all other pixels are assigned "0" (blocked). Subsequently, a minimize filter is applied using a square window, and then a blur filter (e.g., a Gaussian filter or a box filter) is applied to the edges.

[0141] By using such a mask 1826, it is possible to achieve blending that gradually changes the area around the face texture image 1666 superimposed on the initial face texture image 1686. In other words, the mask 1826 is configured so that its transparency changes continuously.

[0142] This blending technique allows for the real-time reproduction of facial expressions reflecting the current frame's imagery, while hairstyles and other elements can be reliably reproduced using the initial texture image 1682.

[0143] In other words, during the 3D model reproduction stage, information such as facial expressions reconstructed from the video of each frame is reflected in the 3D model in real time. However, for textures in areas of the head other than the face region, which may not be reconstructed from the video of that frame, the information from the initial face texture image 1686 is reflected in the 3D model.

[0144] (e4: Face shape reconstruction section 174) The face shape reconstruction unit 174 reconstructs parameters (face expression parameters 184) that represent facial expression, movement, and 3D shape, based on the image of the face region 163 extracted from the 2D image 146. In other words, the face shape reconstruction unit 174 corresponds to a face shape reconstruction unit that reconstructs the 3D shape of a face from a 2D image of a person captured by the camera 140. The face shape reconstruction unit 174 may employ a known algorithm such as "DECA" as described above.

[0145] Figure 9(D) shows an example of a visual representation of the parameters (facial expression parameters 184) that represent the reconstructed facial expression, movement, and 3D shape.

[0146] (e5: Pose estimation unit 176) The pose estimation unit 176 estimates the body pose (posture) for each frame from the 2D image 146. That is, the pose estimation unit 176 estimates the pose of a person from the 2D image of the person captured by the camera 140. The pose estimation unit 176 outputs body pose data 186 for each frame. Typically, the body pose data 186 includes information such as the angles of each joint. The pose estimation unit 176 can employ a known pose estimation algorithm.

[0147] Figure 9(E) shows an example of the pose estimation process and a visual representation of the estimated body pose data 186.

[0148] (e6: Shape integration part 178) The shape integration unit 178 reconstructs the 3D shape of the body corresponding to the captured 2D image 146 by inputting body pose data 186 and facial expression parameters 184 into the initial 3D shape data 162 that was reconstructed during the initial model construction stage.

[0149] More specifically, the shape integration unit 178 reconstructs the 3D shape of the body corresponding to the pose specified by the body pose data 186 and the facial expression defined by the facial expression parameter 184, based on the initial 3D shape data 162. This reconstructs the integrated 3D shape data 188, which represents a 3D shape that integrates the 3D shape of the body and the 3D shape of the face.

[0150] Furthermore, the shape integration unit 178 may reconstruct the integrated 3D shape data 188 using not only the initial 3D shape data 162, but also 3D shape data obtained by incorporating the head 3D shape data 167 into the initial 3D shape data 162.

[0151] Furthermore, the shape integration unit 178 may determine the correspondence based on the initial texture map 1684 (modified body texture map 1644A and face texture map 1664) and then integrate the 3D shape of the body and the 3D shape of the face.

[0152] In this way, the shape integration unit 178 reconstructs the 3D shape of the body corresponding to the pose estimated based on the initial 3D shape data 162 (3D shape data), and integrates the reconstructed 3D shape of the body with the 3D shape of the face reconstructed based on the facial expression parameters 184 to reconstruct the 3D shape of the person captured by the camera 140 (integrated 3D shape data 188). Furthermore, the shape integration unit 178 can improve the reproduction accuracy by integrating the reconstructed 3D shape of the body and the reconstructed 3D shape of the face based on the texture map (initial texture map 1684) included in the initial texture data 168.

[0153] Figure 9(F) shows an example of a visual representation of the reconstructed integrated 3D shape data 188.

[0154] (e7: 3D model generation unit 180) The 3D model generation unit 180 integrates the 3D shape based on the integrated 3D shape data 188 with the blended face texture image 1824. The 3D model generation unit 180 also outputs a 3D model 190 viewed from a specified viewpoint.

[0155] In this way, the 3D model generation unit 180 generates a 3D model 190 of the person captured by the camera 140 based on the 3D shape of the person captured by the camera 140 (integrated 3D shape data 188) and the texture image of the person captured by the camera 140 (blended face texture image 1824).

[0156] Furthermore, the initial texture map 1684 may be referenced when combining the integrated 3D shape data 188 and the blended face texture image 1824 (texture image mapping). That is, the 3D model generation unit 180 may integrate the 3D shape of the person (integrated 3D shape data 188) and the texture image of the person (blended face texture image 1824) based on the initial texture map 1684 included in the initial texture data 168 (texture data).

[0157] Figure 9(F) shows an example of a visual representation of the 3D model 190 as viewed from multiple viewpoints. Note that the 3D model generation unit 180 does not need to simultaneously display 3D models viewed from multiple viewpoints as shown in Figure 9(F), but outputs a 3D model viewed from a single specified viewpoint.

[0158] [F. Variant] Although the example shows a configuration in which the same information processing device 100 performs both the initial model construction process (initial model construction stage) and the 3D model generation process (3D model reproduction stage), some of the processes may be performed by different information processing devices.

[0159] Furthermore, the initial models (initial 3D shape data 162 and initial texture data 168) may be prepared in advance and used as needed when it becomes necessary to reproduce the 3D models.

[0160] Figure 11 is a schematic diagram showing another example of the system configuration of the information processing system 1 according to this embodiment. Referring to Figure 11, for example, the server device 300 pre-stores initial 3D shape data 162 and initial texture data 168 for each user.

[0161] The server device 300 provides the specified initial 3D shape data 162 and initial texture data 168 in response to requests from the information processing devices 100-3 and 100-4. Each of the information processing devices 100-3 and 100-4 uses the initial 3D shape data 162 and initial texture data 168 provided by the server device 300 to perform the process of generating a 3D model (3D model reproduction stage).

[0162] Furthermore, the initial 3D shape data 162 and initial texture data 168 do not necessarily have to be created based on 2D images of a user using the information processing device 100. As described above, in the 3D model reproduction stage, texture images reconstructed by imaging a person are blended, so even if the initial 3D shape data 162 and initial texture data 168 generated from a different person are used, the 3D model of that person can be reproduced.

[0163] Furthermore, the information processing devices 100-1 and 100-2 shown in Figure 1 may work in conjunction with the information processing device 200 to execute the initial model construction process (initial model construction stage) and the 3D model generation process (3D model reproduction stage). The processing performed by each information processing device can be designed arbitrarily.

[0164] [G. Summary] The information processing system 1 according to this embodiment can generate a 3D model of a person from a single frame of 2D video, rather than from multiple 2D videos captured from multiple cameras, when reproducing a 3D model. By reconstructing the shape and texture of the body and face, respectively, facial expressions and gestures can be reproduced with higher accuracy.

[0165] Furthermore, when reproducing a 3D model, it is possible to generate the 3D model from 2D video captured by a single camera frame. Compared to using multiple 2D videos captured by multiple cameras, this reduces the processing load, allowing the 3D model to be reproduced in real time.

[0166] The embodiments disclosed herein should be considered in all respects to be illustrative and not restrictive. The scope of the present invention is indicated by the claims rather than by the description of the embodiments above, and all modifications within the meaning and scope equivalent to the claims are intended to be included. [Explanation of Symbols]

[0167] 1 Information processing system, 2 Network, 10 Person, 20,190 3D model, 100,200 Information processing device, 102 CPU, 104 GPU, 106 Main memory, 108 Display, 110 Network interface, 112 Input device, 114 Optical drive, 116 Optical disc, 118 Camera interface, 120 Storage, 122 Initial model construction program, 124 Reproduction program, 126,162 3D shape data, 140 Camera, 142 Video acquisition unit, 144 Multiple viewpoint video, 146 2D video, 150 Body 3D shape reconstruction unit, 152 3D shape modification unit, 154 Body texture reconstruction unit, 156 Face texture reconstruction unit, 157 Head 3D shape reconstruction unit, 158 Texture integration unit, 160 Body 3D shape data, 163,163A,163B Face region, 164 Body texture data, 166 Face texture data, 167 Head 3D shape data, 168 Initial texture data, 170 Stabilization unit, 172 Texture image blending unit, 174 Face shape reconstruction unit, 176 Pose estimation unit, 178 Shape integration unit, 180 3D model generation unit, 184 Face expression parameters, 186 Body pose data, 188 Integrated 3D shape data, 300 Server device, 1642 Body texture image, 1642A Modified body texture image, 1642H Head portion image, 1644 Body texture map, 1644A Modified body texture map, 1644H Head portion map, 1662,1666 Face texture image, 1664 Face texture map, 1682 Initial texture image, 1684 Initial texture map, 1686 Initial face texture image, 1686A Modified face texture image, 1824; Blended face texture image, 1826; Mask.

Claims

1. Camera and A storage unit that stores 3D shape data indicating a 3D shape of a body and texture data indicating a texture of the body, the 3D shape data being created in advance; a facial texture reconstruction unit that reconstructs a facial texture image from a 2D image of a person captured by the camera; a face shape reconstruction unit that reconstructs a 3D shape of a face from a 2D image of a person captured by the camera; a pose estimation unit that estimates a pose of a person from a 2D image of the person captured by the camera; a shape integration unit that reconstructs a 3D shape of a body corresponding to the estimated pose based on the 3D shape data, and integrates the reconstructed 3D shape of the body and the reconstructed 3D shape of the face to reconstruct a 3D shape of the person captured by the camera; a texture reconstruction unit that reconstructs a texture image of the person captured by the camera by blending a texture image included in the texture data with the reconstructed texture image of the face; and a model generation unit that generates a 3D model of the person captured by the camera based on a 3D shape of the person captured by the camera and a texture image of the person captured by the camera.

2. The texture data is a texture image corresponding to the reconstructed 3D shape of the body, and a texture image corresponding to the reconstructed 3D shape of the face; a texture map corresponding to the reconstructed 3D shape of the body, and a texture map corresponding to the reconstructed 3D shape of the face, a body shape reconstruction unit that reconstructs a 3D shape of a body from a plurality of 2D images of a person captured by the camera; a body texture reconstruction unit that reconstructs a body texture from a plurality of 2D images of a person captured by the camera; a head shape reconstruction unit that reconstructs a 3D shape of a head from a plurality of 2D images of a person captured by the camera; The information processing system of claim 1, further comprising a texture integration unit that determines a correspondence between the reconstructed 3D shape of the body and the reconstructed 3D shape of the head, determines a correspondence between a texture map corresponding to the 3D shape of the body and a texture map corresponding to the 3D shape of the head based on the determined correspondence of the 3D shapes, and generates a texture image corresponding to the 3D shape of the head from a texture image corresponding to the 3D shape of the body based on the determined correspondence of the texture maps.

3. The information processing system according to claim 2 , wherein the shape integration unit integrates the reconstructed 3D shape of the body and the reconstructed 3D shape of the face based on a texture map included in the texture data.

4. The information processing system according to claim 2 , wherein the model generation unit integrates the 3D shape of the person and a texture image of the person based on a texture map included in the texture data.

5. 5. The information processing system according to claim 1, wherein the texture reconstruction unit superimposes a texture image of the person captured by the camera that is transmitted through a mask onto a texture image included in the texture data.

6. a partial image corresponding to a window set in the 2D image of the person captured by the camera is input to the face texture reconstruction unit and the face shape reconstruction unit; The information processing system according to claim 1 , further comprising a stabilization unit that sets the window by temporally smoothing a position of a person in the 2D video.

7. Reconstructing a facial texture image from a 2D image of a person captured by a camera; Reconstructing a 3D shape of a face from a 2D image of the person captured by the camera; estimating a pose of a person from a 2D image of the person captured by the camera; Reconstructing a 3D shape of the body corresponding to the estimated pose based on 3D shape data indicating a 3D shape of the body that has been created in advance; Reconstructing a 3D shape of a person captured by the camera by integrating the reconstructed 3D shape of the body and the reconstructed 3D shape of the face; A step of blending the reconstructed facial texture image with a texture image included in texture data indicating a body texture that has been created in advance, thereby reconstructing a texture image of the person captured by the camera; generating a 3D model of the person captured by the camera based on a 3D shape of the person captured by the camera and a texture image of the person captured by the camera.

8. An information processing program for causing a computer to execute the method according to claim 7.

Images