Image generation system, image processing method, and program

Abstract

To provide a mechanism for generating virtual viewpoint images using data generated by different systems and managed according to different standards. [Solution] The image processing system 160 includes a first system that records a 3D model of a first subject generated using multiple captured images acquired by imaging from multiple imaging devices in association with first time information, and a second system that records second time information measured according to a different standard than the first time information in association with posture information indicating the posture of a second subject generated using multiple captured images acquired by imaging from multiple imaging devices. By converting the first time information, the system identifies the second time information corresponding to the first time information, and generates a virtual viewpoint image based on the 3D model corresponding to the first time information and the posture information corresponding to the identified second time information.

Description

Technical Field

[0001] This disclosure relates to generation processing of virtual viewpoint images.

Background Art

[0002] In recent years, a technology called volumetric capture that can generate a 3D model of a subject from images captured by multiple cameras has attracted attention. In this technology, a 3D model is generated from the captured data of the subject, and an image that cannot be seen by a camera arranged in the real space can be generated as a virtual viewpoint image using a virtual camera (virtual camera) that is operated as an arbitrary viewpoint.

[0003] Regarding volumetric capture, for example, Patent Document 1 discloses a method of capturing different physically separated spaces such as a stadium and a studio as a subject. In this method, by using the same volumetric capture technology to capture different spaces, a part of the 3D model generated from one capture can be replaced with the 3D model generated from the other capture. By combining the 3D models obtained from the same system in this way, one virtual viewpoint image can be generated.

[0004] On the other hand, there is a technology called motion capture as a different imaging method from the above. In this technology, a subject wearing a marker or the like is photographed to obtain information indicating the posture of the subject (for example, the coordinates of each part to which the marker is attached), and the CG model can be moved by animating it to the previously set CG model.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] In recent years, in order to provide even more compelling virtual viewpoint images, there has been a desire to generate a single virtual viewpoint image using different data generated from different systems. For example, a system that generates virtual viewpoint images using volumetric capture generates a 3D model of the subject, and a system that generates virtual viewpoint images using motion capture generates skeletal information of the subject. There is a desire to generate a virtual viewpoint image using this data. However, since each system is designed as a different system, the data management standards for the data generated from each system differ, and there is a risk that a virtual viewpoint image cannot be generated.

[0007] This disclosure aims to provide a mechanism for generating virtual viewpoint images using data generated by different systems and managed according to different standards. [Means for solving the problem]

[0008] To solve the above problems, the image processing system according to this disclosure has the following configuration: a first system that records a 3D model of a first subject generated using multiple captured images acquired by imaging of multiple imaging devices in association with first time information; a second system that records second time information measured according to a different standard than the first time information in association with posture information indicating the posture of a second subject generated using multiple captured images acquired by imaging of multiple imaging devices; identification means that identify the second time information corresponding to the first time information by converting the first time information; and generation means that generate a virtual viewpoint image based on the 3D model corresponding to the first time information and the posture information corresponding to the identified second time information. [Effects of the Invention]

[0009] According to this disclosure, virtual viewpoint images can be generated using data generated by different systems and managed according to different standards. [Brief explanation of the drawing]

[0010] [Figure 1] This is a diagram showing the configuration of the image processing system 160 according to Example 1. [Figure 2] This is a diagram showing the configuration of the image generation device 104 according to Example 1. [Figure 3] This is a diagram illustrating the operation of the virtual camera according to Example 1. [Figure 4] This diagram shows a flowchart of the data storage process generated by each system according to Example 1. [Figure 5] This figure shows an example of the configuration of database 103 according to Example 1. [Figure 6] This figure illustrates the process of generating a virtual viewpoint image in the image processing system 160 according to Example 1. [Figure 7] This figure shows a flowchart of the image generation process according to Example 1. [Figure 8] This figure shows a flowchart of the motion data saving process according to Example 2. [Figure 9] This figure shows an example configuration of database 103 according to Example 2. [Modes for carrying out the invention]

[0011] <Embodiment> According to a preferred embodiment of the present invention, the image processing system includes a first system that records a 3D model of a first subject generated using a plurality of captured images acquired by imaging by a plurality of imaging devices, in association with first time information. The image processing system also includes a second system that records a second time information measured according to a different standard than the first time information, in association with posture information indicating the posture of a second subject generated using a plurality of captured images acquired by imaging by a plurality of imaging devices. The image processing system also includes identification means for identifying the second time information corresponding to the first time information by converting the first time information. The image processing system also includes generation means for generating a virtual viewpoint image based on the 3D model corresponding to the first time information and the posture information corresponding to the identified second time information.

[0012] The second time information, which is measured using a different standard than the first time information, is, for example, information indicating a time counted at a different frame rate than the first time information. Specifically, the second time information is information indicating a time counted at a higher frame rate than the first time information. Alternatively, the second time information may be information indicating a time counted in a different unit than the first time information.

[0013] In this embodiment, the image processing system can generate virtual viewpoint images using data generated by different systems and managed according to different criteria. Furthermore, the image processing system includes interpolation means for interpolating the posture information corresponding to the second time information based on time information before and after the second time information if the second time information corresponding to the first time information is not recorded. The generation means generates a virtual viewpoint image based on the 3D model corresponding to the first time information and the interpolated posture information.

[0014] This configuration makes it possible to generate virtual viewpoint images even when data is managed according to different standards, without the need for correspondence between data generated by different systems.

[0015] Further, the virtual viewpoint image is associated with the 3D model corresponding to the first time information and the posture information corresponding to the specified second time information, and is generated based on a pre-generated 3D model different from the 3D model.

[0016] Also, the posture information is information indicating the positions of each part of the second subject. Each part is, for example, each joint. Alternatively, in the case of motion capture using markers, it is the part to which the marker is attached. Note that the posture information may be any information indicating the posture of the second subject, and for example, may be information indicating the skeleton of the second subject. Note that the posture information is also referred to as a skeleton, an armature, or motion data.

[0017] Note that the first system is a volumetric capture system, and the second system is a motion capture system.

[0018] According to another preferred embodiment of the present embodiment, an image processing method includes a first recording step of associating and recording a 3D model of a first subject generated using a plurality of captured images obtained by capturing with a plurality of imaging devices and first time information. Also. The image processing method includes a second recording step of associating and recording second time information measured based on a criterion different from the first time information and posture information indicating the posture of a second subject generated using a plurality of captured images obtained by capturing with a plurality of imaging devices. Further, the image processing method includes a specifying step of specifying the second time information corresponding to the first time information by converting the first time information. Further, the image processing method includes a generating step of generating a virtual viewpoint image based on the 3D model corresponding to the first time information and the posture information corresponding to the specified second time information.

[0019] According to another preferred embodiment of the present embodiment, a program causes a computer to execute the above-described image processing method.

[0020] <Example> The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0021] <First Example> In this embodiment, volumetric capture and motion capture are used as multiple different imaging systems to capture different spaces. Volumetric capture generates a 3D model of the subject, and motion capture generates information indicating the subject's posture. The method for generating a virtual viewpoint image using this data will then be described.

[0022] The information indicating the subject's posture, generated from motion capture, is referred to as "skeleton" information. It may also be referred to as "armature" or "motion data." In this embodiment, however, it will be referred to as "motion data."

[0023] (Configuration of the virtual viewpoint image generation system) Figure 1 illustrates the configuration of the image processing system 160 according to this embodiment.

[0024] Figure 1(a) shows a diagram of the configuration of the image processing system 160. The image processing system 160 has multiple shooting systems. In this embodiment, the first shooting system is a volumetric capture system 100, the second shooting system is a motion capture system 110, and the system includes a time server 150. In this disclosure, the shooting times of the multiple shooting systems are managed by a common time server 150, but this is not limited to this. Each shooting system may have its own time server.

[0025] (Description of Volumetric Capture System 100) As shown in Figure 1(a), the volumetric capture system 100 has n first sensor systems, from the first sensor system 101a to the first sensor system 101n. Each sensor system in the volumetric capture has at least one imaging device, which is a visible light camera (RGB camera, hereafter simply referred to as a camera). Hereafter, unless otherwise specified, the n first sensor systems will not be distinguished and will be referred to as multiple first sensor systems 101. In this embodiment, the multiple first sensor systems 101 are connected in a daisy-chain configuration, and the information generated by each first sensor system 101 is collected and transmitted to the first sensor recording device 102. However, this is not limited to this configuration, and each first sensor system 101 may transmit information to the first sensor recording device 102.

[0026] Figure 1(b) shows an example of the installation of multiple first sensor systems 101. The multiple first sensor systems 101 are installed so as to surround the first shooting area 120, which is the area to be photographed, and each sensor photographs the first shooting area 120 from a different direction.

[0027] In this embodiment, the first shooting area 120 to be photographed is assumed to be a studio stage where an artist performs live, and n (for example, 100) first sensor systems 101 are installed around the stage. The number of first sensor systems 101 installed is not limited, and the first shooting area 120 to be photographed is not limited to a studio stage. For example, the first shooting area 120 may include a stage set, or the first shooting area 120 may be an arena or an outdoor stadium.

[0028] The subject captured by the volumetric capture system 100 of the first shooting system is referred to as the first subject. In Figure 1(b), the first subject is subject 601 and subject 602 located in the shooting area 120, and is an example of a performance or acting.

[0029] Furthermore, the multiple first sensor systems 101 do not necessarily have to be installed around the entire circumference of the first shooting area 120; they may be installed only around a portion of the first shooting area 120 due to limitations on the installation location, etc. Also, the multiple cameras in the multiple first sensor systems 101 may include shooting devices with different functions, such as telephoto cameras and wide-angle cameras.

[0030] Multiple cameras in the multiple first sensor systems 101 perform synchronized shooting. To perform synchronized shooting, the volumetric capture system 100 is configured to connect to a time server 150 and uses time codes as the shooting time.

[0031] The timecode is information used to uniquely identify the time of capture in the volumetric capture system 100, and is specified in a format such as "day:hour:minute:second.frame number".

[0032] In this embodiment, the shooting rate of the volumetric capture system 100 is set to 59.94 FPS, but it is not limited to this value.

[0033] In this invention, the time code, which is the time of capture of the first imaging system, the volumetric capture system 100, is referred to as the first imaging time.

[0034] Multiple first sensor systems 101 may have microphones (not shown) in addition to cameras. The microphones of each of the multiple first sensor systems 101 synchronize to capture sound. Based on this captured sound, the image generation device 104 can generate an acoustic signal that is reproduced along with the image display. For the sake of simplicity, the description of sound will be omitted from the following explanation, but it will be assumed that images and sound are processed together in principle.

[0035] The first sensor recording device 102 acquires multiple captured images from multiple first sensor systems 101, associates them with the time codes at the time of capture, and saves them in the database 103.

[0036] (Description of motion capture system 110) As shown in Figure 1(a), the second imaging system, motion capture system 110, has m second sensor systems, from second sensor system 111a to second sensor system 111m. Each sensor system in motion capture has an infrared camera. Note that the cameras included in the sensor system are not necessarily limited to infrared cameras; for example, they may be high-speed cameras. Hereafter, unless otherwise specified, the m second sensor systems will not be distinguished and will be referred to as multiple second sensor systems 111.

[0037] Figure 1(c) shows an example of the installation of multiple second sensor systems 111. The multiple second sensor systems 101 are installed so as to surround the second imaging area 130, which is the area to be photographed, and each of them photographs the second imaging area 130 from a different direction.

[0038] In this embodiment, the second shooting area 130 of the object to be photographed is assumed to be a studio stage where an artist performs live, and m units (for example, 20 units) of the second sensor system 111 are installed around the stage. The second shooting area 130 of the object to be photographed is not limited to a studio stage. For example, the second shooting area 130 may include a stage set, or it may be an arena or an outdoor stadium.

[0039] The subject captured by the motion capture system 110 of the second shooting system is referred to as the second subject. In Figure 1(c), the second subject is the subject 603 located in the shooting area 130, and is an example of a subject performing a musical act with markers attached to each part of its body.

[0040] In this embodiment, the first subject and the second subject are in different shooting areas 120 and 130, but a remote camera (not shown) allows them to see each other's movements and communicate. Here, it is assumed that both are playing the same song and performing the same choreography or other actions.

[0041] The motion capture system 110 uses infrared cameras from multiple second sensor systems to track the movement of markers attached to a second subject and acquire the coordinate values ​​in three-dimensional physical space for each part to which the markers are attached. Markers are attached to various parts of the second subject, such as the head, face, shoulders, chest, right arm, left arm, right hand, left hand, waist, right leg, and left leg, allowing for accurate tracking of the movement of the entire subject. Since these motion capture technologies are publicly known, a detailed explanation is omitted.

[0042] Multiple second sensor systems 111 may have microphones (not shown) in addition to cameras. The microphones of each of the multiple second sensor systems 111 synchronize to capture sound. Based on this captured sound, the image generation device 104 can generate an acoustic signal that is reproduced along with the image display. For the sake of simplicity, the description of sound will be omitted from the following explanation, but it will be assumed that images and sound are processed together in principle.

[0043] The second sensor recording device 112 stores the three-dimensional coordinates acquired by the multiple second sensor systems 111 as motion data, along with the system elapsed time during shooting, in the database 103 of the first shooting system's volumetric capture system 100.

[0044] System elapsed time is information used to uniquely identify the time of capture in the motion capture system 110, and is specified, for example, as a time in "seconds" with microsecond precision. However, the format is not limited to seconds as long as it is a value that displays the system elapsed time.

[0045] The system elapsed time is generated based on time information obtained from the time server 150. Alternatively, the time when the motion capture system 110 was started may be kept and the time elapsed since then may be used as the system elapsed time.

[0046] In this embodiment, the shooting rate of the motion capture system 110 is set to 240 FPS, but it is not limited to this value.

[0047] In this invention, the system elapsed time, which is the time of shooting for the second shooting system, the motion capture system 110, is referred to as the second shooting time. However, the second shooting time is not limited to the system elapsed time, as long as it indicates the time of shooting for the second shooting system.

[0048] In this invention, the first and second shooting times are generated based on time information obtained from the same time server 150. Therefore, the volumetric capture system 100 of the first shooting system and the motion capture system 110 of the second shooting system can perform processing in time synchronization. This synchronization process will be described later in Figures 4 and 5.

[0049] Note that the volumetric capture system 100 differs in that its first capture time, represented by a timecode, is in frames, while the motion capture system 110's second capture time, represented by a system elapsed time, is in seconds.

[0050] The image generation device 104 obtains the shooting data or 3D models generated therefrom from the database 103 and generates a virtual viewpoint image.

[0051] The virtual viewpoint image generated by the image generation device 104 represents the view of the subject from the virtual camera 140 (Figure 1(c)). Since the virtual camera is not subject to physical constraints in terms of installation, the virtual viewpoint image is also called a free-viewpoint image. The virtual viewpoint image may be displayed on the display of the image generation device 104 or output to an external system.

[0052] The virtual camera 140 is operated by the virtual camera operating device 120. The virtual camera 140 is set up in a virtual space associated with the first shooting area 120 and the second shooting area 130, and can view the virtual space from a viewpoint different from any camera of the multiple sensor system 101. The virtual camera 140 and its operation will be described later in Figure 3.

[0053] In this embodiment, the virtual camera operating device 120 is configured to be included in the motion capture system 110 with the highest shooting rate in multiple shooting systems, as shown in Figure 1(a). The virtual camera operating device 120 may also be equipped with a display unit such as a display and display the 3D coordinate information of the motion data acquired from the motion capture system 110. Alternatively, it may display a 3D model generated from the motion data, as described later.

[0054] Furthermore, the image generation device 104 and the virtual camera operation device 120 may be integrated into a single unit. In this case, the virtual camera will be operated in a virtual space where a 3D model generated by volumetric capture and a 3D model whose posture has been altered by motion data generated by motion capture are placed. Alternatively, the virtual camera operation device 120 may be included in the first shooting system.

[0055] Note that the configuration of the image processing system 160 is not limited to the example shown in Figure 1(a). The imaging system is not limited to two systems, but may have more. The imaging method is not limited to volumetric capture or motion capture, but may also be other imaging methods.

[0056] In the example shown in Figure 1(a), the database 103 and the image generation device 104 are described as separate devices, but the database 103 and the image generation device 104 may be configured as an integrated unit.

[0057] The configurations of the volumetric capture system 100 and the motion capture system 110, which are multiple different imaging systems used in this embodiment, have been described above.

[0058] (Functional configuration of the image generation device) Figure 2 is a diagram showing the configuration of the image generation device 104 according to this embodiment.

[0059] Figure 2(a) shows an example of the functional configuration of the image generation device 104. The image generation device 104 generates virtual viewpoint images using each of the 3D models generated from different shooting systems. The image generation device 104 comprises a 3D model generation unit 201, a CG processing unit 202, a virtual camera control unit 203, a 3D model synchronization unit 204, and an image generation unit 205.

[0060] The 3D model generation unit 201 uses multiple captured images from the volumetric capture system 100 of the first shooting system, obtained by specifying the time code from the database 103, to generate a 3D model representing the 3D shape of the subject within the shooting area 120. Foreground images are obtained by extracting the foreground region corresponding to objects such as people and musical instruments from the multiple captured images, and background images are obtained by extracting the background region other than the foreground region. The 3D model generation unit 201 then generates a 3D model of the foreground based on the multiple foreground images.

[0061] In this invention, a 3D model generated from multiple captured images of the volumetric capture system 100 of the first imaging system is referred to as the first 3D model.

[0062] The first 3D model is a 3D shape data consisting of a point cloud, generated by a shape estimation method such as the Visual Hull method. The format of the 3D shape data representing the shape of the object is not limited to this. For example, the 3D model of the object may be a mesh model.

[0063] The 3D model generation unit 201 saves the generated first 3D model, along with the time code (first shooting time), to the database 103. An example of the file structure that associates the first 3D model with the first shooting time and is stored in the database 103 is shown in Figure 5. An example of the file structure that shows the details of the first 3D model is shown in Figure 6.

[0064] The 3D model generation unit 201 may be included in the first sensor recording device 102 instead of the image generation device 104. In that case, the first sensor recording device 102 saves the first 3D model to the database 103, and the image generation device 104 reads the first 3D model from the database 103 for use.

[0065] The CG processing unit 202 retrieves motion data stored by the motion capture system 110 of the second shooting system from the database 103. The CG processing unit 202 then processes the retrieved motion data to associate it with a pre-generated CG model. This process allows the pre-created 3D model to move using the motion data retrieved from the motion capture system 110 of the second shooting system. This process is called rigging and is a common process, so a detailed explanation will be omitted. For convenience, the data retrieved from the motion capture system of the second shooting system will be referred to as motion data. In this embodiment, the pre-generated CG model is assumed to be recorded in the CG processing unit 202, but it is not limited to this and may be recorded in the database 103, for example.

[0066] Furthermore, in this invention, a pre-generated 3D model whose posture has been modified using motion data generated by the motion capture system 110 is referred to as a second 3D model. This 3D model is also referred to as a CG model. In this embodiment, the process of changing the posture of a 3D model using motion data is referred to as the process of generating a second 3D model.

[0067] The virtual camera control unit 203 receives input information for the virtual camera 140 from the virtual camera operating device 120 and updates the virtual camera's position and orientation. It also receives input information for the time code and updates the time code. A touch panel, joystick, and keyboard can be used to operate the virtual camera. The updated information indicating the virtual camera's position and orientation is output to the image generation unit 205 as viewpoint information. In this embodiment, in addition to viewpoint information, the updated time code is also output to the image generation unit 205. Input information may also be obtained through the operation of other input devices. Alternatively, a pre-configured virtual camera path may be used. The operation of the virtual camera will be described later in Figure 3.

[0068] The 3D model synchronization unit 204 synchronizes the first 3D model and the second 3D model generated from the shooting data of each shooting system and places them in a single virtual space. The synchronization process is explained in Figure 4.

[0069] The image generation unit 205 generates a virtual viewpoint image based on the first 3D model and the second 3D model arranged by the 3D model synchronization unit 204, and the viewpoint information of the virtual camera set by the virtual camera control unit 203.

[0070] The functional configuration of the image generation device 104 in this embodiment has been described above.

[0071] (Hardware configuration of the image generation device) Next, the hardware configuration of the image generation device 104 will be described with reference to Figure 2(b). The image generation device 104 includes a CPU (Central Processing Unit) 211, RAM (Random Access Memory) 212, and ROM (Read Only Memory) 213. The image generation device 104 also includes an operation input unit 214, a display unit 215, and an external interface 216.

[0072] The CPU 211 processes using programs and data stored in the RAM 212 and ROM 213. The CPU 211 controls the overall operation of the image generation device 104 and executes the processing necessary to realize each function shown in Figure 2(a).

[0073] Furthermore, the image generation device 104 may have one or more dedicated hardware components different from the CPU 211, and the dedicated hardware may perform at least a portion of the processing performed by the CPU 211. Examples of dedicated hardware include ASICs (Application-Specific Integrated Circuits), FPGAs (Field-Programmable Gate Arrays), and DSPs (Digital Signal Processors).

[0074] ROM213 holds programs and data. RAM212 has a work area for temporarily storing programs and data read from ROM213. RAM212 also provides a work area used by CPU211 when executing various processes.

[0075] The operation input unit 214 is, for example, a touch panel and acquires information from user operations. For example, it accepts operations on a virtual camera or time code. The operation input unit 214 may also be connected to an external controller and accept input information related to operations. The external controller may be, for example, a three-axis controller such as a joystick or a mouse. However, the external controller is not limited to these.

[0076] The display unit 215 is a touch panel or screen, etc., and displays a virtual viewpoint image. In the case of a touch panel, the operation input unit 214 and the display unit 215 are integrated into a single unit.

[0077] The external interface 216 transmits and receives information with the database 103, time server 150, etc., for example via a LAN. For example, virtual viewpoint images may be transmitted to an external screen, etc., via an image output port such as HDMI® or SDI. For example, virtual viewpoint images may be transmitted via Ethernet, etc.

[0078] The hardware configuration of the image generation device 104 in this embodiment has been described above.

[0079] (Virtual camera) The operation of the virtual camera 140 (or virtual viewpoint) will be explained with reference to Figure 3. To explain its operation, the position and orientation of the virtual camera, the viewing frustum, etc., will be explained first.

[0080] The virtual camera 140 and its operation are specified using a single coordinate system. The coordinate system used is a typical three-dimensional Cartesian coordinate system consisting of the X, Y, and Z axes, as shown in Figure 3(a).

[0081] The units used in the coordinate system are, for example, meters.

[0082] Naturally, since the virtual camera and 3D model are used in the same virtual space, this coordinate system is also used for the first 3D model and the second 3D model.

[0083] The coordinate system is set and used for the subject of the photograph. Examples of subjects of the photograph include studios and stadium fields. As shown in Figure 3(b), the subject of the photograph includes the entire studio stage 391, as well as performers 393 and other objects 392 on it. The subject may also include the audience seats around the studio, and is not particularly limited.

[0084] The coordinate system is set with the center of stage 391 as the origin (0,0,0) for the target object. The X-axis is defined as the direction of the longer side of stage 391, the Y-axis as the direction of the shorter side of stage 391, and the Z-axis as the vertical direction relative to the stage. Note that the coordinate system settings are not limited to these.

[0085] Next, we will explain the virtual camera using Figures 3(c) and 3(d). A virtual camera is a viewpoint used to draw a virtual viewpoint image. In the square pyramid shown in Figure 3(c), the vertex represents the position 301 of the virtual camera, and the vector extending from the vertex represents the orientation 302 of the virtual camera. The position of the virtual camera is expressed in 3D space coordinates (x, y, z), and the orientation is expressed as a unit vector with scalar components for each axis.

[0086] The orientation 302 of the virtual camera is assumed to pass through the center point of the front clipping surface 303 and the rear clipping surface 304. The space 305 between the front clipping surface 303 and the rear clipping surface is called the viewing frustum of the virtual camera, and it is the range in which the image generation unit 203 generates the virtual viewpoint image (or the range in which the virtual viewpoint image is projected and displayed; hereafter referred to as the display area of ​​the virtual viewpoint image). The orientation 302 of the virtual camera is represented by a vector and is also called the optical axis vector of the virtual camera.

[0087] Figure 3(d) is used to explain the movement and rotation of the virtual camera. The virtual camera moves and rotates within a space represented by three-dimensional coordinates.

[0088] The movement 306 of the virtual camera is the movement of the virtual camera's position 301 and is expressed in terms of its components (x, y, z) along each axis. The rotation 307 of the virtual camera is expressed in terms of yaw (rotation around the Z axis), pitch (rotation around the X axis), and roll (rotation around the Y axis), as shown in Figure 3(a).

[0089] As shown above, by specifying the X, Y, and Z coordinates (x, y, z) of the virtual camera and the rotation angles (Pitch, Roll, Yaw) of the X, Y, and Z axes, you can freely manipulate the position and direction of the virtual camera's shooting.

[0090] These features allow the virtual camera to freely move and rotate within a three-dimensional virtual space where a 3D model generated from the subject is placed, and to generate a virtual viewpoint image of any region within that virtual space.

[0091] Furthermore, virtual camera movements are not limited to these; any movement that can be achieved by a combination of virtual camera movement and rotation is acceptable.

[0092] The position and orientation of the virtual camera in this embodiment have been described above.

[0093] (3D model saving process, database structure, 3D model examples) Figure 5 shows an example of the configuration of the database 103 according to this embodiment.

[0094] Figure 5(a) shows a table that stores the first 3D model generated by the volumetric capture system 100 of the first imaging system. This table is referred to as the first table 501. The first table 501 records the first imaging time and the first 3D model in association.

[0095] Figure 5(b) shows a table illustrating the configuration of the first 3D model. The first 3D model holds data indicating the three-dimensional shape, data indicating the texture, and data indicating the maximum and minimum coordinates. It may also hold identifiers for the object indicated by the three-dimensional shape.

[0096] The 3D shape is information indicating the 3D coordinates (DataPc_t) of the entire point cloud of the first 3D model. If the first 3D model is a mesh model, it includes information indicating the combination of vertices that make up each face of the mesh model, in addition to the coordinates of the vertices that make up the faces. The texture is a texture image (DataTx_t) obtained from captured images to be applied to the point cloud. The texture image may be multiple captured images. The maximum and minimum coordinates are the maximum and minimum values ​​(DataBb_t) of each axis in the 3D coordinates of the point cloud, and are also called bounding boxes. The first 3D model is generated as a general-purpose format such as a colored point cloud in order to be placed in the same virtual space as the second 3D model.

[0097] Furthermore, the first 3D model and the second 3D model are generated based on the same coordinate system. Specifically, the coordinate system of the real space in the volumetric capture system 100 that generates the first 3D model is aligned with the coordinate system of the real space in the motion capture system 110 that generates the motion data for generating the second 3D model. Note that the configuration of the first 3D model is not limited to these, and any configuration that can be placed in the same virtual space as the second 3D model is acceptable.

[0098] However, the data structure is not limited to these if the 3D model is generated from the volumetric capture system 100. For the sake of simplicity, multiple subjects are treated as a single 3D model, but multiple 3D models may be saved for each subject.

[0099] Figure 5(c) shows a table for storing motion data acquired by the motion capture system 110 of the second shooting system. This table is referred to as the second table 502. The second table 502 records the second shooting time, the motion data, and the first shooting time.

[0100] Figure 6 is a diagram illustrating the process of generating a virtual viewpoint image in the image processing system 160 according to this embodiment. Further details will be described later.

[0101] Figure 4 is a flowchart showing the 3D model saving process according to this embodiment.

[0102] Figure 4(a) is a flowchart of the process for saving motion data generated by the motion capture system 110 to the database 103. The second sensor recording device 112 in the motion capture system 110 saves the motion data acquired from the system to the second table 502 of the database 103.

[0103] In steps S401 to S404, the second sensor recording device 112 repeats the motion data saving process according to the shooting interval (frame rate) of the motion capture system 110. When the shooting frame rate is 240 FPS, this process is repeated at intervals of approximately 4.16 milliseconds.

[0104] In step S402, the second sensor recording device 112 updates the system elapsed time, which is the second shooting time. For example, the system elapsed time is counted up according to the shooting rate. The system elapsed time may be updated as needed by connecting to the time server 150 via NTP (Network Time Protocol) or the like. Alternatively, the startup time of the motion capture system 110 may be stored (not shown) in the second table of the database 103, and the elapsed time from that value may be used as the system elapsed time.

[0105] In step S403, the second sensor recording device 112 saves the motion data acquired from the motion capture system 110, along with the current system elapsed time at the second shooting time, to the second table 502 of the database 103. As an example, in the fifth row of the second table 502 in Figure 5(c), the motion data "Data2A226830" is saved with a system elapsed time of "5450.903236".

[0106] Next, Figure 5(d) shows the configuration of the motion data of the motion capture system 110. As shown in the figure, the motion data is stored as the coordinates of each body part, with the 3D coordinates of each marker attached to the second subject being stored. In this embodiment, the head, face, shoulders, chest, right arm, left arm, right hand, left hand, waist, right foot, and left foot are used as examples of body parts to which markers are attached, and the coordinates of each of these (Data2AC1_t to Data2AC11_t in the figure) become the motion data.

[0107] The motion capture system 110 may store CG data in the database 103 (not shown). The CG data may be stored independently of the system's elapsed time and applied to motion data for all system elapsed times. Furthermore, different CG data may be used for each system elapsed time.

[0108] In step S404, the second sensor recording device 112 returns to step S401 and repeats the above motion data saving process according to the shooting interval of the motion capture system 110.

[0109] Figures 6(c) and 6(d) illustrate examples of the second subject of the motion capture system 110 of the second shooting system, which has undergone the motion data saving process described in Figure 4(a), and examples of the second 3D model generated from them.

[0110] Figure 6(c) shows the motion capture system 110 of the second shooting system in action. Markers are attached to various parts of the second subject 603 in the second shooting area 130, and the second subject 603 performs an instrument or act. In the example shown in Figure 6(c), the second subject 603 is shown raising its hand. The motion capture system 110 acquires the coordinates of each marker attached to the second subject 603 as motion data. Here, as an example, the system elapsed time is assumed to be "3784284.284284", and the motion data "Data2A226830" is associated with it and saved in the second table 502.

[0111] Figure 6(d) shows the second 3D model 613 generated by the CG processing unit 202 in the image generation device 104 acquiring the motion data from the second table 502 and associating the motion data with a pre-generated CG model. In other words, the second 3D model is a CG model whose posture has been changed by the motion data. Therefore, this second 3D model 613 is a 3D model for the system elapsed time "3784284.284284". As shown in the figure, the motion capture system 110 can accurately acquire the coordinates of the markers attached to the second subject, and the second 3D model generated by adding this as animation reflects the positional relationship in real space.

[0112] For example, if the position of the second subject 603 in Figure 6(c) is near (x,y,z)=(0,0,0), then the position of the generated second 3D model 613 in the three-dimensional virtual space will be (x,y,z)=(0,0,0).

[0113] Furthermore, in the same figure, the second 3D model 613 reflects a scene where the hand is raised. In the motion capture system 110, not all coordinates of the second subject in real space are reflected in the 3D model; only the coordinates of the parts to which markers are attached are reflected in the second 3D model.

[0114] Note that the data stored in database 103 is not limited to motion data. For example, a second 3D model obtained by applying motion data to CG data may be saved to the second table at regular intervals of system time.

[0115] The motion capture system 110 does not necessarily have to be a motion capture system that uses markers. For example, it may be a markerless motion capture system that uses image recognition to acquire the coordinates of each part of the second subject.

[0116] The motion data saving process in this embodiment has been described above.

[0117] Next, using Figure 4(b), we will explain the flowchart of the 3D model saving process in this embodiment.

[0118] This is the process of saving the first 3D model generated from the volumetric capture system 100 of the first imaging system to the first table 501 of the database 103.

[0119] Furthermore, for the motion data from the second shooting system, the first shooting time is added and saved to the second table 502 in addition to the second shooting time.

[0120] These processes are mainly performed by the 3D model synchronization unit 204 in the image generation device 104, in cooperation with other functional blocks.

[0121] In steps S411 to S416, the 3D model synchronization unit 204 repeats the 3D model saving process. In this embodiment, this process is repeated according to the shooting frame rate of the volumetric capture system 100 of the first shooting system. For example, if the shooting frame rate is 59.94 FPS, this process is repeated at intervals of approximately 16.667 milliseconds.

[0122] In step S412, the timecode of the first imaging time of the volumetric capture system 100 of the first imaging system is updated. For example, the frame number in the timecode format "day:hour:minute:second.frame number" will be incremented. Note that the timecode increment may also be performed by a timecode generator or the like included in the volumetric capture system 100. Here, as an example, we will assume that 19:01:02.034 is specified as the timecode and continue the explanation thereafter.

[0123] In step S413, the 3D model synchronization unit 204 generates a first 3D model from multiple captured images of the volumetric capture system 100 of the first imaging system. The 3D model synchronization unit 204 saves the generated first 3D model, along with the time code of the first imaging time specified in step S412, to the first table 501 of the database 103.

[0124] For example, in the sixth row of the first table 501 in Figure 5(a), the time code "19:01:02.034" is associated with and saved as "Data1A226730", the first 3D model generated by the volumetric capture system 100.

[0125] In step S414, the 3D model synchronization unit 204 converts the timecode, which is the first shooting time, updated in step S412, into the system elapsed time, which is the second shooting time.

[0126] This conversion process converts the timecode, which is the first shooting time, into the time information format notified by the time server 150, and then converts it into the system elapsed time for the second shooting time.

[0127] For example, if the timecode for the first shooting time is specified as "19:01:02.034", converting it to the time server 150 format will result in "19:01:02.567234" if the frame rate is 59.94 FPS. This is because dividing the frame number "034" in the above timecode by 59.94 and displaying it with microsecond precision results in "0.567234". The hours:minutes:seconds can be used as they are.

[0128] Next, the conversion from the time format of the above time server to the system elapsed time for the second shooting time is performed using the startup time of the second shooting system.

[0129] The motion capture system 110 of the second shooting system may notify the volumetric capture system 100 of the first shooting system of its own startup time immediately after startup. Alternatively, the second shooting system may store the startup time in the database 103, and the first shooting system may retrieve it.

[0130] For example, the startup time of the second imaging system is set to "17:30:11.663998" (not shown).

[0131] In this case, the system elapsed time for the second shooting time is "5450.902236" seconds, which is obtained by subtracting the startup time of the second shooting system from the time format of the converted time server, "19:01:02.567234".

[0132] In this way, even with a different shooting system, such as the volumetric capture system 100, the system elapsed time at the second shooting time can be obtained by knowing the startup time of the motion capture system 110. In the example above, the time code "19:01:02.034" of the first shooting time was converted to the system elapsed time "5450.902236" of the second shooting time.

[0133] In step S415, the 3D model synchronization unit 204 searches the second table 502 of the database 103 for the system elapsed time after conversion in the previous step. Using the example given in the previous step, it searches the second table 502 for a matching system elapsed time of "5450.902236" for the second shooting time.

[0134] If the search results show that motion data corresponding to the converted system elapsed time exists, the timecode of the first shooting time before conversion in the previous step is added and saved to the record of that system elapsed time. In this example, in the second table 502, the timecode of the first shooting time, "19:01:02.034", is saved to the record of the system elapsed time of the second shooting time, "5450.902236". The processing when the search results show that no motion data corresponding to the converted system elapsed time exists will be described later.

[0135] As described above, by specifying the timecode of the first shooting time, it becomes possible to obtain the motion data necessary for generating the first 3D model and the second 3D model. In other words, by saving the shooting time of one shooting system to the database used by the other shooting system, a synchronization process is performed to synchronize the use of each 3D model generated from multiple different shooting systems.

[0136] In this example, if the timecode for the first shooting time, "19:01:02.034", is specified, the first 3D model, "Data1A226730", is retrieved from the first table 501. Additionally, the second 3D model, "Data2A226830", is retrieved from the second table 502.

[0137] In this embodiment, in multiple shooting systems, the configuration is such that the first shooting time of the volumetric capture system 100 with a lower shooting rate is stored in the second table of the database used by the motion capture system 110 with a higher shooting rate. This is to ensure that both 3D models are used without interruption by specifying the shooting time of the shooting system with the lower shooting rate.

[0138] Alternatively, in the opposite configuration, the system elapsed time for the second shooting time may be stored in the first table, and both 3D models may be obtained by specifying the system elapsed time. The above conversion process should be performed to match the data generated by the shooting system with a low frame rate. The operator may decide which shooting system to match, or it may be determined by identifying the shooting system with the low frame rate.

[0139] In step S416, the 3D model synchronization unit 204 returns to step S411 and continues the loop processing.

[0140] The first subject example and the first 3D model example generated therefrom in the volumetric capture system 100, which has performed the above 3D model saving process, will be explained using Figures 6(a) and 6(b).

[0141] Figure 6(a) shows an example of a first subject captured by the volumetric capture system 100 of the first shooting system. Similar to Figure 1(b), this is an example of a scene in which two first subjects 601 and 602 are in the first shooting area 120, performing music or acting, and raising their hands. In this example, the time code is "19:01:02.034". As shown in the figure, unlike the second subject, the first subject does not require any special equipment such as markers.

[0142] Figure 6(b) shows the first 3D model example 611 and the first 3D model 612 generated by the 3D model generation unit 201 of the image generation device 104 using the method shown in Figure 2. The first 3D model "Data1A226730" is saved to the first table 501 of the database 103 at the time code "19:01:20.034".

[0143] As shown in Figure 6(b), the volumetric capture system 100 can generate a 3D model similar to the subject. For example, in Figure 6(a), the standing positions of the first subject 601 and 602 are (x,y,z)=(-2,0,0) and (2,0,0), respectively, about 2m away from the center coordinates. The virtual space coordinates of the standing positions of the first 3D models 611 and 612 generated by the volumetric capture system 100 are (x,y,z)=(-2,0,0) and (2,0,0).

[0144] In the figure, the first 3D model 611 and the first 3D model 612 also reflect the action of raising one's hands. Regardless of standing position, when using volumetric capture technology, the shape and positional relationships in real space can be directly reflected in the entire point cloud of the 3D model. The generated 3D model may be a point cloud or a mesh model. In this embodiment, the 3D model will be described as a point cloud.

[0145] The 3D model saving process in this embodiment has been described above.

[0146] (Flowchart for image generation process) The flowchart of the image generation process in this embodiment will be explained using Figure 7.

[0147] In this embodiment, two different 3D models—a first 3D model generated from a volumetric capture system 100 and a second 3D model generated from a motion capture system 110—are placed in a single virtual space to generate a single virtual viewpoint image. This process is mainly performed by the 3D model synchronization unit 204 in the image generation device 104, in cooperation with other functional blocks.

[0148] In steps S701 to S709, the 3D model synchronization unit 204 repeats the image generation process. In this embodiment, the image generation process is repeated according to the frame rate of the virtual viewpoint image to be generated. For example, if the frame rate of the virtual viewpoint image is 59.94 FPS, the virtual viewpoint image is generated at intervals of approximately 16.667 milliseconds. Note that the interval of one loop may be achieved by setting the update rate (refresh rate) of the image display, such as a touch panel, to 59.94 FPS in the image generation device and synchronizing the process with it. The image generation unit 205 then acquires a time code, which is the shooting time of the first shooting system, in accordance with the count-up of the frame rate. Here, as an example of one frame, it will be explained assuming that the time code "19:01:02.034" is specified.

[0149] In step S702, the 3D model synchronization unit 204 receives the timecode for the first shooting time via the virtual camera control unit 203. However, it is not limited to this, and the timecode for the first shooting time may also be specified by the virtual camera operating device 120 included in the motion capture system 110 of the second shooting system, along with the position and orientation of the virtual camera. Alternatively, the virtual camera operating device 120 may receive the system elapsed time for the second shooting time, and the virtual camera control unit 203 may convert it to the timecode for the first shooting time and use it. Alternatively, the frame count of the timecode may be performed automatically.

[0150] In step S703, the 3D model synchronization unit 204 checks whether a record corresponding to the specified time code, which is the first shooting time, exists in the first table of the database 103. If the determination is true, the process proceeds to step S704. If the determination is false, the process proceeds to step S705.

[0151] In step S704, the 3D model synchronization unit 204 reads the first 3D model included in the record of the specified time code from the first table and places it in the three-dimensional virtual space. Figure 6(b) shows an example of the placement of the first 3D model when the time code "19:01:02.034" is specified. In Figure 6(b), the first 3D model 611 and the first 3D model 612 are placed in the virtual space 600.

[0152] In step S705, the 3D model synchronization unit 204 checks whether a record corresponding to the specified time code, which is the first shooting time, exists in the second table of the database 103. If the determination is true, the process proceeds to step S706. If the determination is false, the process proceeds to step S707.

[0153] In step S706, the 3D model synchronization unit 204 reads the motion data included in the record of the specified time code from the second table. The 3D model synchronization unit 204 then outputs the read motion data to the CG processing unit 201, and the CG processing unit 201 obtains a second 3D model generated by associating the motion data with a pre-generated CG model. In this embodiment, Figure 6(d) shows the second 3D model 613 when the time code "19:01:02.034" is specified.

[0154] Here, the 3D model synchronization unit 204 synchronizes the placement of the first 3D model and the second 3D model into a single virtual space 600. An example of this virtual space is shown in Figure 6(e). In Figure 6(e), the first 3D model 611, the first 3D model 612, and the second 3D model 613, specified by the timecode "19:01:02.034", are synchronized and placed on a single virtual space 600. If the movements of subjects 601, 602, and 603 are synchronized, then the movements of each 3D model will be synchronized.

[0155] The virtual space 600 is a different space from the shooting area 120 and shooting area 130, and may be a virtual stage or the like generated by CG, etc.

[0156] In step S707, the 3D model synchronization unit 204 receives user input via the virtual camera control unit 203, moves and rotates the virtual camera in the three-dimensional virtual space, and determines the position and orientation of the virtual camera.

[0157] In step S708, the 3D model synchronization unit 204 projects the first 3D model and the second 3D model, which are placed in the virtual space, onto a virtual camera to generate a virtual viewpoint image. An example of this virtual viewpoint image is shown in Figure 6(f).

[0158] Figure 6(f) is an example of a virtual viewpoint image projected onto the virtual camera set in step S707, showing the first 3D model 611, the first 3D model 612, and the second 3D model 613 at the timecode "19:01:02.034". Similar to the 3D models described in Figure 6(e), the virtual viewpoint image also shows synchronized movements of each subject in a scene where they are performing music or acting and raising their hands.

[0159] In step S709, the 3D model synchronization unit 204 counts up the timecode and returns to step S701 to continue the frame-by-frame loop processing. For the purpose of this explanation, the timecode "19:01:02.034" is used as an example, but the same processing can be performed using any shooting time stored in the first table 501 and second table 502 of the database. Naturally, it is also possible to process timecodes continuously and generate virtual viewpoint images as a video.

[0160] As described above, in a configuration where subjects in different spaces are captured using different shooting systems such as volumetric capture and motion capture, it is possible to generate synchronized 3D models from each captured data and create a single virtual viewpoint image.

[0161] In this embodiment, we have described a case in which motion data for a second shooting time corresponding to the first shooting time exists by performing a conversion process on the first shooting time of the first 3D model. However, because the frame rates of the multiple first sensor systems 101 and the frame rates of the multiple second sensor systems are different, there is a risk that when the conversion process is performed, there will be no record in the second table 502 corresponding to the second shooting time. In that case, motion data for the current system elapsed time and motion data from before that time may be used, and motion data in between may be obtained by interpolation.

[0162] Specifically, in step S415 in Figure 4(b), interpolation is performed if it is determined that motion data corresponding to the second shooting time obtained by converting the first shooting time does not exist in the second table 502. The motion data is interpolated using motion data from the second shooting times before and after the second shooting time that was determined not to exist, from among the second shooting times that exist in the second table 502. For the interpolation process, linear interpolation is performed on the coordinates of each part included in the motion data of the second shooting times before and after to interpolate the motion data for the second shooting time that was determined not to exist. Note that instead of linear interpolation, Lagrangian interpolation or other methods may also be used.

[0163] Therefore, even when data corresponding to the same time information does not exist, a virtual viewpoint image can be generated using data generated from different systems.

[0164] In the above, interpolation processing was performed in step S415 when it was determined that motion data corresponding to the second shooting time obtained by converting the first shooting time does not exist in the second table 502, but this is not limited to that. For example, in step S705, the above interpolation processing may be performed when it is determined that no motion data exists for the first shooting time.

[0165] <Example 2> This embodiment shows an example where the generation time of the first 3D model and the generation time of the second 3D model are different. Specifically, this is an example where the generation time of the first 3D model in the volumetric capture system 100 is longer than the time required for generating motion data and associating the motion data with the CG model in the motion capture system 110. The time required for generating motion data and associating the motion data with the CG model is, in other words, the generation time of the second 3D model. Without the method shown in this embodiment, the generation of one of the 3D models would not be completed in time, and both 3D models could not be placed synchronously in the virtual space. As a result, there is a risk that an unnatural virtual viewpoint image will be generated in which one of the subjects is absent or multiple 3D models that are not synchronized in time are shown.

[0166] In this embodiment, using the same configuration as Embodiment 1, where multiple shooting systems capture different spaces, one 3D model is generated first, and the virtual camera is operated while checking its display. Specifically, the second 3D model in the motion capture system 110 is generated first, and the virtual camera is operated while displaying its virtual viewpoint image. A method for generating the first 3D model of the volumetric capture system 100 using the viewpoint information of the virtual camera will be described. Subsequently, both 3D models are placed synchronously in the virtual space, and a virtual viewpoint image is generated.

[0167] In this embodiment, the image processing system 160 and image generation device 104 from Embodiment 1 are used as they are, and their configurations will not be described. In this embodiment, as mentioned above, the motion data saving process shown in Figure 4 of Embodiment 1, the second table 502 shown in Figure 5(c), and the 3D model saving process are partially different.

[0168] The motion data saving process involved the second sensor recording device 112 in the motion capture system 110 saving the motion data acquired from the system to the second table 502 of the database 103. In addition, in this embodiment, the viewpoint information of the virtual camera is also saved to the second table.

[0169] Figure 8 is a flowchart showing the motion data saving process according to this embodiment.

[0170] In steps S801 to S807, the second sensor recording device 112 repeats the motion data saving process as a loop process. This loop process is repeated according to the shooting frame rate of the motion capture system 110. For example, if the shooting frame rate is 240 FPS, this loop process is repeated at intervals of approximately 4.16 milliseconds.

[0171] In step S802, the system elapsed time, which is the second shooting time, is updated. For example, the system elapsed time is counted up according to the shooting rate. The system elapsed time may also be updated as needed by connecting to the time server 150 via NTP (Network Time Protocol) or the like.

[0172] In step S803, the second sensor recording device 112 saves the motion data acquired from the motion capture system 110, along with the current system elapsed time at the second shooting time, to the second table 502 of the database 103. For example, in the first row of the second table 502, the motion data "Data2A226830" is saved to the record with the system elapsed time "3784284.284284". In this example, the appearance of the second subject is the same as that shown in Figure 6(c)603.

[0173] In step S804, the second sensor recording device 112 generates a second 3D model using the motion data from the previous step. An example of the second 3D model here is the same as that shown in Figure 6(d) 613. In addition, the position and orientation specified in step S805 of the previous loop processing are projected onto the virtual camera, and a virtual viewpoint image is displayed showing only the second 3D model. An example of this virtual viewpoint image is shown in Figure 6(g). As shown in the figure, the virtual viewpoint image here shows only the second 3D model 613 projected. Note that if the current loop processing is the first loop processing, the position and orientation of the virtual camera are at their initial values.

[0174] In step S805, the second sensor recording device 112 accepts operation of the virtual camera. The operator of the virtual camera operates the virtual camera while checking the virtual viewpoint image displayed in the previous step. As shown in Figure 6(g), at this point the virtual viewpoint image is projected only onto the second 3D model, but the operator is generally aware of the size of the stage in the virtual space 600, and operation is possible even with this virtual viewpoint image. Similarly, if the subject is an artist performing or acting, unlike sports, the general movements and their range are known in advance, so operation is possible even with this virtual viewpoint image.

[0175] In step S806, the second sensor recording device 112 saves the position and orientation of the virtual camera specified in step S805 as viewpoint information to the second table 502 of the database 113.

[0176] In step S404, the second sensor recording device 112 acquires viewpoint information of the virtual camera specified by the virtual camera operating device 120 and saves it to the second table of the database 113. Figure 9 shows the second table of the database generated in this embodiment. In the second table 901, for example, in the record of the 5th row in the figure of the second table 502, the viewpoint information "Cam2A226830" is saved with the system elapsed time "5450.903236".

[0177] In step S807, the second sensor recording device 112 returns to step S401 and repeats the motion data saving process described above.

[0178] The motion data saving process in this embodiment has been described above.

[0179] Next, the 3D model saving process in this embodiment will be described. As shown in Embodiment 1, this process mainly involved generating and saving the first 3D model in the volumetric capture system 100 of the first imaging system.

[0180] The 3D model saving process in this embodiment differs from the 3D model saving process in Embodiment 1 (Figure 4(b)) only in step S413. This difference will be explained below.

[0181] In Example 1, in step S413, a first 3D model was generated from multiple images captured by the volumetric capture system 100 using the shape estimation method described in Figure 2.

[0182] In this embodiment, the difference is that the viewpoint information stored in the second table 502, as shown in step S806, is used for coloring the point cloud generated by shape estimation.

[0183] Specifically, based on the position and orientation of the virtual camera 140 specified in the viewpoint information, at least one camera close to it is selected from the multiple first sensor systems 101, and the captured images are used to colorize the point cloud. By performing this process, the view from the virtual camera 140 becomes closer to the actual captured image, thereby improving image quality.

[0184] On the other hand, the process of generating the first 3D model in step S413 takes longer than that of Example 1.

[0185] Therefore, in this embodiment, in addition to a part of the 3D model saving process described above, a part of the image generation process (Figure 7) in Embodiment 1 is modified and used.

[0186] Next, the image generation process in this embodiment will be described. This process mainly involved placing a first 3D model and a second 3D model in a virtual space based on a specified first shooting time, and generating a virtual viewpoint image.

[0187] The image generation process in this embodiment differs from the image generation process in Embodiment 1 (Figure 7) in step S702, and this difference will be explained.

[0188] In Embodiment 1, in step S702, the 3D model synchronization unit 204 receives a time code specification for the first shooting time via the virtual camera control unit 203.

[0189] In this embodiment, the virtual camera control unit 203 uses the timecode of the received first shooting time with a delay equal to the generation processing time of the first 3D model. If no delay is applied, the first 3D model will be in the generation process at the timecode specified by the virtual camera control unit 203, resulting in a problem where it cannot be retrieved from the database 103. It is assumed that the time required for the generation processing of the first 3D model is known in advance.

[0190] For example, if the generation process of the first 3D model (S413) requires 10 seconds, a delay time of 11 seconds with a margin is added and set as the time code in S702. The processing from step S703 onwards can be continued in the same manner as in Example 1.

[0191] By adding the above delay time, the first table 501 of database 103 already contains the first 3D model that has been generated, so in steps S703 and S704 of Example 1, the first 3D model can be placed in the virtual space 600.

[0192] Furthermore, in steps S705 and S706 of Example 1, the second 3D model, with the added delay time, can be placed in the virtual space 600.

[0193] As shown above, the first 3D model and the second 3D model are 3D models corresponding to timecodes with the same delay time added, and therefore can be placed in the virtual space in synchronization, similar to Example 1. The same applies to the virtual viewpoint images generated from these 3D models.

[0194] The image generation process in this embodiment has been described above.

[0195] By using this embodiment, even when the processing times for 3D models generated across different systems differ, it is possible to generate each 3D model synchronously from each captured data point and create a single virtual viewpoint image.

[0196] <Other examples> The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0197] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention.

[0198] Furthermore, the disclosure of this embodiment includes the following configuration, method, and program.

[0199] (Composition 1) A first system that records a 3D model of a first subject generated using multiple captured images acquired by imaging from multiple imaging devices, and first time information in association with each other. A second system that records in association a second time information measured according to a different standard than the first time information, and posture information indicating the posture of a second subject, generated using multiple captured images acquired by imaging from multiple imaging devices, A means for identifying the second time information corresponding to the first time information by converting the first time information, A generation means for generating a virtual viewpoint image based on the 3D model corresponding to the first time information and the pose information corresponding to the identified second time information, An image processing system characterized by having the following features.

[0200] (Configuration 2) The image processing system according to configuration 1, characterized in that the second time information is information indicating a time that is counted at a different frame rate than the first time information.

[0201] (Composition 3) The image processing system according to configuration 2, characterized in that the second time information is information indicating a time that is counted at a higher frame rate than the first time information.

[0202] (Composition 4) The image processing system according to any one of configurations 1 to 3, characterized in that the second time information is information indicating a time that is counted in a different unit than the first time information.

[0203] (Composition 5) If the second time information corresponding to the first time information is not recorded, the system has interpolation means for interpolating the attitude information corresponding to the second time information based on time information before and after the second time information. The image processing system according to any one of configurations 1 to 4, characterized in that the generation means generates a virtual viewpoint image based on the 3D model corresponding to the first time information and the interpolated pose information.

[0204] (Composition 6) The image processing system according to any one of configurations 1 to 5, characterized in that the virtual viewpoint image is generated based on a 3D model corresponding to the first time information and a pre-generated 3D model different from the 3D model, which is associated with the pose information corresponding to the identified second time information.

[0205] (Composition 7) The image processing system according to any one of configurations 1 to 6, characterized in that the posture information is information indicating the position of each part of the second subject.

[0206] (Composition 8) The image processing system according to any one of configurations 1 to 7, characterized in that the first system is a volumetric capture system.

[0207] (Composition 9) The image processing system according to any one of configurations 1 to 8, characterized in that the second system is a motion capture system.

[0208] (method) A first recording step involves recording a 3D model of a first subject, generated using multiple captured images acquired by imaging from multiple imaging devices, in association with first time information. A second recording step involves recording, in association with a second time information measured according to a different standard than the first time information, and posture information indicating the posture of a second subject, which is generated using multiple captured images acquired by imaging from multiple imaging devices. A process of identifying the second time information corresponding to the first time information by converting the first time information, A generation step of generating a virtual viewpoint image based on the 3D model corresponding to the first time information and the pose information corresponding to the identified second time information, An image processing method characterized by having the following features.

[0209] (program) A program for causing a computer to perform the image processing method described above. [Explanation of Symbols]

[0210] 100 Volumetric Capture Systems 104 Image generation device 110 Motion Capture System 201 3D Model Generation Unit 202 CG Processing Unit 204 3D Model Synchronization Unit 205 Image Generation Unit

Claims

1. A first system that records a 3D model of a first subject generated using multiple captured images acquired by imaging with multiple imaging devices, and associates this model with first time information. A second system that records in association a second time information measured according to a different standard than the first time information, and posture information indicating the posture of a second subject, generated using multiple captured images acquired by imaging from multiple imaging devices, A means for identifying the second time information corresponding to the first time information by converting the first time information, A generation means for generating a virtual viewpoint image based on the 3D model corresponding to the first time information and the pose information corresponding to the identified second time information, An image processing system characterized by having the following features.

2. The image processing system according to claim 1, characterized in that the second time information is information indicating a time that is counted at a different frame rate than the first time information.

3. The image processing system according to claim 2, characterized in that the second time information is information indicating a time that is counted at a higher frame rate than the first time information.

4. The image processing system according to claim 1, characterized in that the second time information is information indicating a time that is counted in a different unit than the first time information.

5. If the second time information corresponding to the first time information is not recorded, the system has interpolation means for interpolating the attitude information corresponding to the second time information based on time information before and after the second time information. The image processing system according to claim 1, characterized in that the generation means generates a virtual viewpoint image based on the 3D model corresponding to the first time information and the interpolated pose information.

6. The image processing system according to claim 1, characterized in that the virtual viewpoint image is generated based on a 3D model corresponding to the first time information and a pre-generated 3D model different from the 3D model, which is associated with the pose information corresponding to the identified second time information.

7. The image processing system according to claim 1, characterized in that the posture information is information indicating the position of each part of the second subject.

8. The image processing system according to claim 1, characterized in that the first system is a volumetric capture system.

9. The image processing system according to claim 1, characterized in that the second system is a motion capture system.

10. A first recording step involves recording a 3D model of a first subject, generated using multiple captured images acquired by imaging from multiple imaging devices, in association with first time information. A second recording step involves recording, in association with a second time information measured according to a different standard than the first time information, and posture information indicating the posture of a second subject, which is generated using multiple captured images acquired by imaging from multiple imaging devices. A process of identifying the second time information corresponding to the first time information by converting the first time information, A generation step of generating a virtual viewpoint image based on the 3D model corresponding to the first time information and the pose information corresponding to the identified second time information, An image processing method characterized by having the following features.

11. A program for causing a computer to perform the image processing method described in claim 10.

Images