Information processing device, information processing method, and program

Abstract

To enable output of information for generating a virtual viewpoint video in which a subject is disposed at a desired position in the virtual viewpoint video.SOLUTION: A virtual camera path data processing device 1 comprises: a virtual camera information acquiring unit 101 which, in correspondence to frames constituting a virtual viewpoint video generated on the basis of a plurality of captured images obtained by capturing a subject with a plurality of capturing devices, acquires a parameter representing a position of a virtual viewpoint and a visual line direction from the virtual viewpoint; a coordinate system adjustment information acquiring unit 102 which specifies a position and an orientation of the subject in the virtual viewpoint video; and a virtual camera path data outputting unit for outputting virtual viewpoint data in which the acquired parameters and information based on the specified position and orientation of the subject are associated with each other.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a technology for generating a virtual viewpoint video. [Background technology]

[0002] In recent years, a technology that generates a video (virtual viewpoint video) viewed from a specified viewpoint (virtual viewpoint) by placing multiple camera devices around a shooting area and using multiple captured images acquired from each camera device has been attracting attention. With the technology that generates virtual viewpoint video, for example, a video creator can create content with a powerful viewpoint from a video of a soccer or basketball game. In addition, a user watching the content can watch the game while freely moving the viewpoint, which gives the user a high sense of realism compared to conventional captured images. Patent Document 1 shows a technology that determines a virtual viewpoint by operating a device or a UI screen. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2017-212592 A Summary of the Invention [Problem to be solved by the invention]

[0004] In generating a virtual viewpoint video, a case may be considered in which a subject captured by a plurality of image capturing devices is placed at an arbitrary position in the virtual viewpoint video. However, when placing a subject at an arbitrary position in the virtual viewpoint video, a problem may occur in that an appropriate virtual viewpoint video is not generated by simply determining a virtual viewpoint. The above problem has not been taken into consideration in the virtual viewpoint data generated by the method described in Patent Document 1.

[0005] The present disclosure has been made in consideration of the above-mentioned problems, and aims to make it possible to output information that will enable the generation of a virtual viewpoint video in which a subject is placed at a desired position within the virtual viewpoint video. [Means for solving the problem]

[0006] An information processing system according to the present disclosure includes an acquisition means for acquiring first information indicating a position of three-dimensional shape data of a subject generated based on a plurality of captured images and second information indicating a position of a virtual viewpoint and a line of sight direction from the virtual viewpoint, a first modification means for changing a position of the three-dimensional shape data of the subject based on the first information, a second modification means for changing a position of the virtual viewpoint based on an amount of change made by the first modification means and the second information, and a modification means for modifying a position of the virtual viewpoint based on the position of the virtual viewpoint and the line of sight direction from the virtual viewpoint after the change made by the second modification means. Photographed image A means for identifying the Photographed image and a generating means for generating a virtual viewpoint image based on the position of the three-dimensional shape data of the subject changed by the first changing means. Effect of the Invention

[0007] According to the present disclosure, it becomes possible to output information for generating a virtual viewpoint video in which a subject is positioned at a desired position within the virtual viewpoint video. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating a configuration of an information processing system. [Diagram 2] FIG. 11 is a diagram for explaining coordinate system adjustment information. [Diagram 3] FIG. 2 is a diagram illustrating an example of a format of sequence data. [Figure 4] FIG. 11 is a diagram illustrating an example of a format of virtual camera path data. [Diagram 5] FIG. 11 is a diagram illustrating an example of a format of virtual camera path data. [Figure 6]FIG. 11 is a diagram illustrating an example of a format of virtual camera path data. [Figure 7] 11 is a flowchart for explaining the operation of the virtual camera path data generation processing device. [Figure 8] FIG. 2 is a diagram for explaining the configuration of a virtual viewpoint video generating device. [Figure 9] 10 is a flowchart for explaining the operation of the virtual viewpoint video generating device. [Figure 10] FIG. 2 is a diagram for explaining an example of a communication state between devices; [Figure 11] FIG. 11 is a diagram illustrating an example of virtual camera path data. [Figure 12] 2 is a diagram for explaining the hardware configuration of a virtual camera path data processing device. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that the components described in the following embodiments are examples of the embodiments, and the present disclosure is not limited to them.

[0010] (Embodiment 1) In this embodiment, an example of processing for generating virtual camera path data for generating a virtual viewpoint video based on shooting data acquired by shooting from different directions with multiple shooting devices will be described. In this embodiment, the virtual viewpoint used when generating the virtual viewpoint video is called a virtual camera. That is, the virtual camera is a camera virtually placed at the position of the virtual viewpoint, and the position of the virtual viewpoint and the line of sight direction from the virtual viewpoint correspond to the position of the virtual camera and the attitude of the virtual camera, respectively.

[0011] In addition, in this embodiment, information representing the movement path and change in attitude of the virtual camera when generating a virtual viewpoint video is called virtual camera path data. In other words, it can be said that the virtual camera path data is virtual viewpoint data including parameters of the position of the virtual viewpoint and the line of sight direction from the virtual viewpoint. The virtual camera path data in this embodiment includes parameters representing the three-dimensional position of the virtual camera and parameters representing the attitude of the virtual camera in the pan, tilt, and roll directions. Note that the contents of the virtual camera path data are not limited to the above. For example, the virtual camera path data may include parameters corresponding to the size of the field of view (angle of view) of the virtual viewpoint.

[0012] In addition, the virtual camera path data in this embodiment has parameters for a plurality of video frames constituting the virtual viewpoint video. That is, the virtual camera path data is configured such that virtual camera parameters are associated with each of a plurality of frames constituting the video of the virtual viewpoint video, and this configuration represents the movement path and changes in posture of the virtual camera in the virtual viewpoint video.

[0013] In addition, the virtual viewpoint image in this embodiment is also called a free viewpoint image, but is not limited to an image corresponding to a viewpoint freely (arbitrarily) designated by the user, and for example, an image corresponding to a viewpoint selected by the user from a plurality of candidates is also included in the virtual viewpoint image. In addition, the designation of the virtual viewpoint may be performed by a user operation, or may be performed automatically based on the result of image analysis, etc. In addition, in this embodiment, the case where the virtual viewpoint image is a moving image will be mainly described. In this embodiment, a moving image is composed of a plurality of images (frames). Therefore, an image in which the virtual camera parameters change for each frame while the subject reflected in the virtual viewpoint image is stationary can also be said to be a moving image. Furthermore, an image in which the virtual camera parameters do not change for each frame while the subject reflected in the virtual viewpoint image is stationary can be treated as a moving image, as it has a plurality of frames, although it appears to be stationary as an image. In addition, this embodiment is also applicable to a virtual viewpoint image composed of one frame, that is, a still image.

[0014] <Hardware configuration> 12 is a block diagram showing an example of the hardware configuration of a computer applicable to a virtual camera path data processing device described later. The camera path data processing device 1 has a CPU 1201, a ROM 1202, a RAM 1203, an operation unit 1204, an output unit 1205, an external storage device 1206, an I / F 1207, and a bus 1208. The hardware configuration described below can also be applied to other devices in an information processing system described later.

[0015] The CPU 1201 controls the entire computer using computer programs and data stored in the RAM 1202 and ROM 1203, and executes each process as a process performed by a device described later. That is, the CPU 1201 functions as each processing unit in a virtual camera path data processing device described later.

[0016] The RAM 1202 has an area for temporarily storing computer programs and data loaded from an external storage device 1206, and data acquired from the outside via an I / F (interface) 1207. Furthermore, the RAM 1202 has a work area used when the CPU 1201 executes various processes. That is, the RAM 1202 can be allocated as a frame memory, for example, or can provide various other areas as appropriate.

[0017] The ROM 1203 stores setting data for the computer, a boot program, etc. The operation unit 1204 is composed of a keyboard, a mouse, etc., and can be operated by a user of the computer to input various instructions to the CPU 1201. The output unit 1205 displays the results of processing by the CPU 1201. The output unit 1205 is composed of, for example, a liquid crystal display.

[0018] The external storage device 1206 is a large-capacity information storage device, such as a hard disk drive. The external storage device 1206 stores an operating system (OS) and computer programs for causing the CPU 1201 to realize the functions of each unit of the virtual camera path data processing device. Furthermore, the external storage device 1206 may store each image data to be processed.

[0019] Computer programs and data stored in the external storage device 1206 are loaded into the RAM 1202 as appropriate under the control of the CPU 1201, and become the subject of processing by the CPU 1201. Networks such as a LAN or the Internet, and other devices such as a projector or display device can be connected to the I / F 1207, and the computer can obtain and transmit various information via this I / F 1207. Reference numeral 1208 denotes a bus that connects the above-mentioned components.

[0020] The hardware configuration is not limited to this. For example, at least one of the operation unit 1204, the output unit 1205, and the external storage device 1206 may be configured to be connected externally as a device different from the virtual camera path data processing device 1. In this case, the CPU 1201 functions as a reception control unit that receives input from the operation device and the external storage device, and an output control unit that outputs data to an output device such as a display device and the external storage device.

[0021] Furthermore, the present invention may be realized in the following form: That is, the computer program code read from the storage medium is written into a memory provided in a function expansion card inserted into a computer or a function expansion unit connected to the computer. Then, based on the instructions of the computer program code, a CPU provided in the function expansion card or function expansion unit performs part or all of the actual processing to realize the above-mentioned functions.

[0022] The code stored on the storage medium is computer program code that corresponds to at least a portion of the processes described above or similar processes.

[0023] <System configuration> 1 is a diagram showing an example of the configuration of an information processing system including a virtual camera path data processing device in this embodiment. The information processing system 10 includes a virtual camera path data processing device 1, a shooting device 2, a shape estimation device 3, a storage device 4, a model operation device 5, a virtual camera operation device 6, and a sequence data processing device 7.

[0024] The virtual camera path data processing device 1 receives an input for determining parameters of the position and attitude of the virtual camera from a virtual camera operation device 6 described later, and generates virtual camera path data. The virtual camera path data processing device 1 will be described in detail later.

[0025] The photographing device 2 is a photographing device that acquires photographed images used to generate a virtual viewpoint video. In this embodiment, a virtual viewpoint video is generated based on a plurality of photographed images obtained using a plurality of photographing devices. Although only the photographing device 2 is illustrated in FIG. 1, the information processing system 10 includes a plurality of photographing devices. In the following description, when there is no particular distinction between the plurality of photographing devices, they will be described as the photographing device 2. The plurality of photographing devices photograph the photographing area from a plurality of different directions. The photographing area is, for example, a stadium where rugby or soccer is played, a hall or stage where a concert is held, a photography studio, etc. The plurality of photographing devices are installed in different positions and directions so as to surround such a photographing area, and photograph in synchronization. The plurality of photographing devices do not need to be installed around the entire circumference of the photographing area, and may be installed only in a part of the photographing area depending on restrictions on the installation location, etc. The number of the plurality of photographing devices is not limited, and for example, when the photographing area is a rugby stadium, about tens to hundreds of photographing devices may be installed around the stadium.

[0026] In addition, multiple image capturing devices with different angles of view, such as a telephoto camera and a wide-angle camera, may be installed. For example, if a telephoto camera is used, an object can be captured with high resolution, and the resolution of the generated virtual viewpoint image is improved. In addition, if a wide-angle camera is used, the range that can be captured by one camera is wide, and the number of cameras can be reduced. The multiple image capturing devices are synchronized with one piece of time information in the real world, and time information indicating the capture time is added to the image of each frame of the captured image.

[0027] The photographing device 2 may be composed of one camera or multiple cameras. Furthermore, the photographing device 2 may include devices other than cameras. For example, it may include a distance measuring device using laser light or the like.

[0028] In addition, the state of the image capture device 2 may be controllable. The state of the image capture device refers to the state of the image capture device, such as its position, attitude, focal length, optical center, and distortion. The position and attitude of the image capture device may be controlled by the image capture device itself, or may be controlled by a camera platform that controls the position and attitude of the image capture device.

[0029] The photographing device 2 has a unique identification number for distinguishing it from other photographing devices. The photographing device 2 may have other functions, such as a function to extract a foreground image from an image acquired by photographing, and may also include hardware (such as a circuit or device) that realizes the function.

[0030] The shape estimation device 3 generates shape data representing the shape of the subject based on image data acquired from the photographing device 2. In this embodiment, the shape estimation device 3 generates three-dimensional shape data representing the three-dimensional shape of the subject. A method for generating three-dimensional shape data in this embodiment will be described below.

[0031] First, a plurality of photographed images are obtained by photographing each photographed region from different directions using a plurality of imaging devices 2. Next, a foreground image in which a foreground region corresponding to an object such as a person or a ball is extracted from the plurality of photographed images, and a background image in which a background region other than the foreground region is extracted are obtained. Here, a foreground image is an image in which an object region (foreground region) is extracted from a photographed image photographed and acquired by a photographing device. An object extracted as a foreground region refers to a dynamic object (moving body) that moves (whose position or shape may change) when photographed from the same direction in a time series. For example, in a competition, an object may include a person such as a player or a referee in the field where the competition is held, and in a ball game, an object may include a ball in addition to a person. In a concert or entertainment, a singer, a player, a performer, or a presenter are examples of objects. Note that if a background is registered in advance using a background image or the like, even a stationary object is extracted as a foreground region if it does not exist in the background registered in advance.

[0032] Moreover, the background image is an image of at least an area (background area) different from the foreground object. Specifically, the background image is an image in a state where the foreground object is removed from the photographed image. Moreover, the background refers to a photographed object that is stationary or continues to be nearly stationary when photographed from the same direction in time series. Such photographed objects are, for example, a stage for a concert or the like, a stadium for an event such as a competition, a structure such as a goal used in a ball game, or a field. However, the background is an area different from at least the foreground object, and the photographed object may include other objects in addition to the object and the background. The process of generating the foreground image and the background image from the photographed image may be performed by the shape estimation device 3 or may be performed by the photographing device 2. When the process is performed by the photographing device 2, the photographing device 2 outputs the foreground image and the background image to the shape estimation device 3.

[0033] The shape estimation device 3 uses the foreground image to generate three-dimensional shape data of the subject in the foreground by a shape estimation method such as a shape-from-silhouette method. The three-dimensional shape data is, for example, point cloud model data, billboard data, mesh model data, etc. In the following description, the three-dimensional shape data of the subject is also referred to as a subject model. The shape estimation device 3 also uses the foreground image to generate texture data for coloring the three-dimensional shape data. The three-dimensional shape data of the background is generated by previously performing three-dimensional measurements of a shooting area such as a stadium or a venue. The shape estimation device 3 generates texture data for coloring the three-dimensional shape data of the background based on the background image. The shape estimation device 3 transmits the generated three-dimensional shape data and texture data to the storage device 4.

[0034] The virtual viewpoint video in this embodiment is generated, for example, by the following method. That is, the virtual viewpoint video is generated by mapping texture data to the three-dimensional shape data of the foreground according to parameters of the position and orientation of the virtual camera and performing rendering. Rendering is also performed similarly for the three-dimensional shape data of the background. Note that the method of generating the virtual viewpoint video is not limited to this, and various methods can be used, such as a method of generating the virtual viewpoint video by projective transformation of a captured image without using three-dimensional shape data.

[0035] In the following description, data used to generate a virtual viewpoint video, such as three-dimensional shape data and texture data, will also be collectively referred to as material data. In addition, in this embodiment, a configuration in which three-dimensional shape data is generated has been described, but this embodiment is also applicable to a case in which image-based rendering is performed as a method for generating a virtual viewpoint video.

[0036] The storage device 4 is a device that stores the material data generated by the shape estimation device 3. For example, it is configured by a semiconductor memory or a magnetic recording device. Each piece of material data stored in the storage device 4 is linked to the corresponding shooting time information. Specifically, the shooting time information associated with the captured image used to generate the material data is linked to the material data, thereby linking the material data to the shooting time information. The shooting time information is associated with the material data by, for example, adding the shooting time information to the metadata of the material data. Note that the device that adds the shooting time information is not particularly limited, and the shooting time information may be added by the shape estimation device 3 or the storage device 4. The storage device 4 outputs the material data in response to a request from another device.

[0037] The model operation device 5 is a device that instructs to set three-dimensional shape data of the background and to place three-dimensional shape data of the subject on the set three-dimensional shape data of the background. Setting the three-dimensional shape data of the background includes, for example, setting a coordinate system of the background in the virtual viewpoint video and determining the three-dimensional shape data of the background. The placement of the three-dimensional shape data of the subject is specified by a user (operator) using, for example, a joystick, a jog dial, a touch panel, a keyboard, or a mouse. The model operation device 5 outputs placement information that indicates the placement of the three-dimensional shape data based on an instruction from the user. The placement information includes, but is not limited to, information such as the position and orientation of the three-dimensional shape data.

[0038] The model operation device 5 displays an image showing a three-dimensional space in which a background and a subject are arranged, based on the setting of the three-dimensional shape data of the background and the arrangement information specified by the user. The user considers the setting of the three-dimensional shape data of the background and the arrangement of the three-dimensional shape data of the subject while referring to the three-dimensional space thus displayed. The model operation device 5 transmits the set arrangement information to the virtual camera path data processing device 1. Note that the designation of the arrangement information of the subject model is not limited to being specified by the user, and may be automatically designated by recognizing the subject through video analysis or the like.

[0039] The virtual camera operation device 6 is an operation device for designating parameters of the position and attitude of the virtual camera. The virtual camera operation device 6 is composed of, for example, a joystick, a jog dial, a touch panel, a keyboard, and a mouse. Note that the parameters of the virtual camera that can be designated include information such as the position, attitude, and angle of view of the virtual camera, but are not limited thereto, and other information may be designated.

[0040] The virtual camera operation device 6 in this embodiment transmits the virtual camera parameters designated by the user to the model operation device 5. The model operation device 5 generates a virtual viewpoint image based on the received virtual camera parameters and the material data acquired from the storage device 4. The user who operates the virtual camera operation device 6 designates the virtual camera parameters and considers the virtual camera path while referring to the virtual viewpoint image representing the three-dimensional space displayed by the model operation device 5. The displayed image may be an image that assists the user in designating the virtual camera parameters, and may be, for example, a simple model of the subject or a simple image using a marker indicating only the position of the subject. The virtual camera parameters may be designated without generating and displaying the image. The designation of the virtual camera parameters is not limited to designation by the user, and may be automatically designated by recognizing the subject. The virtual camera parameters designated by the virtual camera operation device 6 are transmitted to the virtual camera path data processing device 1.

[0041] In addition, the virtual camera operation device 6 can specify the shooting time of the subject in the video in response to a user operation. Specifically, the user can pause, reverse play, or fast forward the movement of the subject in the video displayed on the virtual camera operation device 6. This corresponds to pausing, reversing, or fast forwarding the shooting time when the shooting device 2 shoots. Even if the shooting time is changed, the virtual camera can be operated, so that it is possible to generate a virtual viewpoint video in which the virtual camera is moved while the movement of the subject is paused, for example. Even if continuous shooting is not performed or if shooting is performed with an interval between each subject, it is possible to generate a virtual viewpoint video including subjects shot at different times by referring to the shooting time information. In such a video, the shooting time and the playback time on the virtual viewpoint video are treated as independent parameters. The virtual camera operation device 6 transmits shooting time information indicating the shooting time corresponding to the video to be displayed to the virtual camera path data processing device 1.

[0042] The sequence data processing device 7 acquires virtual camera path data from the virtual camera path data processing device 1. The sequence data processing device 7 also acquires material data corresponding to the virtual camera path data from the storage device 4. The sequence data processing device 7 then generates sequence data for storing or outputting the acquired data. The format of the sequence data will be described later. Note that material data is not necessarily required, and sequence data containing only virtual camera path data may be generated. Furthermore, the virtual camera path may include not only one pattern but also multiple patterns of virtual camera path data.

[0043] The above is the configuration of the information processing system in this embodiment. Here, the problem to be solved by this system will be described with reference to FIG. 2. FIG. 2 is a diagram for explaining the relationship between the arrangement information of the subject model and the coordinate system adjustment information. As a premise, for example, when generating a virtual viewpoint video including a plurality of subjects, a problem may occur in which the plurality of subjects cannot be photographed simultaneously in the same photographing area due to restrictions on the location where the photographing is performed. In such a case, a method may be used in which the plurality of subjects are photographed at different locations or at different times, and the positions of the three-dimensional shape data of the photographed subjects are changed so that they are included in one virtual viewpoint video. However, as described above, when generating a virtual viewpoint video by rendering according to virtual camera parameters, there is a risk that an appropriate virtual viewpoint video will not be generated simply by moving the three-dimensional shape data of the subject and determining the virtual camera parameters. More specifically, the texture data for coloring the three-dimensional shape data of the subject is determined based on the pixel values ​​of the captured image specified according to the position and orientation of the virtual camera. Therefore, if the position of the three-dimensional shape data of the subject is moved, it may be difficult to refer to the pixel values ​​of the appropriate captured image. As a result, an appropriate virtual viewpoint video may not be obtained.

[0044] The virtual camera path data processing device 1 in this embodiment generates virtual camera path data including coordinate system adjustment information as information for adjusting the virtual camera path based on the position of the three-dimensional shape data of the subject. As a result, information capable of generating a desired virtual viewpoint video is output even in the virtual viewpoint video generation method such as this embodiment. The coordinate system adjustment information will be described below.

[0045] FIG. 2(a) shows an example of coordinate systems 2001-2004 of a three-dimensional space corresponding to the shooting space at the time of shooting, and three-dimensional shape data of a subject (hereinafter also referred to as subject model) 2005-2012, which are shot at the same place at different times. The coordinate systems of the three-dimensional space at the time of shooting are the same if the configuration of the imaging device and the calibration data are the same. Here, for the sake of explanation, the coordinate systems 2001-2004 are assumed to be the same, but they may be different. Also, subject model 2005 and subject model 2006 are assumed to have been shot at the same time. Similarly, subject model 2007 and subject model 2008, subject model 2009 and subject model 2010, and subject model 2011 and subject model 2012 are assumed to have been shot at the same time. However, the above four sets of subject models are assumed to have been shot at different times. The number of subjects included in the shooting is not limited to two, and may be one or more.

[0046] FIG. 2(b) shows an example of a coordinate system 2101 of a CG three-dimensional space or a three-dimensional space of another shooting space, which is different from the coordinate systems 2001 to 2004. FIG. 2(b) is set as a background for arranging the subject model here. FIG. 2(c) shows an example in which subject models 2005 to 2012 are arranged on the coordinate system 2101 based on a user's instruction. Subject models 2201 and 2202 in FIG. 2(c) correspond to subject models 2007 and 2008, and coordinate system 2203 corresponds to coordinate system 2101. Note that in FIG. 2(c), all subject models photographed at the same time are included for simplicity, and their positional relationships are maintained, but this is not limited thereto. For example, the relative positional relationship of subject model 2202 to subject model 2201 may be changed, or only subject model 2201 may be arranged, or subject model 2201 may be duplicated and arranged. In this case, since the positional relationship and number of subjects in the three-dimensional space at the time of shooting change, a desired virtual viewpoint image can be obtained by also storing information on the subject model or target area to be rendered as a virtual viewpoint image. The model operating device 5 makes it possible to recognize the three-dimensional space in which the subjects are located by performing a display such as that shown in Fig. 2(c).

[0047] Fig. 2(d) shows an example of a virtual camera path 2301 for the three-dimensional space and subject model shown in Fig. 2(c). The virtual camera path 2301 is specified, for example, by a user operating the virtual camera operating device 6 while watching an image displayed by the model operating device 3. A virtual camera 2302 represents a virtual camera such as a position and posture corresponding to a certain frame in the virtual camera path 2301. Moreover, a coordinate system 2303 in Fig. 2(d) corresponds to the coordinate system 2203 and the coordinate system 2101. Note that the virtual camera path 2301 is created based on the coordinate system 2303. In this case, the position of the virtual camera 2302 is described as (Xc, Yc, Zc).

[0048] FIG. 2(e) is a diagram for explaining the difference between the coordinate system 2403 at the time of photographing and the different coordinate system 2404 for the object models 2401 and 2402. The object models 2401 and 2402 in FIG. 2(e) correspond to the object models 2201 and 2202 and 2007 and 2008. The coordinate system 2403 corresponds to the coordinate system 2002 and represents the same coordinate system. Furthermore, the coordinate system 2404 corresponds to the coordinate system 2101, the coordinate system 2203, and the coordinate system 2303, and represents the same coordinate system. Since the positional relationship between each object model and the coordinate system 2403 at the time of photographing the object is always constant, if the position or orientation of each object model is changed, the position or orientation of the coordinate system 2403 changes by the same amount as the change. Therefore, if each object model is rearranged to an arbitrary position as in FIG. 2(c), the positional relationship and orientation between the coordinate systems 2403 and 2404 change according to the amount of change 2405. For example, assume that only the position of object model 2401 has been changed. In this case, if the position (center of gravity) of object model 2401 is expressed as (Xw, Yw, Zw) when viewed from coordinate system 2404 and as (Xv, Yv, Zv) when viewed from coordinate system 2403, then the amount of change 2405 is (Xw-Xv, Yw-Yv, Zw-Zv). This amount of change 2405 becomes coordinate system adjustment information.

[0049] FIG. 2(f) is a diagram for explaining an example when the virtual camera path 2301 in FIG. 2(d) is adjusted to a virtual camera path 2501 based on a coordinate system 2503. The virtual camera 2502 represents a virtual camera such as a position and posture corresponding to a certain frame in the virtual camera path 2501, and the coordinate system 2503 corresponds to the coordinate systems 2303, 2203, and 2101. In this case, the position of the virtual camera 2502 after adjustment is described as (XC, YC, ZC). In addition, the object models 2504 and 2505 correspond to the object models 2201 and 2202, 2007 and 2008, 2401, and 2402, respectively. Here, in the virtual camera paths 2501 and 2301, in order to generate similar virtual viewpoint images except for the background, the relative positions and orientations of the object models as seen from the virtual camera must not change. However, if the virtual camera path 2301 is applied to the coordinate system 2503 without any adjustment, the virtual camera 2502 will be located at a position (Xc, Yc, Zc) in the coordinate system 2503. At this time, the values ​​of Xc, Yc, and Zc are highly likely to be different from XC, YC, and ZC, respectively. If they are not the same, the relative positions and orientations of the object models as seen by the virtual camera will be lost. Therefore, when the position of the object model is changed, it is necessary to adjust the position (Xc, Yc, Zc) of the virtual camera in the coordinate system 2503. In the case of FIG. 2(f), the adjustment is performed by subtracting the change amount 2405 (Xw-Xv, Yw-Yv, Zw-Zv) from the coordinates (Xc, Yc, Zc).

[0050] In this way, the relative positional relationship between the subject model and the virtual camera is maintained by moving the position of the virtual camera in accordance with the movement of the position of the subject model, which makes it possible to refer to appropriate pixel values ​​of the captured image when coloring the subject model.

[0051] In the above example, the case where only the position of the virtual camera is changed has been described, but the present invention is not limited to this. For example, when the orientation of the subject model is changed, a transformation matrix for changing the attitude of the virtual camera in accordance with the changed orientation is specified. This transformation matrix becomes coordinate system adjustment information. Thus, in this embodiment, coordinate system adjustment information for changing at least one of the position and attitude of the virtual camera is acquired based on the arrangement information representing the position and orientation of the subject model.

[0052] <Functional configuration of the virtual camera path data processing device> 1, the configuration of the virtual camera path data processing device 1 will be described. The virtual camera path data processing device 1 has a virtual camera information acquisition unit 101, a coordinate system adjustment information acquisition unit 102, a virtual camera path data generation unit 103, and a virtual camera path data output unit 104.

[0053] Virtual camera information acquisition unit 101 acquires virtual camera parameters specified by virtual camera operation device 6. Note that virtual camera information acquisition unit 101 may acquire the virtual camera parameters for all frames determined by virtual camera operation device 6 all at once, or may continue to acquire the virtual camera parameters for one frame each time virtual camera operation device 6 outputs the virtual camera parameters for one frame.

[0054] It should be noted that the virtual camera information acquisition unit 101 in this embodiment also acquires shooting time information corresponding to the time when the subject was photographed. The shooting time information in this embodiment is absolute time in a certain standard time, but is not limited to this. The shooting time information may be expressed, for example, as a relative time with respect to a certain reference time, and a relative frame number with respect to a frame corresponding to the certain reference time. In addition, for example, when material data corresponding to a specific scene during shooting (for example, the first 45 minutes of a soccer game) or a certain game is managed as a group, a relative time based on the start timing of the scene or game may be used as the shooting time information.

[0055] Since the shooting time information is information used to identify material data corresponding to a certain frame, information other than the shooting time information may be acquired as long as the material data can be identified. For example, if an identification number or ID is assigned to identify the material data, the virtual camera information acquisition unit 101 may be configured to acquire the identification number and ID of the material data instead of the shooting time information. The virtual camera information acquisition unit 101 may acquire the shooting time information for all frames determined by the virtual camera operation device 6 all at once, or may continue to acquire the information for each frame.

[0056] The coordinate system adjustment information acquisition unit 102 acquires the placement information of the subject model specified by the model operating device 5, and specifies the position and orientation of the subject in the virtual viewpoint video based on the acquired placement information. The coordinate system adjustment information acquisition unit 102 also acquires the coordinate system adjustment information based on the placement information. Note that the model operating device 5 may acquire the coordinate system adjustment information, and the coordinate system adjustment information acquisition unit 102 may acquire the coordinate system adjustment information acquired by the model operating device 5.

[0057] The virtual camera path data generating unit 103 generates virtual camera path data in which the virtual camera parameters acquired by the virtual camera information acquiring unit 101 and the coordinate system adjustment information acquired by the coordinate system adjustment information acquiring unit 102 are associated with each other. The concept of virtual camera path data will be described with reference to FIG. 11. FIG. 11 is an example of virtual camera path data in which coordinate system adjustment information is stored in the virtual camera path 2301 in FIG. 2(d). Here, the virtual camera path time information is information representing the number of a frame constituting the virtual camera path. In this example, the virtual camera path has 100 frames, and it is represented that there are four types of coordinate system adjustment information of the virtual camera for each frame. Note that the coordinate system adjustment information is given as a position offset for adjusting the position of the virtual camera. The virtual camera parameter information includes, for example, the position (x, y, z) of the virtual camera in a three-dimensional space, and pan, tilt, and roll values ​​(pan, tilt, roll) representing the attitude of the virtual camera in a three-dimensional space. Note that the virtual camera path data may include parameters corresponding to the angle of view and distortion of the virtual camera.

[0058] The shooting time information is information for identifying the material data used to generate the virtual viewpoint video, and in the example of Fig. 11, it is represented by a frame number indicating the relative frame number with respect to the frame of a predetermined captured image. In the example of Fig. 11, for example, there are four shooting time information of the subject corresponding to the virtual camera path time information 0, namely, 0, 1000, 2000, and 3000. That is, it is shown that there are four pieces of material data used when generating the frame of the virtual viewpoint video corresponding to the virtual camera path time information 0. The virtual camera path data processing device 1 in this embodiment generates virtual camera path data as shown in Fig. 11. The method of using the virtual camera path data will be described later in the second embodiment.

[0059] Returning to Fig. 1, the virtual camera path data output unit 104 performs processing to add header information and the like to the virtual camera path data generated by the virtual camera path data generation unit 103, and outputs the data. The virtual camera path data may be output as a data file or as packet data. Furthermore, the virtual camera path data may be output on a frame-by-frame basis, or may be output for the entire virtual camera path or for every fixed number of frames.

[0060] <Virtual camera path data format> 4 shows an example of a format of virtual camera path data generated by the virtual camera path data processing device 1. The virtual camera path data processing device 1 generates a file corresponding to a predetermined format as described below.

[0061] The virtual camera path data shown in FIG. 4 is an example of a data set having virtual camera parameters for a predetermined number of frames. At the beginning of the data set, a virtual camera path data header is saved, and the header saves that this data set is a data set of virtual camera path data and the data size of the data set. Next, the number of frames M of the stored virtual camera path data is written. Then, information on the format of the virtual camera path data is written. This is information indicating the format of how the virtual camera path data is stored, and is information indicating whether the data related to the virtual camera path is stored by type or by frame. FIG. 3 is an example in which data is stored by type. Next, the number of data is written as L in the virtual camera path data header. The information of each data is saved below. In this embodiment, four data are included: virtual camera path time information, shooting time information, camera parameter information, and coordinate system adjustment information. Note that the virtual camera path time information is information indicating a frame in the virtual camera path, and corresponds to the camera path frame in FIG. 11 described above. That is, the virtual camera path time information is expressed by any information that can distinguish frames in the virtual camera path.

[0062] In the information of the data unit of the virtual camera path, a data type code is first saved. In this embodiment, the data set includes virtual camera path time information, shooting time information, camera parameter information, and virtual advertisement display instruction information. Each of these is expressed as a virtual camera path data type code. The virtual camera path data type code is expressed as a 1-byte code as shown in FIG. 3. However, the data type and code are not limited to this, and may be, for example, a code longer than 2 bytes or a code shorter than 2 bytes depending on the information to be described. Other data used when generating a virtual viewpoint video may be used. In addition, the virtual advertisement display instruction information is instruction information for displaying a virtual advertisement as additional information in a virtual viewpoint video generated using the virtual camera path data. The virtual advertisement is an example of additional information, and instruction information for adding any additional information such as information about a subject or an effect other than the virtual advertisement may be written in the virtual camera path data.

[0063] Next, access information (pointer) to each data is saved. Then, format information related to data such as shooting time information and camera parameter information is written. For example, the format of virtual camera path time information indicates whether it is a relative time based on the beginning or a relative frame number. The format of shooting time information may be a format that uses absolute time based on a certain standard time (for example, Greenwich Mean Time or Japan Standard Time) when the subject is actually shot, a relative time to a certain reference time, a relative frame number to a reference frame, or the like. Alternatively, the format may be a format that uses information other than shooting time information, such as a file path or pointer to material data. When describing a relative time or a relative frame number to a reference time as shooting time information, the reference time information of the virtual camera path is written in the virtual camera path sequence description in the sequence header. This reference time information is saved as an absolute time based on a certain standard time, an absolute time based on a management time in a certain content (for example, the start time of a specific match, etc.), or the like. Specifically, for example, the year, month, day, hour, minute, and second may be expressed as integers, and the time after the decimal point may be expressed as a floating point, fixed point, integer, or the like.

[0064] The format of the virtual camera parameters is, for example, a quaternion representation of values ​​representing the position, orientation, etc. of the virtual camera. Note that the method of representing the virtual camera parameters is not limited to this.

[0065] The format of the coordinate system adjustment information is expressed, for example, by offset information of the position of the virtual camera and transformation matrix information representing the transformation of the position and posture of the camera. Note that the virtual camera path time information and the camera parameter information correspond one-to-one, but the virtual camera path time information and the shooting time information may correspond one-to-many according to the number of subjects for which the coordinate system is adjusted. For example, in the case of FIG. 2(d), if the subjects in each shooting space are treated as subject groups, there are four types of subject groups of shooting time information corresponding to one virtual camera path time information. Furthermore, the coordinate system adjustment information may be in units of a specified shooting time zone, not in units of a frame. For example, in the case of FIG. 2(d), if the coordinate system adjustment information does not change in each shooting time zone of the four subject groups, the number of coordinate system adjustment information may be 4, not M×4. That is, the coordinate system adjustment information data set may be in a format as shown in FIG. 5. Note that the coordinate system adjustment information format code may be written in the format information of the data information in the virtual camera path data header described in FIG. 4. Note that when the number of subject groups K is 1, the information held by the coordinate system adjustment information may be only the coordinate system adjustment data.

[0066] Hereinafter, the actual data of each piece of virtual camera path data is written according to the format described in the virtual camera path data header. Note that at the beginning of each piece of data, a start code indicating the beginning of that data is written.

[0067] Fig. 6 is a diagram for explaining another example of the configuration of virtual camera path data. In Fig. 4, various data included in the virtual camera parameters are stored on a frame-by-frame basis. When data is stored on a frame-by-frame basis, a frame data header is saved at the beginning of each frame data, and a code indicating the start of the frame data and information indicating what data is stored in what order in the frame data are written.

[0068] Storing various data on a frame-by-frame basis has the advantage that it is easy to extract data on a frame-by-frame basis, for example when wanting to extract some frames of the virtual camera path data and use them to generate a virtual viewpoint video.

[0069] 3 is a diagram showing an example of the format of sequence data output by the sequence data processing device 7. The sequence data processing device 7 outputs sequence data including a plurality of virtual camera path data for generating one virtual viewpoint video. The sequence data also includes material data used to generate the virtual viewpoint video. For example, a sequence is generated for each video clip or each cut of shooting. In this way, with a configuration including virtual camera path data and material data, the device that receives the sequence data can generate a virtual viewpoint video.

[0070] Each sequence includes a sequence header, and the sequence header stores information that can identify the corresponding material sequence data. For example, but not limited to, a sequence header start code related to the material data that can uniquely identify the material sequence data, information on the location and date of shooting the subject, path information indicating the location of the material sequence data, etc. are stored. In addition, the sequence header includes information indicating that the above-mentioned virtual camera path data is included. For example, the information may be information indicating the data set included in the sequence header, or information indicating the presence or absence of virtual camera path data.

[0071] Next, information about the entire sequence data of the virtual camera path is saved. For example, information about the creator of the virtual camera path, information about the right holder, the name of the sequence, the name of the event in which the subject was photographed, the camera frame rate at the time of photographing, etc. may be included in the virtual camera path sequence description information. In addition, for example, time information based on the virtual camera path, image size and background data information assumed at the time of rendering the virtual viewpoint video, etc. may be included in the virtual camera path sequence description information. However, it is not necessary to include all of this information, and any information may be included.

[0072] In the sequence, each virtual camera path data is stored in a data set unit. The number N of data sets is written in the sequence header. The following data set unit information is stored. In this embodiment, two data sets are included: virtual camera path data and material data. In the data set unit information, an identification ID of the data set is given first. The identification ID is given as an ID unique to all data sets. Next, a data set type code is stored. In this embodiment, the data sets are assumed to include a data set representing the virtual camera path data and data representing the material data. Each is expressed as a data set type code. The data set type code is expressed as a 2-byte code shown in FIG. 2. However, the data type and code are not limited to this. Other data used when generating a virtual viewpoint video may be used. Next, a pointer to the data set is stored. However, it is sufficient if it is information for accessing each data set, and is not limited to a pointer. For example, it may be a file name in a file system constructed in the storage device 4.

[0073] The sequence data processing device 7 may generate sequence data including only virtual camera path data or only material data. By generating sequence data, data used to generate one virtual viewpoint video such as a video clip can be collectively managed as one data.

[0074] According to the format described above, data is generated that includes coordinate system adjustment information for changing the position and orientation of the virtual camera, and a plurality of virtual camera parameters corresponding to the frames that constitute the virtual viewpoint video. Note that the above-described format is an example, and the configuration and included information are not limited to the above.

[0075] <Processing flow> The operation of the virtual camera path data generation processing device 1 in this embodiment will be described with reference to the flowchart in Fig. 7. The process shown in Fig. 7 is realized by the CPU 1201 reading and executing a program stored in the ROM 1202 or the external storage device 1206. The process is started when an instruction to generate virtual camera path data is received from the virtual camera operation device 6.

[0076] In step S701, the virtual camera path data generating unit 103 generates header information for generating virtual camera path data, and sets the header of the virtual camera path data. In step S702, the virtual camera information acquiring unit 101 and the coordinate system adjustment information acquiring unit 102 acquire each piece of data information of the virtual camera path, i.e., virtual camera path time information, shooting time information, camera parameter information, and coordinate system adjustment information, from the virtual camera operation device 6. In addition, the virtual camera path data generating unit 103 generates a header for each piece of data.

[0077] In step S703, data storage is repeated on a frame-by-frame basis from the start of the virtual camera path. In step S704, the virtual camera path data generation unit 103 generates virtual camera path data storing the coordinate system adjustment information acquired by the coordinate system adjustment information acquisition unit 102 for the virtual camera information acquired by the virtual camera information acquisition unit 101. In step S705, steps S703 to S704 are repeated until the virtual camera path ends or input on a frame-by-frame basis ends. In step S706, the virtual camera path data output unit 104 stores header information and the like for the virtual camera path data generated by the virtual camera path data generation unit 103, and outputs it as final virtual camera path data.

[0078] The data exchanged at one time may be in units of frames, or may be collected in units of multiple frames or virtual camera paths. For example, the virtual camera information acquisition unit 101 may receive virtual camera information in units of frames or multiple frames. When processing is performed in units of multiple frames, the coordinate system adjustment information for each subject model may be described in a coordinate system adjustment information data set in units of subjects.

[0079] The above processing makes it possible to generate virtual camera path data that can generate a virtual viewpoint video that takes into account changes in the position of the subject model.

[0080] (Embodiment 2) In this embodiment, an example will be described in which a process for generating a virtual viewpoint video is performed based on the virtual camera path data generated by the virtual camera path data processing device 1 in the first embodiment.

[0081] <System configuration and functional configuration of the virtual viewpoint video generation device> FIG. 8 is a diagram showing an example of the configuration of a system including a virtual viewpoint video generation device in this embodiment. The virtual viewpoint video generation device 800 is connected to a virtual camera path data processing device 1 and a storage device 4. The configurations of the virtual camera path data processing device 1 and the storage device 4 are the same as those of the first embodiment. Hereinafter, a description of the same configuration as that of the first embodiment will be omitted. The hardware configuration of the virtual viewpoint video generation device 800 is assumed to be the same as the hardware configuration shown in FIG. 10. The virtual viewpoint video generation device 800 may be included in the information processing system 10 in the first embodiment.

[0082] Virtual viewpoint video generation device 800 has a virtual camera path data acquisition unit 801, a virtual camera information acquisition unit 802, a material data management unit 803, a coordinate system adjustment information acquisition unit 804, a camera parameter information adjustment unit 805, a virtual viewpoint video generation unit 806, and a virtual viewpoint video output unit 807. Virtual viewpoint video generation device 800 in this embodiment acquires virtual camera path data from virtual camera path data processing device 1, and generates a virtual viewpoint video based on the acquired virtual camera path data. Each processing unit will be described below.

[0083] The virtual camera path data acquisition unit 801 acquires the virtual camera path data output from the virtual camera path data processing device 1. The virtual camera path data acquisition unit 801 may acquire the virtual camera path data output from the virtual camera path data processing device 1 as a data file or as packet data. The virtual camera path data acquisition unit 801 may acquire the virtual camera path data in units of frames, in units of a certain number of frames, or in units of one or more virtual camera path data sets. When a plurality of virtual camera path data sets are acquired, the virtual viewpoint video output unit 807 described later can output the virtual viewpoint video corresponding to each virtual camera path data set in a distinguished manner. Each virtual camera path data set can be distinguished by an identification ID described in each virtual camera path data set header. The virtual viewpoint video output unit 807 described later may perform a process of adding an identification ID described in the virtual camera path data set to the metadata of the virtual viewpoint video to be output.

[0084] The virtual camera information acquisition unit 802 acquires the camera parameters of the virtual camera from the virtual camera path data acquired by the virtual camera path data acquisition unit 801. The virtual camera information acquisition unit 802 also acquires shooting time information corresponding to the virtual camera path time information included in the virtual camera path data. Note that, without being limited to this, instruction information for adding additional information such as a virtual advertisement and information on the rendering target may also be acquired.

[0085] The material data management unit 803 acquires material data corresponding to the shooting time information acquired by the virtual camera information acquisition unit 802 from the storage device 4. As described in the first embodiment, the material data is associated with the shooting time information, so the material data management unit 803 can acquire the material data by referring to the shooting time information associated with the material data. Note that the material data management unit 803 holds the correspondence between the acquired material data, the virtual camera path data set, and the virtual camera path time information.

[0086] The material data acquired by the material data management unit 803 is necessary material data based on the method for generating a virtual viewpoint video in the virtual viewpoint video generation unit 806. For example, in the case of a generation method based on a foreground model or background model, point cloud model data or mesh model data of the foreground or background, and photographed images and camera calibration data for generating corresponding texture images and textures are acquired. In the case of a generation method that does not use a foreground model or background model, photographed images, camera calibration data, etc. are acquired.

[0087] A coordinate system adjustment information acquisition unit 804 acquires coordinate system adjustment information for adjusting the coordinate system of the virtual camera from the virtual camera path data acquired by the virtual camera path data acquisition unit 801. A camera parameter information adjustment unit 805 changes the virtual camera parameters based on the coordinate system adjustment information acquired by the coordinate system adjustment information acquisition unit 804.

[0088] A virtual viewpoint video generation unit 806 generates a virtual viewpoint video using the virtual camera parameters changed by the camera parameter information adjustment unit 805 and the material data acquired by the subject data management unit 803. Note that the virtual camera parameters and the material data used correspond to each other. The corresponding virtual camera parameters and material data are identified, for example, by referring to the same virtual camera path time information.

[0089] Here, an example of a method for generating a virtual viewpoint video based on virtual camera path data will be described. First, the camera parameter information adjustment unit 805 changes (adjusts) the virtual camera parameters based on the coordinate system adjustment information corresponding to each piece of shooting time information. Also, the virtual viewpoint video generation unit 806 generates a virtual viewpoint video independently using material data corresponding to each piece of shooting time information based on the virtual camera parameters after the coordinate system adjustment. At this time, a distance image representing the distance from the virtual camera to the subject model is also generated for each virtual viewpoint video. Then, the virtual viewpoint video generation unit 806 synthesizes the multiple virtual viewpoint videos based on the multiple independently generated virtual viewpoint videos and the distance image generated together so that the subject model closer to the virtual camera position is displayed in the foreground.

[0090] The virtual camera path data shown in FIG. 11 will be taken as an example for explanation. For virtual camera path time information 0, the camera parameter information adjustment unit 805 adjusts the virtual camera parameter P0 based on the coordinate system adjustment information corresponding to each of the four pieces of shooting time information. In addition, the virtual viewpoint video generation unit 806 generates four virtual viewpoint videos using the four virtual camera parameters after the coordinate system adjustment and the material data corresponding to each of the shooting times 0, 1000, 2000, and 3000. At this time, a distance image representing the distance from the virtual camera to the subject model is also generated for each virtual viewpoint video. Based on the four generated virtual viewpoint videos and the distance image generated together, the virtual viewpoint video generation unit 806 synthesizes the four virtual viewpoint videos so that the subject model closer to the virtual camera position is reflected in the foreground. As a result, four virtual viewpoint videos corresponding to the same virtual camera path time information 0 are synthesized, and a frame of one virtual viewpoint video is generated. By performing the same process on frames corresponding to other virtual camera path time information, 100 frames of virtual viewpoint videos are finally generated.

[0091] By carrying out the above processing, one virtual viewpoint image can be generated. In the above generation method, the virtual viewpoint images are generated independently for each piece of shooting time information and then synthesized, but this is not limited to the above. For example, virtual viewpoint images based on each piece of shooting time information may be generated in order and overwritten in order according to the distance from the virtual camera to the subject model. In addition, the coloring method used when generating the virtual viewpoint image is based on the virtual camera information, which is assumed to provide a higher quality image, but other methods such as general texture mapping may also be used.

[0092] The virtual viewpoint video output unit 807 acquires the virtual viewpoint video from the virtual viewpoint video generation unit 806, and outputs the virtual viewpoint video using a display device such as a display. Note that the virtual viewpoint video output unit 807 may output the virtual viewpoint video acquired from the virtual viewpoint video generation unit 806 as an image data file or packet data.

[0093] <Processing flow> The operation of the virtual viewpoint video generation device in the above configuration will be described with reference to the flowchart in Fig. 9. Processing is started when the virtual viewpoint video generation device 800 acquires virtual camera path data. In step S900, a camera path data header is read from the virtual camera path data. Based on the header information, memory required for processing is allocated, etc. Next, each piece of data information is read, and the header of each piece of information and access information for the data are acquired.

[0094] In step S901, the virtual viewpoint video generating device 800 repeats acquisition of data input frame by frame from the start of the virtual camera path. In step S902, the virtual camera path data acquiring unit 801 acquires virtual camera path time information, shooting time information, camera parameter information, and coordinate system adjustment information corresponding to the frame input in step S901 from the virtual camera path data processing device 1. In step S903, the virtual camera information acquiring unit 802 acquires virtual camera information of the frame, that is, camera parameter information, based on the virtual camera path data acquiring unit 801 or the camera parameter header. In addition, the virtual camera information acquiring unit 802 acquires shooting time information of the frame based on the shooting time information header. Since the data size of these pieces of information is fixed length, the position of the beginning of the data to be read can be specified by the number of frames from the beginning.

[0095] In step S904, the subject data management unit 803 acquires, from the storage device 4, material data corresponding to the shooting time information acquired by the virtual camera information acquisition unit 802. Material data is selected from the subject data stored in the storage device 4 based on the shooting time.

[0096] In step S905, the coordinate system adjustment information acquisition unit 804 acquires the coordinate system adjustment information of the frame from the virtual camera path data acquisition unit 801 based on the coordinate system adjustment information header. In step S906, the camera parameter information adjustment unit 805 adjusts the virtual camera parameters acquired by the virtual camera information acquisition unit based on the coordinate system adjustment information acquired by the coordinate system adjustment information acquisition unit 804. Specifically, the camera parameter information adjustment method will be described using the example shown in FIG. 2. For example, the camera parameter information of a certain frame acquired by the virtual camera information acquisition unit is set as the virtual camera parameters of the virtual camera 2302. At this time, the coordinate system adjustment information acquired by the coordinate system adjustment information acquisition unit 804 corresponds to the change amount 2405. In the example of FIG. 2, the subject model is not changed in orientation, and is arranged by translating by the change amount 2405. Therefore, the adjustment of the camera parameter information is performed by subtracting the change amount 2405 from the position coordinates in the camera parameter information.

[0097] In step S907, the virtual viewpoint video generation unit 804 generates a virtual viewpoint video based on the subject data acquired by the subject data management unit 803 and the camera parameter information adjusted in step S906. In step S908, the virtual viewpoint video output unit 605 outputs the virtual viewpoint video generated by the virtual viewpoint video generation unit 604 using a display device such as a display, or outputs it as a data file or packet data. In step S909, steps S901 to S908 are repeated until the virtual camera path ends or the input in frame units ends.

[0098] FIG. 10 is a diagram showing the state of communication between each unit. First, the virtual camera path data processing device 1 is started and notifies the virtual viewpoint video generation device 800 of the start of generation of the virtual viewpoint video. Here, although not shown in FIG. 8, a control unit that controls the virtual viewpoint video generation device, which is composed of a CPU or the like, notifies the virtual camera path data acquisition unit 801 and each unit of the start of generation of the virtual viewpoint video, and each unit prepares for it. Next, the virtual camera path data processing device 1 transmits virtual camera path data to the virtual camera path data acquisition unit 801. The virtual camera path data acquisition unit 801 transmits virtual camera information and shooting time information in the virtual camera path data obtained by interpreting the header of the transmitted virtual camera path data to the virtual camera information acquisition unit 802, and transmits coordinate system adjustment information to the coordinate system adjustment information acquisition unit 804.

[0099] The virtual camera information acquisition unit 802 sends the shooting time information to the subject data management unit 803, and sends the camera parameter information to the camera parameter information adjustment unit 805. Furthermore, the coordinate system adjustment information acquisition unit 804 sends the acquired coordinate system adjustment information to the camera parameter information adjustment unit 805. The subject data management unit 803 acquires material data corresponding to the input shooting time information from the storage device 4, and sends the acquired material data to the virtual viewpoint video generation unit 806. The camera parameter information adjustment unit 805 adjusts the virtual camera parameters based on the acquired coordinate system adjustment information and virtual camera parameters, and sends the adjusted virtual camera parameters to the virtual viewpoint video generation unit 806. The virtual viewpoint video generation unit 806 renders a virtual viewpoint video based on the acquired subject data and adjusted camera parameter information.

[0100] As soon as rendering is complete, the virtual viewpoint video generation unit 806 sends the generated virtual viewpoint video to the virtual viewpoint video output unit 807 and requests information about the next frame of the virtual viewpoint video to be generated. Thereafter, in order to process the next frame, the acquisition of virtual camera path data, acquisition of material data corresponding to the virtual camera path data, acquisition and adjustment of virtual camera parameters, and generation and output of the virtual viewpoint video are repeated. When a transmission end is sent from the virtual camera path data processing device 1 to the control unit that controls the virtual viewpoint video generation device, all processing ends.

[0101] In the present embodiment, the process is shown in a sequential flow in the flowchart, but is not limited to this. For example, output of a plurality of virtual camera path data may be performed in parallel.

[0102] The data exchanged at one time may be in units of frames, or may be exchanged in units of multiple frames. For example, the virtual camera path data acquisition unit 801 may receive the virtual camera path data in units of multiple frames. In this case, for example, the virtual camera path data for the received frames may be stored in the virtual camera path data acquisition unit 801, and the stored virtual camera path data information may be transmitted sequentially in units of frames, or may be transmitted in units of multiple frames. Also, the transmission order of the virtual camera parameters and the subject data sent to the virtual viewpoint video generation unit 806 is not limited to this, and the order may be reversed, or they may be sent simultaneously.

[0103] As described above, by using the virtual camera path data including the virtual camera parameters and the coordinate system adjustment information, even if the position of the three-dimensional shape data of the subject is changed, an appropriate virtual viewpoint video can be generated. In this embodiment, the virtual viewpoint video generation device 800 is configured to directly acquire the virtual camera path data from the virtual camera path data processing device 1, but is not limited to this. The virtual viewpoint video generation device 800 may be configured to acquire the virtual camera path data stored in the storage device 4 or another storage device, for example. In addition, for example, the virtual viewpoint video generation device 800 may generate a virtual viewpoint video using the sequence data generated by the sequence data generation device 7 in the first embodiment. At this time, if the sequence data includes material data, the virtual viewpoint video generation device 800 can generate a virtual viewpoint video without referring to the material data stored in the storage device 4.

[0104] (Other embodiments) In the above-mentioned embodiment, the coordinate system adjustment information is used as the change information for changing at least one of the position and the orientation of the subject in the virtual viewpoint video, and the virtual camera parameters are changed, but the present invention is not limited thereto. In the above-mentioned embodiment, the color information of the subject model is acquired based on the pixel value of the captured image according to the virtual camera parameters, and the method of rendering the virtual viewpoint video is used. However, as the subject model used to generate the virtual viewpoint video, a subject model that has already been colored (with a fixed color) may be used. The colored subject model may be, for example, data having colored point cloud model data and mesh model data, and texture data of a fixed color. When using a colored subject model, it is not necessary to consider the relationship between the virtual camera and the captured image used for coloring. Therefore, in this case, virtual camera path data is generated in which information indicating how much the position and orientation of the subject model are changed from the time of shooting is associated with the virtual camera parameters. This makes it possible to generate a virtual viewpoint video in which the position and orientation of the subject model are changed. In addition, the above-mentioned change information is information indicating how much the subject is changed from the position and orientation of the subject at the time of shooting, but the present invention is not limited thereto. For example, it may be information that indicates the amount of change in the position and orientation of the subject from a reference frame among multiple frames in the virtual camera path data.

[0105] Furthermore, for example, the information included in the virtual camera path data may be information representing the position and orientation of the subject in the virtual viewpoint video, that is, the placement information itself generated by the model operating device 5. In this case, the virtual camera path data is data in which the information on the position and orientation of the subject, which is the placement information, is associated with the virtual camera parameters.

[0106] The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments is supplied to an apparatus or an apparatus via a network or a storage medium, and one or more processors in the apparatus or the apparatus's computer read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more functions. [Explanation of symbols]

[0107] 1 Virtual camera path data processing device 101 Virtual camera information acquisition unit 102 Coordinate system adjustment information acquisition unit 104 Virtual camera path data output unit

Claims

1. an acquisition means for acquiring first information indicating a position of three-dimensional shape data of a subject generated based on a plurality of captured images, and second information indicating a position of a virtual viewpoint and a line of sight direction from the virtual viewpoint; a first change means for changing a position of the three-dimensional shape data of the subject based on the first information; second change means for changing a position of the virtual viewpoint based on an amount of change made by the first change means and the second information; an identification means for identifying a captured image based on the position of the virtual viewpoint and the line of sight direction from the virtual viewpoint after the change made by the second change means; a generating means for generating a virtual viewpoint image based on the photographed image specified by the specifying means and the position of the three-dimensional shape data of the subject changed by the first changing means; An information processing system comprising:

2. 2. The information processing system according to claim 1, wherein the generating means determines a color of the subject included in the virtual viewpoint image based on the photographed image specified by the specifying means.

3. 3. The information processing system according to claim 1, wherein the photographed image specified by the specifying unit is included in the plurality of photographed images.

4. The information processing system according to any one of claims 1 to 3, characterized in that the second change means changes the position of the virtual viewpoint based on a change direction and an amount of change in the position of the three-dimensional shape data of the subject due to the change made by the first change means and on the second information.

5. 5. The information processing system according to claim 1, wherein an amount of change made by said first change means and an amount of change made by said second change means are the same.

6. The first change means changes a direction of the three-dimensional shape data of the subject, The information processing system according to any one of claims 1 to 5, characterized in that the second change means identifies the captured image based on a change direction and an amount of change in the orientation of the three-dimensional shape data of the subject caused by the change made by the first change means.

7. The information processing system according to claim 1 , wherein the virtual viewpoint image is generated based on the position of the virtual viewpoint and the line of sight direction from the virtual viewpoint after the change made by the second change means.

8. 8. The information processing system according to claim 1, wherein the first change means changes a position of the three-dimensional shape data of the subject based on a user operation.

9. The information processing apparatus according to claim 1 , wherein the second information is acquired based on a user operation.

10. an acquisition step of acquiring first information indicating a position of three-dimensional shape data of a subject generated based on a plurality of captured images, and second information indicating a position of a virtual viewpoint and a line of sight direction from the virtual viewpoint; a first changing step of changing a position of the three-dimensional shape data of the subject based on the first information; a second change process of changing the position of the virtual viewpoint based on an amount of change made by the first change process and the second information; a specifying step of specifying a captured image based on a position of the virtual viewpoint and a line of sight direction from the virtual viewpoint after the change made in the second changing step; a generating step of generating a virtual viewpoint image based on the photographed image identified in the identifying step and the position of the three-dimensional shape data of the subject changed in the first changing step; 13. An information processing method comprising:

11. A program for causing a computer to function as the information processing system according to any one of claims 1 to 9.

Images