Information processing device, information processing method, and program

Abstract

To output material data related to generation of a virtual viewpoint image that can be processed by a device of a data output destination.SOLUTION: An information processor 100 comprises: an acquisition unit 105 for acquiring material data that is used for generating a virtual viewpoint image based on a plurality of photographed images obtained by photographing an object with a plurality of cameras 101 and is represented by a first form; a conversion unit 107 for converting a form of the acquired material data from the first form to a second form on the basis of information for identifying a form of material data that can be processed by another device of an output destination of the material data; and a transmission / reception unit 108 for outputting the material data converted into the second form to the another device.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to a technique for generating a virtual viewpoint image. [Background technology]

[0002] In recent years, a technology that captures an image of a subject by arranging multiple imaging devices around an imaging area and generates an image (virtual viewpoint image) viewed from a specified viewpoint (virtual viewpoint) using the multiple captured images acquired from each imaging device has been attracting attention. With this technology, for example, highlight scenes of soccer or basketball games can be viewed from various angles, providing the user with a high sense of realism compared to normal images.

[0003] Patent Document 1 describes a system that generates point cloud data consisting of a plurality of points indicating three-dimensional positions as three-dimensional shape data that represents the shape of a subject. The system described in Patent Document 1 also generates an image viewed from an arbitrary viewpoint (virtual viewpoint image) by performing a rendering process using the generated point cloud data.

[0004] Patent Document 2 describes a technology for restoring a wide range of three-dimensional models including not only the front area of ​​an object but also the side area from information acquired by an RGB-D camera. According to Patent Document 2, an object area is extracted based on depth data of a depth map, and a three-dimensional surface model of the object is generated based on the depth data. Also, according to Patent Document 2, a virtual viewpoint image is generated by performing a rendering process using a three-dimensional mesh model generated using the three-dimensional surface model. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2018 / 147329 [Patent Document 2] JP 2016-71645 A Summary of the Invention [Problem to be solved by the invention]

[0006] As described above, there are a number of methods for generating a virtual viewpoint image, and each method uses different data. Therefore, if the data output to the device that executes the process related to the generation of the virtual viewpoint image is not compatible with the output device, the process may not be executed appropriately.

[0007] The present disclosure has been made in view of the above-mentioned problems. It is an object of the present disclosure to make it possible to output material data related to the generation of a virtual viewpoint image that can be processed by a device to which the data is output. [Means for solving the problem]

[0008] The information processing device according to the present disclosure performs image processing based on a plurality of captured images obtained by capturing images of a subject using a plurality of imaging devices. Generated by Used to generate virtual viewpoint images 3D shape data and is represented by the first form 3D shape data An acquisition means for acquiring the Generate virtual viewpoint images using 3D shape data Other Equipment Information on the software used to generate virtual viewpoint images The acquisition means acquires the 3D shape data a conversion means for converting the format of the first format into a second format, and 3D shape data and an output unit for outputting the signal to the other device. Effect of the Invention

[0009] According to the present disclosure, it becomes possible to output material data related to the generation of a virtual viewpoint image that can be processed by a device to which the data is output. [Brief description of the drawings]

[0010] [Figure 1] FIG. 2 is a block diagram illustrating an example of a hardware configuration of an information processing device. [Diagram 2] FIG. 2 is a diagram illustrating an example of a configuration of a virtual viewpoint image generating system including an information processing device according to the first embodiment. [Diagram 3] 4 is an example of a data structure of material data stored in a storage unit. [Figure 4] 4 is an example of a data structure of material data stored in a storage unit. [Diagram 5] 4 is an example of a data structure of material data stored in a storage unit. [Figure 6] 4 is an example of a data structure of material data stored in a storage unit. [Figure 7] 4 is an example of a data structure of material data stored in a storage unit. [Figure 8] 4 is an example of a data structure of material data stored in a storage unit. [Figure 9] 4 is an example of a data structure of material data stored in a storage unit. [Figure 10] 11 is an example of information for identifying a conversion process. [Figure 11] 4 is a flowchart illustrating an example of processing performed by the virtual visual point image generating system according to the first embodiment. [Figure 12] 4 is an example of a data structure of material data stored in a storage unit. [Figure 13] 4 is an example of a data structure of material data stored in a storage unit. [Figure 14] 4 is an example of a data structure of material data stored in a storage unit. [Figure 15] 4 is an example of a data structure of material data stored in a storage unit. [Figure 16] 4 is an example of a data structure of material data stored in a storage unit. [Figure 17] FIG. 11 is a diagram illustrating an example of the configuration of a virtual viewpoint image generating system including an information processing device according to a second embodiment. [Figure 18] 10 is a flowchart illustrating an example of processing performed by the virtual visual point image generating system according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that the components described in the following embodiments are examples of the embodiments, and the present disclosure is not limited to them.

[0012] (First embodiment) In this embodiment, an information processing device that outputs data (hereinafter referred to as material data) used to generate a virtual viewpoint image based on a plurality of captured images obtained by capturing an object by a plurality of imaging devices to a display device will be described.

[0013] Fig. 1 is a block diagram showing an example of the hardware configuration of a computer applicable to the information processing device in this embodiment. The information processing device has a CPU 1601, a RAM 1602, a ROM 1603, an operation unit 1604, an output unit 1605, an external storage device 1606, an I / F 1607, and a bus 1608. The hardware configuration shown in Fig. 1 is applicable to any device included in a virtual viewpoint image generation system described later.

[0014] The CPU 1601 controls the entire computer using computer programs and data stored in the RAM 1602 and ROM 1603. The RAM 1602 has an area for temporarily storing computer programs and data loaded from the external storage device 806, data acquired from the outside via an I / F (interface) 1607, and the like. Furthermore, the RAM 1602 has a work area used when the CPU 1601 executes various processes. That is, the RAM 1602 can be allocated as a frame memory, for example, or can provide various other areas as appropriate.

[0015] The ROM 1603 stores setting data for the device, a boot program, etc. The operation unit 1604 is made up of input devices such as a keyboard, a mouse, and a joystick, and can input various instructions to the CPU 1601 based on user operations using these input devices. The output unit 1605 outputs the results of processing by the CPU 1601. The output unit 1605 is made up of, for example, a liquid crystal display.

[0016] The external storage device 1606 can be an information storage device such as a hard disk drive. The external storage device 1606 stores an OS (operating system) and computer programs for causing the CPU 1601 to realize the functions of each processing unit of the device in this embodiment. The computer programs and data stored in the external storage device 1606 are loaded into the RAM 802 as appropriate under the control of the CPU 1601, and become the subject of processing by the CPU 1601. Furthermore, the external storage device 1606 may store each image data to be processed, and various information used in processing.

[0017] The I / F 1607 can be connected to a network such as a LAN or the Internet, and other devices such as a projector or display device, and the computer can obtain and transmit various information via the I / F 1607. For example, when the information processing device 200 is connected to an external device by wire, a communication cable is connected to the communication I / F 1607. When the information processing device has a function of wirelessly communicating with an external device, the communication I / F 1607 is equipped with an antenna. The bus 1608 is a bus that connects the above-mentioned components.

[0018] At least one of the above-described components may be connected to the outside of the information processing device as another device. This also applies to any device included in the virtual viewpoint image generation system described later.

[0019] <Configuration of the virtual viewpoint image generation system> Next, the configuration of the virtual viewpoint image generating system in this embodiment will be described. The virtual viewpoint image in this embodiment is also called a free viewpoint image, but is not limited to an image corresponding to a viewpoint freely (arbitrarily) specified by the user, and for example, an image corresponding to a viewpoint selected by the user from a plurality of candidates is also included in the virtual viewpoint image. In this embodiment, the case where the virtual viewpoint is specified by a user operation will be mainly described, but the virtual viewpoint may be automatically specified based on the result of image analysis, etc. In addition, in this embodiment, the case where the virtual viewpoint image is a moving image will be mainly described, but the virtual viewpoint image may be a still image.

[0020] 2 is a diagram for explaining the configuration of the virtual viewpoint image generation system 1. The virtual viewpoint image generation system 1 captures images of a subject using a plurality of imaging devices installed in a facility such as a sports field (stadium) or a concert hall, and generates data (material data) for generating a virtual viewpoint image based on the obtained plurality of captured images. The virtual viewpoint image generation system 1 includes cameras 101a-z, an input unit 102, a storage unit 103, a material data generation unit 104, an information processing device 100, and terminals 109a-d. The functional configuration of the information processing device 100 will be described later.

[0021] The cameras 101a-z are arranged to surround a subject and capture images in synchronization. However, the number of cameras is not limited to that shown in FIG. 2. The cameras 101a-z are connected to a network and to an input unit 102. The cameras 101a-z capture images in synchronization. That is, frames of images captured by the cameras 101a-z are acquired at the same time. The acquired captured images are assigned time information related to the capture time and a frame number, and are transmitted to the input unit 102. The time information can be in any format. Each camera is assigned a camera ID, and the camera ID information is assigned to the captured images acquired by each camera. In the following description, the cameras 101a-z will be simply referred to as cameras 101 unless there is a particular reason to distinguish them.

[0022] The input unit 102 inputs image data obtained by imaging performed by the cameras 101a to 101z, and outputs the image data to the storage unit 103. The storage unit 103 is a storage unit that temporarily stores the input image data.

[0023] The material data generating unit 104 generates material data used to generate a virtual viewpoint image by using image data stored in the storage unit 103. Here, the material data in this embodiment refers to data for generating a virtual viewpoint image, and is generated based on a captured image. The material data is, for example, data of a foreground image and a background image extracted from a captured image, a model representing the shape of a subject in a three-dimensional space (hereinafter also referred to as three-dimensional shape data), and texture data for coloring the three-dimensional shape data. The material data generating unit 104 in this embodiment generates material data in a format preset in the virtual viewpoint image generating system 1.

[0024] The terminals 109a to d are display devices that acquire material data from the information processing device 100 described later, and generate and display a virtual viewpoint image based on the material data. The terminals 109a to d may be, for example, a PC, a smartphone, a tablet, etc. In the following description, the terminals 109a to d will be simply referred to as terminals 109 unless there is a particular reason to distinguish them.

[0025] Next, the functional configuration of the information processing device 100 included in the virtual viewpoint image generation system 1 will be described with reference to Fig. 2. The information processing device 100 includes an acquisition unit 105, a storage unit 106, a conversion unit 107, and a transmission / reception unit 108.

[0026] The acquisition unit 105 acquires material data and stores it in the storage unit 108. In this embodiment, the acquisition unit 105 acquires material data from the material data generation unit 104 in the virtual viewpoint image generation system 1. When saving the acquired material data, the acquisition unit 105 saves the material data in a predetermined format. For example, the material data is saved in a format that allows data to be read and written in association with time information, frame numbers, etc., by generating a table for data access to read each piece of data. The types of material data that the acquisition unit 105 can acquire and the format of the material data when saved will be described later.

[0027] The storage unit 106 stores the input material data. The storage unit 106 corresponds to the external storage device 1606 in Fig. 1, and is composed of a semiconductor memory, a magnetic recording device, etc. Data is written and read based on instructions from the acquisition unit 105, and the written data is output to the transmission / reception unit 108 in accordance with a read instruction.

[0028] The conversion unit 107 converts the format of the material data stored in the storage unit 106 into a format that can be processed by the terminal 109, which is the output destination of the material data. The conversion process will be described later. The transmission / reception unit 108 communicates with the terminal 109, and transmits and receives requests from the terminal and data. The terminals 109a to d acquire material data from the information processing device 100. The terminals 109a to d also accept an operation related to the designation of a virtual viewpoint by a user, and generate a virtual viewpoint image corresponding to the designated virtual viewpoint based on the material data. The terminals 109a to d also display the generated virtual viewpoint image. In this embodiment, the terminal 109a generates a virtual viewpoint image using a foreground image and point cloud data. The terminal 109b generates a virtual viewpoint image using colored point cloud data. The terminal 109c generates a virtual viewpoint image using a foreground image and a distance image. The terminal 109d generates a virtual viewpoint image using mesh data.

[0029] Here, as described above, the material data used when generating a virtual viewpoint image may differ depending on the method of generating the virtual viewpoint image. The factors that cause the material data used to differ include, for example, the method of generating a virtual viewpoint image adopted by the terminal and the type of software for generating a virtual viewpoint image used by the terminal. Another factor is that the format of the material data that the terminal can handle is different. Another factor is that the display capabilities and processing capabilities of the terminal are different, such as a virtual viewpoint image based on mesh data can be displayed but a virtual viewpoint image based on point cloud data cannot be displayed. Due to these factors, even if material data corresponding to a virtual viewpoint image that the terminal cannot display is output to the terminal, a problem may occur in which the terminal cannot generate a virtual viewpoint image. To solve this problem, the information processing device 100 in this embodiment converts the format of the acquired material data into a format that can be processed by the terminal 109, which is the output destination of the material data, and outputs the converted material data to the terminal 109.

[0030] <Examples of material data types and formats> The types of material data that can be acquired by the acquisition unit 105 of the information processing device 100 in this embodiment will be described below. Since the acquisition unit 105 in this embodiment acquires material data generated by the material data generation unit 104, the material data generated by the material data generation unit 104 and the method of generating the same will be described below. Note that the types of material data described below are merely examples, and a configuration in which material data other than the following is generated may also be used.

[0031] First, a case will be described in which the material data generating unit 104 generates point cloud data representing the three-dimensional shape of a subject and a foreground image as material data. Here, the foreground image is an image in which an area corresponding to an object (e.g., a player or a ball) is extracted from a captured image. For example, the material data generating unit 104 captures an image in which the subject is not captured as a reference image, and extracts the area corresponding to the subject by using the difference between the reference image and the input image data (captured image). Based on the result of extracting the subject area, the material data generating unit 104 also generates a silhouette image in which the pixel value of the area corresponding to the subject is set to 1 and the pixel value of the other areas is set to 0.

[0032] Furthermore, the material data generating unit 104 generates point cloud data using the generated silhouette image by using a shape-from-silhouette method. The generated point cloud data is data that represents the three-dimensional shape of the subject by a set of multiple points having coordinate information in three-dimensional space. Furthermore, the material data generating unit 104 obtains a circumscribing rectangle of the extracted subject area, cuts out an area in the captured image corresponding to the circumscribing rectangle, and generates this as a foreground image. The foreground image is used as texture data for coloring the three-dimensional shape data when generating a virtual viewpoint image. Note that the method of generating the point cloud data and the foreground image is not limited to the above-mentioned method, and any generation method can be used.

[0033] Next, a case will be described in which the material data generating unit 104 generates data including a point cloud representing the three-dimensional shape of a subject and information representing the color of the subject (hereinafter, colored point cloud data) as material data. The colored point cloud data can be generated using point cloud data generated by the above-mentioned method and the captured images or foreground images of each camera 101 acquired from the storage unit 103. For each point included in the point cloud data, the material data generating unit 104 identifies the color (pixel value) of the captured image acquired by the camera 101 that captures the three-dimensional position corresponding to the point. The material data generating unit 104 can generate colored point cloud data, which is point cloud data having color information, by associating the identified color with the point.

[0034] Next, a case will be described in which the material data generating unit 104 generates a distance image representing the distance from each camera 101 to a subject as material data. The material data generating unit 104 generates a distance image by calculating parallax between a plurality of captured images, for example, by a stereo matching method, and determining pixel values ​​of the distance image from the parallax data. Note that the method of generating the distance image is not limited to this. For example, the material data generating unit 104 may generate a distance image by using point cloud data generated by the above-mentioned method and determining the distance from the three-dimensional position of each point in the point cloud data to each camera 101. The material data generating unit 104 may also be configured to separately acquire a distance image using a distance camera using an infrared sensor or the like.

[0035] Next, a case will be described in which the material data generating unit 104 generates mesh data representing the three-dimensional shape of the subject as material data. The material data generating unit 104 generates mesh data, which is a collection of multiple polygons, using, for example, the method described in Patent Document 2. When using this method, distance image data generated by a streaking method is used as a depth map. Note that the method of generating mesh data is not limited to this. For example, the material data generating unit 104 may generate mesh data by converting point cloud data and colored point cloud data generated by the above-mentioned method.

[0036] Next, a case where the material data generating unit 104 generates billboard data that represents an overview of a subject will be described. The billboard data is data obtained by coloring a plate-shaped polygon using a foreground image. When generating billboard data, the material data generating unit 104 generates a plate-shaped polygon and a foreground image by the above-mentioned method.

[0037] Note that a configuration may be adopted in which part of the processing performed by the above-mentioned material data generation unit 104 is performed by another device, such as an imaging device and an information processing device 100 described below. For example, the processing of generating a silhouette image and / or a foreground image when point cloud data is generated may be performed by the camera 101. In this case, the camera 101 assigns various types of information, such as time information, frame number, and camera ID, to the silhouette image and / or foreground image generated based on the captured image, and transmits them to the input unit 102.

[0038] Next, an example of the format of the material data when the above-mentioned material data is stored will be described below. The material data generated by the material data generating unit 104 is acquired by the acquiring unit 105 and stored in the storing unit 106 in the format described below.

[0039] FIG. 3 shows an example of the format of the material data stored in the storage unit 106. The material data is stored as a sequence corresponding to one collection as shown in FIG. 3(a). A sequence can be generated for, for example, a period during which imaging was performed, an imaged sport, or even for each event or cut that occurred during a sport. The storage unit 106 manages data in sequence units. Each sequence includes a sequence header. As shown in FIG. 3(b), the sequence header stores a sequence header start code indicating the start of the sequence. Next, information about the entire sequence is stored. For example, the name of the sequence, the imaging location, the date and time when imaging started, time information indicating the time, etc., the frame rate, and the image size are stored. The name of the sequence includes information for identifying the sequence, such as the name of a sport or event.

[0040] In the sequence, each piece of material data is stored in a unit called a data set. The number of data sets (Number of Data Set) M is written in the sequence header. The following information is stored in units of data sets. In the data set unit information, an identification ID of the data set is given first. The identification ID is given by the storage unit 106 or an ID that is unique among all data sets. Next, the type of the data set is stored. In the example shown in FIG. 3(a), point cloud data, foreground image, colored point cloud data, range image data, and mesh data are included in the sequence as data sets, but it is not necessary for all of these data sets to be included in the sequence. In this embodiment, the material data acquired by the acquisition unit 105 is included in the sequence as a data set. For example, when the acquisition unit 105 acquires point cloud data and foreground image data, the sequence includes data sets of the point cloud data and the foreground image data. The data sets included in the sequence are each expressed by a data set class code. The data set class code is expressed as a 2-byte code shown in FIG. 3(e). However, the type and code of data are not limited to this. Data representing other three-dimensional shape data may be used. Next, a pointer to the data set is stored. However, any information for accessing each data set is sufficient, and is not limited to a pointer. For example, a file system may be constructed in the storage unit, and the file name may be used.

[0041] Next, the format of each data set included in the sequence will be described. In this embodiment, the types of data sets will be described as point cloud data, foreground image, colored point cloud data, range image data, mesh data, and billboard data.

[0042] Fig. 4 shows an example of the configuration of a foreground image dataset. For the sake of explanation, it is assumed that the foreground image dataset is stored on a frame-by-frame basis, but this is not limiting. As shown in Fig. 4(a), a foreground image data header is stored at the beginning of the dataset. The header stores information that this dataset is a foreground image dataset, the number of frames, etc.

[0043] The information included in each frame is shown in FIG. 4(b). Each frame stores time information (Time information) indicating the time of the first frame of the foreground image data, and the data size of the frame (Data Size). The data size is for referencing the data of the next frame, and may be stored together in the header. Next, the number of objects (Number of Objects) P ​​for generating a virtual viewpoint image at the time indicated by the time information is stored. Furthermore, the number of cameras (Number of Cameras) C used for imaging at that time is stored. Next, the camera ID (Camera ID of Cth Camera) of the camera used is stored. Next, image data of the foreground image in which the object is reflected (1st Foreground Image of Cth Camera) is stored for each object. At the beginning of the image data, as shown in FIG. 4(c), the data size of the foreground image, the size of the foreground image, the bit depth of the pixel value, and the pixel value are stored in raster order. Thereafter, foreground image data from each camera is stored for each object. Note that if the object is not reflected in the camera, NULL data may be written, or the number of cameras reflected in each object and the corresponding camera ID may be stored.

[0044] FIG. 5 shows an example of the configuration of a point cloud data dataset. For the sake of explanation, it is assumed that the point cloud dataset is stored on a frame-by-frame basis, but this is not limiting. A point cloud data header is stored at the beginning of the dataset. As shown in FIG. 5(a), the header stores information such as that the dataset is a point cloud data dataset and the number of frames.

[0045] The information contained in each frame is shown in FIG. 5(b). Time information indicating the time of the frame is stored in each frame. Next, the data size of the frame is stored. This is for referencing the data of the next frame, and may be stored together in the header. Next, the number of objects at the time indicated by the time information, i.e., the number of point cloud data (Number of Objects) P, is stored. Thereafter, the point cloud data for each object is stored in order. First, the number of coordinate points constituting the point cloud of the first object (Number of Points in 1st Object) is stored. Thereafter, the coordinates of the points (Point coordination in 1st Object) are stored. In the same manner, the point cloud data of all objects included in the time corresponding to the time information is stored.

[0046] In this embodiment, the coordinate system is stored as three-axis data, but this is not limited to this, and polar coordinates or other coordinate systems may be used. The data length of the coordinate values ​​may be written in the point cloud data header as fixed, or the data length may be different for each point cloud. If the data length differs for each point cloud, the data size is stored for each point cloud. By referring to the data size, the storage position of the next point cloud can be specified from the number of coordinate points.

[0047] FIG. 6 shows an example of the configuration of a colored point cloud data dataset. For the sake of explanation, it is assumed that the colored point cloud dataset is stored on a frame-by-frame basis, but this is not limiting. A colored point cloud data header is stored at the beginning of the dataset. As shown in FIG. 6(a), the header stores information such as that the dataset is a colored point cloud data dataset and the number of frames.

[0048] The information included in each frame is shown in FIG. 6(b). In each frame, the point cloud data of each frame stores time information (Time information) that indicates the time of the frame. Next, the data size of the frame (Data Size) is stored. This is for referencing the data of the next frame, and may be stored together in the header. Next, the number of objects at the time indicated by the time information, i.e., the number of colored point cloud data (Number of Objects) P, is stored. Thereafter, the colored point cloud data for each object is stored in order. First, the number of coordinate points that make up the point cloud of the first object (Number of Points in 1st Object) is stored. Thereafter, the coordinates of the points and the color information of the points (Point coordination in 1st Object) are stored. In the same manner, the colored point cloud data of all objects included in the time corresponding to the time information is stored.

[0049] In this embodiment, the coordinate system is three-axis data, and color information is stored as values ​​of the three primary colors of RGB, but is not limited to this. The coordinate system may be polar coordinates or other coordinate systems. Furthermore, color information may be expressed as information such as a uniform color space, brightness, and chromaticity. Furthermore, the data length of the coordinate values ​​may be fixed and described in the colored point cloud data header, or the data length may be different for each colored point cloud. If the data length differs for each colored point cloud, the data size is stored for each point cloud. By referring to the data size, the storage position of the next colored point cloud can be identified from the number of coordinate points.

[0050] Fig. 7 shows an example of the configuration of a data set of range image data. For the sake of explanation, it is assumed that the range image data set is stored on a frame-by-frame basis, but this is not limiting. A range image data header is stored at the beginning of the data set. As shown in Fig. 7(a), the header stores information such as that this data set is a data set of range image data, the number of frames, etc.

[0051] The information included in each frame is shown in Fig. 7(b). Time information (Time information) that indicates the time of the frame is stored in each frame. Next, the data size of the frame (Data Size) is stored. This is for referencing the data of the next frame, and may be stored together in the header. Since the distance image data is acquired on a camera-by-camera basis, the number of cameras (Number of Cameras) C used in the frame is stored. Next, the camera ID (Camera ID of Cth Camera) is stored in order. Next, the distance image data (Distance Image of Cth Camera) is stored. At the beginning of the distance image data, as shown in Fig. 7(c), the data size of the distance image, the size of the distance image, the bit depth of the pixel value, and the pixel value are stored in raster order. The distance image data from each camera is stored successively. If the camera does not capture the subject, NULL data may be written, or only the number of cameras C that capture the subject and the corresponding camera ID may be stored.

[0052] FIG. 8 shows an example of the configuration of a mesh data set. For the sake of explanation, it is assumed that the mesh data set is stored on a frame-by-frame basis, but this is not limiting. A mesh data header is stored at the beginning of the data set. As shown in FIG. 8(a), the header stores information such as that the data set is a mesh data set, the number of frames, etc.

[0053] FIG. 8B shows information included in each frame. Time information (Time information) indicating the time of the frame is stored in each frame. Next, the data size (Data Size) of the frame is stored. This is for referring to the data of the next frame, and may be stored together in the header. Next, the number of objects (Number of Objects) P ​​is stored. Mesh data for each object is stored below. The mesh data for each object stores the number of polygons (Number of Points in Pth Object) that constitute the mesh data at the beginning. Furthermore, the mesh data for each object stores data for each polygon, that is, the coordinates of the vertices of the polygon and the color information of the polygon (Polygon information in Pth Object). In the same manner, mesh data for all objects included in the time corresponding to the time information is stored below. Note that the above-mentioned mesh data is mesh data that assigns one color to one polygon, but may be mesh data that does not include color information. In the following description, the mesh data shown in FIG. 8 is referred to as colored mesh data, and mesh data that does not include color information is simply referred to as mesh data. When a virtual viewpoint image is generated using colored mesh data, the color of the polygon is determined regardless of the position of the virtual viewpoint and the line of sight from the virtual viewpoint. On the other hand, when a virtual viewpoint image is generated using mesh data that does not include color information, the polygon is colored using a foreground image according to the position of the virtual viewpoint and the line of sight from the virtual viewpoint.

[0054] FIG. 9 shows an example of the configuration of a billboard data dataset. For the sake of explanation, it is assumed that the billboard dataset is stored on a frame-by-frame basis, but this is not limiting. At the beginning of the dataset, a billboard data header is stored as shown in FIG. 9(a). The header stores information such as that this dataset is a billboard data dataset and the number of frames. Time information indicating the time of the frame is stored in the billboard data of each frame. Next, the data size of the frame is stored. Below, billboard data for each camera is stored for each subject.

[0055] Billboard data is saved for each subject as a combination of billboard polygon data for each camera and image data to be texture-mapped. First, one polygon data of the billboard (Polygon billboard of Cth Camera for Pth Object) is recorded. Next, foreground image data for texture mapping (Foreground Image of Cth Camera for Pth Object) is saved. The billboard method can significantly reduce processing and data volume by treating the subject as a single polygon. Note that the number of polygons representing billboard data is not limited to one, and is assumed to be fewer than the number of polygons in mesh data.

[0056] In this embodiment, the coordinate system describing the vertices is three-axis data, and color information is saved as the values ​​of the three primary colors of RGB, but this is not limited to this. The coordinate system may be polar coordinates or other coordinate systems. Furthermore, color information may be expressed as information such as a uniform color space, luminance, and chromaticity. Furthermore, existing formats such as the PLY format and STL format may be used. Support is possible by preparing a dataset class code corresponding to each format.

[0057] <Example 1 of conversion processing performed by the conversion unit> Next, the conversion process performed by the conversion unit 107 will be described. The conversion unit 107 performs a process of converting the material data acquired by the acquisition unit 105 and stored in the storage unit 106 in a predetermined format into a format that can be processed by the output destination device. As described above, since there are a plurality of methods for generating a virtual viewpoint image, the material data used may differ depending on the generation method. On the other hand, in a system that generates material data, only material data in a predetermined format may be generated. In this case, if the material data generated by the system is in a format that cannot be processed by a device that performs processing related to the generation of the virtual viewpoint image, a problem such as a virtual viewpoint image not being generated may occur. In order to solve this problem, the material data can be converted into a processable format via the conversion unit 107, so that the output destination device of the material data can perform appropriate processing.

[0058] The conversion unit 107 first identifies material data that can be processed by a device to which the material data is to be output. The transmission / reception unit 108 acquires information for identifying the format of material data that the terminal 109 can process from the terminal 109 connected to the information processing device 100. Here, the fact that the terminal 109 can process material data means that the terminal 109 can interpret data in the file of the acquired material data and process it appropriately according to the contents described. For example, a file format of material data that the terminal 109 does not support and cannot read is not processable.

[0059] The acquired information may include, for example, the specifications of the terminal, information about the type, information about the processing capacity of the terminal, etc. Furthermore, for example, the acquired information may include information about a method for generating a virtual viewpoint image used by the terminal, an application and software used for generating and playing the virtual viewpoint image, etc. Furthermore, for example, the acquired information may include information about a format of a virtual viewpoint image that can be displayed by the terminal, etc. The transmission / reception unit 108 transmits the acquired information to the conversion unit 107. The conversion unit 107 specifies the format of the material data to be output to the terminal 109 based on the acquired information.

[0060] Furthermore, conversion unit 107 specifies the format of the material data stored in storage unit 106. Conversion unit 107 specifies the type of material data by, for example, referring to the data set class code of the material data stored in storage unit 106 and the header of the data set. If the format and type of material data acquired by acquisition unit 105 are predetermined, it is not necessary to refer to the data set. If the format of the material data to be output to terminal 109 does not match the format of the material data stored in storage unit 106, conversion unit 107 performs a conversion process of the material data stored in storage unit 106. Transmitting / receiving unit 108 outputs the material data obtained by the conversion process to the output destination device.

[0061] As an example of the conversion process, a case where only the foreground image dataset and the point cloud dataset are stored in the storage unit 106 will be described. Here, it is assumed that the terminal 109c is connected to the information processing device 100 and the output of a distance image is requested from the terminal 109c. In this case, the conversion unit 107 compares the request for the dataset from the terminal 109c input from the transmission / reception unit 108 with the format of the material data stored in the storage unit 309. As a result of the comparison, it is determined that the material data of a suitable format is not stored. The conversion unit 107 acquires the point cloud dataset from the storage unit 106, obtains the distance from each point of the point cloud data to each camera, generates distance image data, and converts it into a distance image dataset. That is, the point cloud dataset of FIG. 5 is converted into the distance image dataset of FIG. 7. The transmission / reception unit 108 outputs the distance image data obtained by the conversion process to the terminal 109.

[0062] In this way, conversion unit 107 converts the material data stored in storage unit 106 into a predetermined format in response to a request from terminal 109. Note that if the format of the material data requested by terminal 109 matches the format of the material data stored in storage unit 106, no conversion process is performed. In this case, for example, conversion unit 107 may acquire the material data stored in storage unit 106 and transmit it to transmission / reception unit 108 without conversion, or may transmit it directly from storage unit 106 to transmission / reception unit 108.

[0063] The conversion unit 107 performs the conversion process as follows, by the same process as the above-mentioned process flow. For example, when the point cloud data and foreground image data are stored in the storage unit 106, the data can be converted into colored point cloud data, distance image, mesh data and foreground image, colored mesh data, and billboard data in response to a request from the terminal 109. Also, for example, when the colored point cloud data is stored in the storage unit 106, the data can be converted into colored mesh data, distance image, and billboard data in response to a request from the terminal 109. Also, for example, when the mesh data and foreground image data are stored in the storage unit 106, the data can be converted into point cloud data and foreground image data, colored point cloud data, colored mesh data, distance image, and billboard data in response to a request from the terminal 109. Also, for example, when the colored mesh data is stored in the storage unit 106, the data can be converted into colored point cloud data, distance image, and billboard data in response to a request from the terminal 109.

[0064] Furthermore, for example, when depth image data and a foreground image are stored in storage unit 106, the data can be converted into billboard data in response to a request from terminal 109. Furthermore, for example, when billboard data and foreground image data are stored in storage unit 106, the data can be converted into point cloud data and foreground image data, colored point cloud data, mesh data and foreground image, colored mesh data, and depth image. When converting billboard data into point cloud data or mesh data representing a three-dimensional shape, the conversion is performed, for example, by executing a volume intersection method using the billboard data.

[0065] In the above example, the conversion unit 107 performs the conversion process of the material data based on the information acquired from the terminal 109, but the present invention is not limited to this. When the terminal to which the material data is to be output is known in advance, the conversion unit 107 can convert the material data to be output to the terminal in advance. For example, it is assumed that the point cloud data and the foreground image data are stored in the storage unit 106, and the terminals 109a, b, and d are connected to the information processing device 100. The terminal 109a generates a virtual viewpoint image using the foreground image and the point cloud data. The terminal 109b generates a virtual viewpoint image using the colored point cloud data. The terminal 109d generates a virtual viewpoint image using the colored mesh data. Here, the conversion unit 107 refers to the table shown in FIG. 10(a) and determines what to convert the point cloud data and the foreground image data stored in the storage unit 106 into. The conversion flag in FIG. 10(a) indicates that conversion is performed with "1" and that conversion is not performed with "0". Conversion unit 107 converts the point cloud and foreground image data into colored point cloud data and colored mesh data based on the value of the conversion flag. Transmission / reception unit 108 outputs the colored point cloud data and colored mesh data generated by the conversion process of conversion unit 107 to terminals 109b and 109d, respectively. Transmission / reception unit 108 also outputs the point cloud data and foreground image data stored in storage unit 106 to terminal 109a.

[0066] With the configuration described above, even if the type of material data acquired by acquisition unit 105 is different from the type of material data requested by terminal 109, it is possible to convert the data into an appropriate format and output the data. Note that, although the above example describes an example in which point cloud data and foreground image data are acquired, the same conversion process is performed when other material data is acquired.

[0067] <Example 2 of conversion processing performed by the conversion unit> Next, another example of the conversion process performed by the conversion unit 107 will be described. The conversion unit 107 can convert not only the type of material data but also the content of the material data. Here, as the content to be converted, the resolution and frame rate, which are information specifying the data amount of the material data, will be described. Furthermore, as the content to be converted, the format will also be described.

[0068] First, the conversion process of the resolution of the material data will be described. For example, when point cloud data is stored in the storage unit 106, the conversion unit 107 can change the resolution of the point cloud data. The resolution here refers to the density of a plurality of points constituting the point cloud data. The density of the plurality of points corresponds to the number of points used to express the three-dimensional shape of the subject. The higher the density of the plurality of points, the more detailed the three-dimensional shape can be expressed, and therefore the higher the resolution. On the other hand, the lower the density of the plurality of points, the lower the resolution. The conversion unit 107 performs conversion processing to increase the density of the points constituting the point cloud data (to increase the resolution) or to decrease the density (to decrease the resolution) according to information acquired from the terminal 109. When increasing the density, the conversion unit 107 performs, for example, a process of interpolating between a plurality of points. Also, when increasing the density, the conversion unit 107 performs a process of thinning out the points. The point interpolation process and the point thinning process may be performed uniformly on the point cloud data, or may be performed by any method. For example, a point interpolation process and a point thinning process may be performed on the face of a player so that the point density is higher than that of the body of the player. Furthermore, when the conversion unit 107 performs the interpolation or thinning of points of the point cloud data, the conversion unit 107 changes the contents of the point cloud data stored in the storage unit 106. In this case, the conversion unit 107 adds and deletes information on the coordinates of the points in FIG. 5(b).

[0069] The above-mentioned method of converting the resolution of point cloud data can also be applied to colored point cloud data. A case will be described in which the material data stored in the storage unit 106 is voxel data, which is a set of voxels having a predetermined volume. The resolution of the voxel data corresponds to the size of each voxel (voxel size) that represents a three-dimensional shape. The smaller the voxel size, the more detailed the three-dimensional shape can be expressed, and therefore the higher the resolution. On the other hand, the larger the voxel size, the lower the resolution. The conversion unit 107 performs conversion processing to make the size of the voxels that make up the voxel data smaller (higher the resolution) or larger (lower the resolution) according to information acquired from the terminal 109.

[0070] A case where the material data stored in the storage unit 106 is mesh data will be described. The resolution of the mesh data corresponds to the size of the polygons (polygon size) constituting the mesh data. The smaller the polygon size, the finer the three-dimensional shape can be expressed, and therefore the higher the resolution. On the other hand, the larger the polygon size, the lower the resolution. The conversion unit 107 performs conversion processing to make the size of the polygons constituting the mesh data smaller (to increase the resolution) or smaller (to decrease the resolution) according to the information acquired from the terminal 109. The method of converting the size of the polygon is, for example, by processing such as increasing the number of vertices of the polygon (increasing the number of polygons) or decreasing the number of vertices (decreasing the number of polygons). Furthermore, when the conversion unit 107 changes the number of vertices of the polygon, the content of the polygon information shown in FIG. 8(b) is changed according to the change processing. The above-described conversion processing of the resolution of the mesh data can also be applied to colored mesh data.

[0071] Also, the conversion unit 107 converts the resolution of the foreground image. The conversion unit 107 can perform a process of increasing pixel value information (increasing the resolution) by, for example, super-resolution processing, or reducing pixel value information (reducing the resolution) by replacing an average value of a plurality of pixel values ​​with one pixel value. The conversion of the resolution of the foreground image is performed in accordance with material data, such as point cloud data or mesh data, that is colored using the foreground image. For example, when the conversion unit 107 converts the resolution of the point cloud data or mesh data to be lower, the conversion unit 107 also changes the resolution of the foreground image to be lower. For example, when the conversion unit 107 converts the resolution of the point cloud data or mesh data to be higher, the conversion unit 107 also changes the resolution of the foreground image to be higher. However, the present invention is not limited to this, and for example, even if the resolution of the point cloud data or mesh data is lowered, the resolution of the foreground image may not be changed. By doing so, although the three-dimensional shape is roughly expressed, the color becomes close to the color of the actual subject, and there is an effect that the deterioration of the image quality of the virtual viewpoint image is suppressed while reducing the amount of data.

[0072] The conversion unit 107 may also perform a process of converting the precision of coordinate values ​​of points in the point cloud data, coordinate values ​​of vertices of meshes in the mesh data, and distance values ​​in the distance image data. That is, the conversion unit 107 can perform a process of changing the precision of the data by converting these values ​​into floating point, fixed point, or integer. The conversion unit 107 may also perform a conversion process of reducing the bit depth of pixel values ​​of the foreground image by quantization or the like, and increasing the bit depth by interpolation or the like.

[0073] Next, a conversion process of the frame rate of the material data will be described. The conversion unit 107 can perform a conversion process to increase or decrease the frame rate of the material data stored in the storage unit 106. For example, a case where the frame rate of the material data stored in the storage unit 106 is 60 fps will be described. When the conversion unit 107 increases the frame rate, for example, it generates frames by interpolation to set the frame rate to 120 fps. When the conversion unit 107 decreases the frame rate, for example, it performs a process of thinning frames to set the frame rate to 30 fps. Note that the value of the frame rate after the change and the method of changing the frame rate are not limited to the above. For example, the frame rate may be changed only in a specific range in the sequence of the data set. Note that the change of the frame rate can be applied to any type of material data, such as foreground image data, point cloud data, mesh data, distance image data, and billboard data.

[0074] Next, a process of converting the format of material data will be described. The format here refers to, for example, a file extension, a data structure, and the like. When the file extension of the material data that can be processed by the terminal 109 is different from the file extension of the material data stored in the storage unit 106, the conversion unit 107 performs a process of converting the file format. This process is applicable to any type of material data. In addition, for example, the conversion unit 107 may perform a process of changing the shape of a polygon constituting the mesh data stored in the storage unit 106. For example, when the shape of a polygon of the mesh data stored in the storage unit 106 is a triangle and the shape of a polygon of the mesh data that can be processed by the terminal 109 is a hexagon, the conversion unit 107 performs a process of converting the shape of the polygon from a triangle to a hexagon. In this way, the conversion unit 107 can change the shape of the polygon to any polygon.

[0075] In the above-described example, the conversion process of the resolution, frame rate, and format of the material data is performed based on the information acquired from the terminal 109, but this is not limited to the above. When the terminal to which the material data is to be output is known in advance, the conversion unit 107 can convert the material data to be output to the terminal in advance. In this case, the conversion unit 107 performs the conversion process of the material data by referring to a table such as that shown in FIG. 10(b), for example. In the example of FIG. 10(b), the conversion unit 107 converts the resolution of the point cloud data stored in the storage unit 106 to be lower, and converts the frame rate to be smaller.

[0076] By converting information for specifying the amount of data, such as the resolution and frame rate of the material data, it becomes possible to output material data according to, for example, the processing capability and display capability of the terminal. For example, if the resolution of the material data is equivalent to 4K and the terminal can display 8K video, the conversion unit 107 converts the material data to have a higher resolution. Conversely, if the terminal can only display 2K video, the conversion unit 107 converts the material data to have a lower resolution.

[0077] <Operation of the information processing device> Next, the operations of the information processing device 100 and the terminal 109 that acquires material data from the information processing device will be described with reference to the flowchart of Fig. 11. The processing of Fig. 11 is realized by the CPU of each device in the virtual viewpoint image generation system 1 reading and executing a program stored in a ROM or an external storage device. When the power supply of the information processing device 100 is turned on, the processing is started.

[0078] In S1100, the acquisition unit 105 acquires material data from another device. Here, the acquisition unit 105 acquires material data generated by the material data generation unit 104. In S1101, the acquisition unit 105 stores the acquired material data in the storage unit 106 in the above-mentioned format. In S1102, the transmission / reception unit 108 receives a request for outputting material data from the terminal 109 connected to the information processing device 100. At this time, the transmission / reception unit 108 acquires information for identifying the format of material data that the terminal 109 can process from the terminal 109. Note that the information acquired by the transmission / reception unit 108 from the terminal is not limited to the above-mentioned information, and may be information specifying specific material data such as point cloud data. In addition, the information processing device 100 may be configured to hold in advance in an external storage device or the like a table in which the types of multiple terminals are associated with the material data to be output to each terminal, and the transmission / reception unit 108 acquires this table. The transmission / reception unit 108 transmits the acquired information to the conversion unit 107.

[0079] In addition, in S1102, the transmitting / receiving unit 108 acquires information for identifying a sequence of the material data. The information for identifying the sequence may be, for example, a sequence name. The transmitting / receiving unit 108 transmits the acquired information such as the sequence name to the conversion unit 107, and instructs the conversion unit 107 to acquire the corresponding sequence.

[0080] In S1103, conversion unit 107 determines whether conversion processing of the material data is necessary based on the format of the material data stored in storage unit 106 and the information acquired from transmission / reception unit 108. If the format of the material data to be output to terminal 109, which is specified based on the information acquired from transmission / reception unit 108, does not match the format of the material data stored in storage unit 106, conversion unit 107 determines that conversion processing is necessary (Yes in S1103). Then, in S1104, conversion unit 107 performs conversion processing of the material data stored in storage unit 106. On the other hand, if the format of the material data to be output to terminal 109 matches the format of the material data stored in storage unit 106, conversion unit 107 determines that conversion processing is unnecessary (No in S1103) and skips S1104.

[0081] In S1105, the transmission / reception unit 108 outputs the material data acquired from the conversion unit 107 to the terminal 109. In S1106, the terminal 109 acquires the material data and executes the processes of S1107 and S1108. The processes of S1107 and S1108 are repeatedly performed for each frame of the virtual viewpoint image to be generated.

[0082] In S1107, the terminal 109 determines the position of the virtual viewpoint and the line of sight from the virtual viewpoint based on a user operation or the like. In S1108, the terminal 109 generates a virtual viewpoint image based on the material data acquired from the information processing device 100 and the determined position of the virtual viewpoint and the line of sight from the virtual viewpoint. In a terminal that generates a virtual viewpoint video from foreground image data and point cloud data such as the terminal 109a, the position of a point of the point cloud data in the virtual viewpoint image is first specified. Also, a virtual viewpoint image is generated by coloring a pixel value corresponding to the position of the point by projection of the foreground image data. In a method of generating a virtual viewpoint image from the foreground image data and the point cloud data, the color with which the point is colored changes depending on the position of the virtual viewpoint and the line of sight from the virtual viewpoint. In a terminal that generates a virtual viewpoint image from colored point cloud data such as the terminal 109b, the position of a point of the point cloud data in the virtual viewpoint video is also first specified. A virtual viewpoint image is generated by coloring a pixel value corresponding to the position of the specified point with a color associated with the point. In a method for generating a virtual viewpoint image using a colored point cloud, the color corresponding to each point is fixed, so that the points are colored in a predetermined color regardless of the position of the virtual viewpoint and the line of sight from the virtual viewpoint.

[0083] In a terminal such as terminal 109c that generates a virtual viewpoint image from distance image data and foreground image data, the position of an object on the virtual viewpoint image is identified from a plurality of distance image data, and the virtual viewpoint image is generated by texture mapping using the foreground image. In a terminal such as terminal 109d that generates a virtual viewpoint image from mesh data, the virtual viewpoint image is generated by pasting color values ​​and images onto a surface seen from the virtual viewpoint, as in normal computer graphics.

[0084] In S1109, the terminal 109 displays the generated virtual viewpoint image on the display unit. In S1110, the terminal 109 determines whether to end the repetitive process. For example, when the generation of the virtual viewpoint image is ended by a user instruction or the like, the repetitive process ends.

[0085] With the above configuration and operation, the information processing device 100 performs conversion processing based on information for identifying the format of material data that the terminal 109 can process, and outputs the obtained material data to the terminal 109. As a result, even if the format of the material data held by the information processing device 100 is different from the format of the material data that the terminal can process, the terminal can output the material data that the terminal can process. Note that, in the process shown in FIG. 11, after the conversion process is performed in S1104, processing for each frame of the virtual viewpoint image is performed, but this is not limited to this. That is, the information processing device 100 may read out the material data stored in the storage unit 108 according to the determined virtual viewpoint, perform the conversion process, and output the material data to the terminal 109. In this case, for example, the processes from S1102 to S1104 in FIG. 11 are performed between S1107 and S1108.

[0086] The information processing device 100 may be configured to store the converted material data acquired by the conversion unit 107 in the storage unit 106. At this time, the material data generated by the conversion process may be stored in the same storage unit 106, or may be stored in different storage units for each type of material data. Furthermore, when the material data is stored in different storage units, the information processing device 100 may have a configuration having a plurality of storage units, or each storage unit may be included in the virtual viewpoint image generation system 1 as a different device. When a plurality of types of material data are stored in different storage units, the information processing device 100 is configured to output information such as a pointer indicating the location where the material data to be output to the terminal 109 is stored to the terminal 109. This allows the terminal 109 to access the storage unit in which the material data required for generating the virtual viewpoint image is stored, and to acquire the material data. (Modification of the First Embodiment) In the first embodiment, a configuration has been described in which material data is generated in the virtual viewpoint image generation system 1 and acquired by the acquisition unit 105 in the information processing device, but the present invention is not limited to this. The information processing device 100 can acquire material data generated in another system or acquire material data from a plurality of devices, for example. Then, the information processing device 100 can convert the acquired material data into a format suited to the terminal 109 and output it to the terminal 109.

[0087] In the first embodiment, the foreground image is divided into each object and stored, but the present invention is not limited to this, and the image captured by the camera may be stored. In the first embodiment, the point cloud data is stored in units of frames, but the present invention is not limited to this. For example, it is possible to identify each object and store the object in units of objects.

[0088] In the first embodiment, information for identifying the data is included in the header of the sequence in order to identify the three-dimensional shape data, but this is not limiting and the data may be managed in a list associated with the data. The material data stored in the storage unit 106 may be encoded. In this case, the acquisition unit 105 may be configured to encode the material data when storing it in the storage unit 106, or the material data generation unit 104 may be configured to acquire the material data that has been encoded.

[0089] In the first embodiment, the foreground image data and the point cloud data are stored as separate data sets, but the present invention is not limited to this. FIG. 12 shows an example of the configuration of a foreground image and point cloud data set in which the foreground image data and the point cloud data are integrated. In the format shown in FIG. 12, a foreground image and point cloud data header (Foreground & Point Cloud Model Header) is stored at the beginning of the data set, similar to the foreground image data set in FIG. 4. In this data header, information that this data set is a foreground image and point cloud data data set, the number of frames, etc. are stored. Next, the foreground image data and point cloud data of each frame are stored. In the foreground image and point cloud data of each frame, time information (Time information) indicating the time of the frame and the data size (Data Size) of the frame are stored. Next, the number of objects (Number of Objects) P ​​corresponding to the time indicated by the time information is stored. Furthermore, the number of cameras (Number of Cameras) C used for imaging at that time is stored. Next, the camera ID (Camera ID of Cth Camera) of the camera used is stored. Next, the data of each object is stored. First, the number of coordinate points that make up the point cloud of the subject is saved, similar to the point cloud data in Figure 5. The coordinates of the points are then saved. Next, the foreground image data from each camera that captured the subject is stored in order, similar to Figure 4. From then on, point cloud data and foreground image data are saved for each subject.

[0090] By configuring as described above, the data set of the required combination can be integrated and saved by the terminal that is the renderer. This makes it possible to simplify the reading of data. Note that, although the format in which the point cloud data and the foreground image data are integrated has been described here, the format is not limited to this. For example, it is possible to configure a format in which any data is combined and integrated, such as a format in which a distance image and a foreground image are integrated, or a format in which three or more pieces of material data are integrated. In addition, the conversion unit 107 is also capable of conversion such as integration of material data as described here and separation of integrated material data.

[0091] Alternatively, the frame data of the point cloud data and the frame data of the foreground image data may simply be interleaved and stored alternately as shown in Fig. 16. Similarly, distance image data may be stored instead of the point cloud data. The conversion unit 107 may also convert the material data into the data structure shown in Fig. 16.

[0092] Moreover, material data may be managed on an object-by-object basis, as shown in FIG.

[0093] An object data set is defined as a new data set for the sequence. Note that there may be multiple object data sets. For example, object data sets may be created for each team to which the subject belongs.

[0094] The object data header at the beginning of the object data set stores an object data header start code (Object Data Header) indicating the beginning of the object data set. Furthermore, the object data header stores a data size (Data size). Next, the number of objects (Number of Objects) P ​​included in the object data set is stored. The following information on the data for each object is stored.

[0095] The object data information (Pth Object Data Description) stores the ID number (Object ID) of the object and the size of the object data information (Data Size). Next, a data set class code (Data set class code) indicating the type of data as a data set is stored. Also, a pointer to the object data of the object is stored. Also, metadata (Meta data) related to the object is stored. For example, in the case of sports, information such as the names of players, their teams, and uniform numbers can be stored as metadata.

[0096] Data based on the data set class code is stored in each object data. FIG. 13 is an example of a data set that integrates a foreground image and point cloud data. The size of the object data set, the time information of the first frame, the frame rate, etc. are stored as a header at the beginning. Data is stored frame by frame. In the data by frame, the time information of the frame (Time information) and the data size of the frame (Data size) are stored at the beginning. Next, the size of the point cloud data in the frame of the subject (Data Size of Point Cloud) and the number of points (Number of Points) are stored. Next, the coordinates of the points of the point cloud (Point coordination) are stored. Next, the number of cameras (Number of Cameras) C that captured the subject is stored. Next, the camera IDs (Camera ID of 1st to Cth Camera) and foreground image data (Data Size of Foreground image of 1st to Cth Camera) of each camera are stored. Next, the camera ID and foreground image data are stored for each camera.

[0097] If the dataset of the object data is colored point cloud data, a dataset class code is set as shown in Fig. 14(a), and time information, data size, number of points in the colored point cloud, and data of the colored point cloud are stored as data for each frame. If the dataset of the object data is mesh data, a dataset class code is set as shown in Fig. 14(b), and time information, data size, number of polygons, and data of each polygon are stored as data for each frame. If the dataset of the object data is distance images, a dataset class code is set as shown in Fig. 14(c), and time information, data size, number of cameras, and data of each distance image are stored as data for each frame.

[0098] In this way, by managing and outputting data for each subject, it is possible to specify a subject such as a player or performer, and shorten the time it takes to read out an image centered on the specified subject. This is effective when generating an image centered on the movement of a specified player, or a virtual viewpoint image that follows behind the subject. It also becomes possible to easily search for a specified player. Furthermore, the conversion unit 106 can convert from a format in which data is managed on a frame-by-frame basis as shown in Figs. 4 to 9, to a format in which data is managed for each subject as shown in Figs. 13 and 14. Of course, it is also possible to convert from the format in Figs. 13 and 14 to the format in Figs. 4 to 9.

[0099] The acquisition unit 105 may configure a sequence for each type of material data as shown in FIG. 15. For example, when the material data acquired by the acquisition unit 105 is foreground image data and point cloud data, a sequence is generated for each of the foreground image data and point cloud data as shown in FIG. 15, and stored in the storage unit 106. According to this configuration, the sequences can be output together in accordance with the rendering method used by the output destination terminal, so that the sequences can be output quickly. The conversion unit 107 can convert the format in which the material data is collected into one sequence as shown in FIG. 3 into a sequence divided for each piece of material data as shown in FIG. 15. Of course, it is also possible to convert from the format of FIG. 15 to the format of FIG. 3. The conversion unit 107 may be configured to perform a process of adding predetermined information to the header of the material data, etc., so that the output destination device can process the material data. Conversely, the conversion unit 107 may be configured to perform a process of deleting information such as metadata that is unnecessary for the output destination device.

[0100] In addition, in the present embodiment, the information processing device 100 has been described as being a device different from the material data generation unit 104 and the terminal 109. However, the present invention is not limited to this. In other words, the information processing device 100 may be configured to be integrated with a device that generates material data or a device that acquires material data.

[0101] A case will be described where the functions of information processing device 100 are included in a material data generating device including material data generating unit 104. In this case, acquisition unit 105 and conversion unit 107 can specify in advance the type of material data to be generated. Therefore, conversion unit 107 performs conversion processing of the material data by referring to information for specifying material data that terminal 109 can process, which is acquired from terminal 109, and the table shown in Fig. 10, etc.

[0102] On the other hand, a case will be described where the functions of the information processing device 100 are included in a device that acquires material data and performs processing related to generating a virtual viewpoint image. For example, it is assumed that the functions of the information processing device 100 are included in a terminal 109. In this case, the conversion unit 107 can specify in advance the format of material data that the terminal can process. Therefore, the conversion unit 107 converts the format of material data acquired from another device into the format of material data specified in advance. At this time, information specifying the type, etc. of the acquired material data (for example, a data set class code, etc.) is added to the material data, so that the conversion unit 107 can determine which conversion process to perform.

[0103] In the first embodiment, the information for identifying the format of the material data that the terminal 109 can process has been described as being information that does not change, such as the type of the terminal, but is not limited thereto. That is, the information for identifying the format of the material data that the terminal 109 can process may be information that changes dynamically. For example, when the material data that the terminal 109 can process changes due to an increase in the processing load, the terminal 109 transmits the information again to the information processing device 100. This enables the information processing device 100 to determine appropriate material data to be output to the terminal 109 and perform the conversion process. Furthermore, the information processing device 100 may dynamically determine the material data to be generated by the conversion process in response to a change in the communication bandwidth of the communication path with the terminal 109.

[0104] Second Embodiment In the first embodiment, the system configuration in which the terminal 109 generates a virtual viewpoint image is described. In the present embodiment, the configuration in which an information processing device generates a virtual viewpoint image is described. Note that the hardware configuration of the information processing device 200 in this embodiment is similar to that in the first embodiment, and therefore the description is omitted. Also, the same reference numerals are used for the functional configuration similar to that in the first embodiment, and the description is omitted.

[0105] FIG. 17 is a diagram for explaining the configuration of a virtual viewpoint image generating system 2 having an information processing device 200. The information processing device 200 has a conversion unit 201, a virtual viewpoint image generating unit 202, and a transmission / reception unit 203 in addition to the same configuration as the information processing device 100 of the first embodiment. The terminals 202a to d set a virtual viewpoint based on a user operation or the like, and transmit virtual viewpoint information indicating the position of the set virtual viewpoint and the line of sight direction from the virtual viewpoint to the information processing device 200. The terminals 204a to d do not have a function (renderer) for generating a virtual viewpoint image, and only set the virtual viewpoint and display the virtual viewpoint image. In the following description, the terminals 204a to d will be simply referred to as terminals 204 unless there is a particular reason to distinguish them.

[0106] The transmitting / receiving unit 203 receives virtual viewpoint information from the terminal 204 and transmits it to the virtual viewpoint image generating unit 202. The transmitting / receiving unit 203 also acquires information for identifying the type of the virtual viewpoint image to be output to the terminal 204 from the terminal 204. The transmitting / receiving unit 203 also has a function of transmitting the generated virtual viewpoint image to the terminal that transmitted the virtual viewpoint information. The virtual viewpoint image generating unit 202 generates a virtual viewpoint image based on the virtual viewpoint information and material data acquired from the storage unit 106 or the conversion unit 201. The method of generating a virtual viewpoint image using material data is similar to the method described in the first embodiment.

[0107] The conversion unit 201 converts the material data stored in the storage unit 106 by a process similar to the conversion process in the first embodiment according to the format of the virtual viewpoint image to be output to the terminal 204. For example, the point cloud data and foreground image data are stored in the storage unit 106, and the terminal 204, which is the output destination of the virtual viewpoint image, displays the virtual viewpoint image generated based on the colored mesh data. In this case, the conversion unit 201 converts the point cloud data and foreground image data stored in the storage unit 106 into colored mesh data. The virtual viewpoint image generation unit 202 generates a virtual viewpoint image using the colored mesh data acquired by the conversion process by the conversion unit 201, and outputs it to the terminal 204 via the transmission / reception unit 203. Also, for example, the point cloud data and foreground image data are stored in the storage unit 106, but the terminal 204 cannot display the virtual viewpoint image generated using the point cloud data stored in the storage unit 106 due to the display capability. In this case, the conversion unit 201 determines the content of the conversion process of the material data based on information on the display capability acquired from the terminal 204. Here, it is assumed that the conversion unit 201 has determined that the terminal 204 can display a virtual viewpoint image generated by converting the point cloud data to a lower resolution. The conversion unit 201 converts the point cloud data stored in the storage unit 106 to a lower resolution (increase the voxel size). The virtual viewpoint image generation unit 202 generates a virtual viewpoint image using the material data obtained by the conversion process by the conversion unit 201, and outputs the generated virtual viewpoint image to the terminal 204 via the transmission / reception unit 203.

[0108] The conversion unit 201 can also convert the format of the virtual viewpoint image generated by the virtual viewpoint image generation unit 202. At this time, the virtual viewpoint image generation unit 202 may use the material data stored in the storage unit 106, or may use the material data converted by the conversion unit 201. For example, it is assumed that the resolution of the virtual viewpoint image generated by the virtual viewpoint image generation unit 202 is higher than a predetermined value and cannot be displayed on the terminal 204. In this case, the conversion unit 201 converts the format of the generated virtual viewpoint image so that the resolution of the virtual viewpoint image becomes lower. It is also assumed that an application or software for reproducing the virtual viewpoint image used by the terminal 204 can only reproduce virtual viewpoint images in a specific format. In this case, the conversion unit 201 converts the format of the generated virtual viewpoint image so that the format of the virtual viewpoint image is a format that can be reproduced by the terminal 204. For example, if the terminal 204 is a head mounted display (HMD), the generated virtual viewpoint image is converted into an image for the HMD. In this way, the conversion unit 201 can convert the virtual viewpoint image according to the display capability of the terminal, the format of the virtual viewpoint image that the terminal can display, the type of the terminal, and the like.

[0109] As described above, the conversion unit 201 converts at least one of the material data stored in the storage unit 106 and the virtual viewpoint image generated by the virtual viewpoint image generation unit 202, thereby making it possible to output a virtual viewpoint image in an appropriate format to the terminal 204. Note that this embodiment can be implemented in combination with the first embodiment. That is, a configuration may be adopted in which material data that can be processed by a device is output to a device having a renderer, and a virtual viewpoint image that can be displayed by a device is output to a device not having a renderer. Even in this configuration, the conversion unit executes a conversion process of converting the format of at least one of the material data and the virtual viewpoint image according to the output destination device.

[0110] The processing performed by the information processing device 200 and the terminal 204 will be described with reference to Fig. 18. Here, an example of processing when the conversion unit 201 converts material data will be described. Note that the same processes as those in Fig. 11 are denoted by the same reference numerals, and descriptions thereof will be omitted.

[0111] In S1801, the transmission / reception unit 203 acquires information for identifying the format of the virtual viewpoint image to be output to the terminal 204 from the terminal 204. The acquired information may be, for example, the same information as that described in the first embodiment. The conversion unit 201 performs conversion processing in S1103 and S1104 according to the acquired information.

[0112] In S1802, the virtual viewpoint image generating unit 202 and the terminal 204 repeatedly execute the processes from S1803 to S1109 for each frame of the virtual viewpoint image to be generated. In S1803, the transmitting / receiving unit 203 acquires virtual viewpoint information representing the position of the virtual viewpoint and the line of sight direction from the virtual viewpoint from the terminal 204. In addition, the transmitting / receiving unit 203 transmits the acquired virtual viewpoint information to the virtual viewpoint image generating unit 202. In S1804, the virtual viewpoint image generating unit 202 generates a virtual viewpoint image based on the material data obtained by the conversion process by the conversion unit 201 and the virtual viewpoint information acquired from the transmitting / receiving unit 203. In S1805, the transmitting / receiving unit 203 outputs the generated virtual viewpoint image to the terminal 204. The terminal 204 displays the virtual viewpoint image at S1109.

[0113] In S1806, the information processing device 200 determines whether to end the repetitive processing. For example, when the generation of the virtual viewpoint image is ended by an instruction from the terminal, the repetitive processing ends. With the configuration described above, a virtual viewpoint image in an appropriate format is output according to the terminal 204. Note that, when the conversion unit 201 converts the format of the virtual viewpoint image, a step of conversion processing of the virtual viewpoint image is added after S1804 in the processing of FIG. 18. In this case, the conversion processing of the material data (S1103, S1104) is not essential.

[0114] In the process shown in Fig. 18, the conversion process is performed in S1104, and then the process for each frame of the virtual viewpoint image is performed, but this is not limited to the above. That is, the information processing device 100 may read out the material data stored in the storage unit 108 according to the acquired virtual viewpoint information, and perform the conversion process to generate the virtual viewpoint image. In this case, for example, the processes from S1801 to S1104 in Fig. 18 are performed between S1803 and S1804.

[0115] (Other embodiments) In the above-described first and second embodiments, the example of outputting the material data to a terminal that displays a virtual viewpoint image has been described, but the output destination device is not limited to this. For example, the above-described embodiments are applicable even when the material data is output to another device that acquires the material data and performs a predetermined process. In this case, the information processing device 100 determines the material data to be output to the other device based on information for identifying the format of the material data that the other output destination device can process, and performs conversion processing of the material data.

[0116] The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more functions. [Explanation of symbols]

[0117] 100 Information processing device 105 Acquisition Department 107 Conversion unit 110 Transmitter / receiver

Claims

1. an acquisition means for acquiring three-dimensional shape data that is generated based on a plurality of captured images obtained by capturing images of a subject using a plurality of imaging devices and is used to generate a virtual viewpoint image, the three-dimensional shape data being expressed in a first format; a conversion means for converting a format of the three-dimensional shape data acquired by the acquisition means from the first format to a second format based on information of software used for generating a virtual viewpoint image, the software being used by another device that generates a virtual viewpoint image using the three-dimensional shape data; an output means for outputting the three-dimensional shape data converted into the second format by the conversion means to the other device; 13. An information processing device comprising:

2. 2. The information processing method according to claim 1, wherein the format of the three-dimensional shape data converted by said conversion means is a format that specifies the type of the three-dimensional shape data.

3. 3. The information processing apparatus according to claim 2, wherein the three-dimensional shape data represented in the second format is data including point cloud data representing the three-dimensional shape of the subject and texture data representing a color of the subject.

4. 3. The information processing apparatus according to claim 2, wherein the three-dimensional shape data represented in the second format is data including point cloud data representing the three-dimensional shape and color of the subject.

5. 3. The information processing apparatus according to claim 2, wherein the three-dimensional shape data represented in the second format is data including mesh data representing the three-dimensional shape and color of the subject.

6. 3. The information processing apparatus according to claim 2, wherein the three-dimensional shape data represented in the second format is data including mesh data representing the three-dimensional shape of the subject and texture data representing a color of the subject.

7. 7. The information processing apparatus according to claim 4, wherein the three-dimensional shape data represented in the first format is data including point cloud data representing the three-dimensional shape of the subject and texture data representing a color of the subject.

8. 7. The information processing apparatus according to claim 6, wherein the three-dimensional shape data represented in the first format is data including point cloud data representing the three-dimensional shape and color of the subject.

9. 6. The information processing apparatus according to claim 3, wherein the three-dimensional shape data represented in the first format is data including mesh data representing the three-dimensional shape of the subject and texture data representing a color of the subject.

10. 5. The information processing apparatus according to claim 4, wherein the three-dimensional shape data represented in the first format is data including mesh data representing the three-dimensional shape and color of the subject.

11. 11. The information processing apparatus according to claim 1, wherein the information includes information regarding a type of the other apparatus.

12. 12. The information processing apparatus according to claim 1, wherein the information includes information regarding a format of a virtual viewpoint image that can be displayed by the other apparatus.

13. a generating means for generating a virtual viewpoint image represented in a third format based on the three-dimensional shape data acquired by the acquiring means, The conversion means converts the format of the virtual viewpoint image generated by the generation means from the third format to a fourth format.

13. The information processing apparatus according to claim 1,

14. an acquiring step of acquiring three-dimensional shape data expressed in a first format, the three-dimensional shape data being generated based on a plurality of captured images obtained by capturing images of a subject using a plurality of imaging devices and used to generate a virtual viewpoint image; a conversion step of converting the format of the three-dimensional shape data acquired in the acquisition step from the first format to a second format based on information of software used for generating a virtual viewpoint image, the software being used by another device that generates a virtual viewpoint image using the three-dimensional shape data; an output step of outputting the three-dimensional shape data converted into the second format by the conversion step to the other device; 13. An information processing method comprising:

15. A program for causing a computer to function as the information processing device according to any one of claims 1 to 13.

Images