Information processing system, method for operating information processing system, and program

Abstract

The present disclosure relates to an information processing system, a method for operating an information processing system, and a program that enable efficient rendering and reproduction of virtual viewpoint images. A subject is imaged by a plurality of cameras, and an aggregate texture is generated by packing silhouette images comprising the silhouette of the subject that are generated on the basis of camera parameters and mesh data, obtained by meshing a three-dimensional model of the subject, from each of a plurality of sets of image data captured by the plurality of cameras, the camera parameters and mesh data being used when generating a virtual viewpoint image from the plurality of sets of image data. The present disclosure can be applied to virtual viewpoint image reproduction devices.

Description

Information processing system, method of operating the information processing system, and program

[0001] The present disclosure relates to an information processing system, a method of operating the information processing system, and a program, and more particularly to an information processing system, a method of operating the information processing system, and a program capable of efficiently reproducing virtual viewpoint images (free viewpoint images, volumetric images).

[0002] Techniques for generating virtual viewpoint (free viewpoint, volumetric) images by synthesizing images captured using a plurality of cameras have been proposed.

[0003] For example, a technique has been proposed in which rectangular images of a plurality of viewpoints obtained by photographing an object from a plurality of viewpoints are aggregated and transmitted as an aggregated texture, and the receiving side draws a virtual viewpoint image based on the aggregated texture (see Patent Documents 1 and 2).

[0004] International Publication No. 2018 / 150933 International Publication No. 2024 / 053371

[0005] By the way, in the techniques of Patent Documents 1 and 2, an aggregated texture in which rectangular images obtained by photographing an object serving as a subject from a plurality of viewpoints are aggregated is transmitted. However, since the aggregated texture is formed by aggregating rectangular images that include a large amount of background area other than the object serving as the subject, the size cannot be made sufficiently small. Therefore, the data amount of the aggregated texture becomes unnecessarily large, leading to an increase in processing load and a decrease in transmission efficiency, and the drawing and reproduction of virtual viewpoint images cannot be performed efficiently.

[0006] The present disclosure has been made in view of such a situation, and particularly realizes efficient drawing and reproduction of virtual viewpoint images.

[0007] An information processing system and program, representing one aspect of this disclosure, is an information processing system and program that captures an object with multiple cameras and generates a virtual viewpoint image from multiple image data captured by the multiple cameras. The system includes mesh data in which a three-dimensional model of the object is meshed, and a texture packing unit that packs silhouette images consisting of the silhouette of the object, generated from each of the multiple image data based on camera parameters, to generate an aggregated texture.

[0008] One aspect of the information processing system operation method of this disclosure is an information processing system operation method that includes a texture packing process to generate an aggregated texture by packing silhouette images consisting of the silhouette of the subject, which are generated from each of the multiple image data, based on mesh data of a three-dimensional model of the subject and camera parameters, which are used when generating a virtual viewpoint image from multiple image data captured by the multiple cameras, and when generating a virtual viewpoint image from the multiple image data.

[0009] In one aspect of this disclosure, a subject is captured by multiple cameras, and a mesh data of a three-dimensional model of the subject, used when generating a virtual viewpoint image from multiple image data captured by the multiple cameras, and silhouette images consisting of the silhouette of the subject, generated from each of the multiple image data based on camera parameters, are packed together to generate an aggregated texture.

[0010] This is a diagram illustrating an example configuration of an image processing system for generating volumetric images. This is a diagram illustrating the process of generating volumetric images. This is a diagram illustrating rendering based on vertices that are not viewpoint-dependent and viewpoint-dependent. This is a diagram illustrating an example of a data structure required for view-dependent rendering. This is a diagram illustrating an example of a data structure required for view-independent rendering. This is a diagram illustrating an example of a data structure required for rendering according to this disclosure. This is a diagram illustrating an overview of an information processing system to which the technology of this disclosure is applied. This is a diagram illustrating the generation of a multi-view imaging system used to generate virtual viewpoint images in an information processing system to which the technology of this disclosure is applied. This is a flowchart illustrating the virtual viewpoint image display process by the information processing system of Figure 7. This is a diagram illustrating a detailed configuration example of the 3D model generation unit of Figure 6. This is a diagram illustrating image data acquired by the data acquisition unit. This is a diagram illustrating a silhouette image generated by the silhouette generation unit. This is a diagram illustrating mesh data. This is a diagram illustrating the data structure of mesh data. This is a diagram illustrating a UV map texture. This is a diagram illustrating the UV coordinates of the UV map texture assigned to the mesh data. This is a flowchart illustrating the 3D model generation process by the 3D model generation unit of Figure 10. This is a diagram illustrating a detailed configuration example of the encoding unit of Figure 6. This is a diagram illustrating a reprojected silhouette image generated by the reprojected silhouette generation unit. This is a diagram illustrating an aggregated texture. This figure illustrates an example of the data structure for offset information. This figure illustrates a two-dimensional code encoded with offset information. This is a flowchart illustrating the encoding process by the encoding unit in Figure 18. This is a flowchart illustrating the texture packing process. This figure illustrates a detailed configuration example of the decoding unit in Figure 6. This figure illustrates the processing of the texture extraction unit. This is a flowchart illustrating the decoding process by the decoding unit in Figure 25. This figure illustrates a detailed configuration example of the rendering unit in Figure 6. This is a flowchart illustrating the rendering process by the rendering unit in Figure 28. This figure illustrates a modified example where the aggregated texture does not include a UV map texture. This shows an example of a computer configuration.

[0011] Preferred embodiments of this disclosure will be described in detail below with reference to the attached drawings. In this specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant descriptions will be omitted.

[0012] The following describes embodiments for carrying out the technology of this disclosure. The description will be in the following order: 1. Overview of this disclosure 2. Preferred embodiments 3. Modifications 4. Description of a computer to which this technology is applied 5. Application examples

[0013] <<1. Overview of this Disclosure>> <Generation of Virtual Viewpoint Images> This disclosure, in particular, realizes efficient rendering and playback of virtual viewpoint images (free viewpoint images, volumetric images).

[0014] Therefore, in explaining the technology of this disclosure, we will briefly describe the configuration for generating virtual viewpoint images and the generation process.

[0015] To generate a virtual viewpoint image, images of the subject taken from multiple viewpoint positions are required.

[0016] Therefore, an information processing system like the one shown in Figure 1 is used to generate virtual viewpoint images.

[0017] The information processing system 11 in Figure 1 is equipped with multiple cameras 31-1 to 31-8 capable of capturing images of the subject 32 from many viewpoint positions.

[0018] In Figure 1, an example is shown with eight cameras 31, but any other number of cameras may be used. Also, in Figure 1, an example is shown where cameras 31-1 to 31-8 at eight viewpoint positions are arranged to surround the subject 32 in two dimensions, but more cameras 31 may be arranged to surround it in three dimensions.

[0019] Hereafter, unless there is a need to distinguish between cameras 31-1 to 31-8, they will simply be referred to as camera 31, and the other components will be referred to similarly.

[0020] Cameras 31-1 to 31-8 each capture images of the subject 32 from multiple different viewpoint positions.

[0021] Hereafter, images of the subject 32 captured by multiple cameras 31 from multiple different viewpoints will also be referred to as multi-view images.

[0022] The virtual viewpoint image is generated by rendering at the virtual viewpoint position from the multi-view images captured by the information processing system 11 in Figure 1. In generating the virtual viewpoint image, 3D data of the subject 32 is generated from the multi-view images, and based on the generated 3D data, the virtual viewpoint image (rendered image) is generated by rendering the subject 32 at the virtual viewpoint position.

[0023] In other words, as shown in Figure 2, three-dimensional data 52 of the subject 32 is generated based on the multi-view images 51 of the subject 32.

[0024] Then, based on the 3D data 52 of the subject 32, rendering is performed at the virtual viewpoint position to generate a virtual viewpoint image 53.

[0025] <Rendering> Next, rendering for generating a virtual viewpoint image 53 at the virtual viewpoint position will be explained using the example of assigning color based on predetermined vertices in the subject 32.

[0026] Rendering can be categorized into two types: view-independent rendering, which does not depend on the viewpoint, and view-dependent rendering, which depends on the viewpoint position.

[0027] When assigning color to vertices P on the subject 32 from a virtual viewpoint position using view-independent rendering, based on multi-view images of the subject 32 from viewpoint positions cam0 to cam2, the process is as shown in the left part of Figure 3.

[0028] In other words, as shown in the upper left of Figure 3, the pixel values ​​corresponding to vertices P in the multi-view images from viewpoint positions cam0 to cam2 are combined, taking into account the angle between the line of sight direction to vertex P at each viewpoint position cam0 to cam2 and the normal direction at vertex P, and this is used as the pixel value Pi viewed from the virtual viewpoint VC, as shown in the lower left of Figure 3. View-independent rendering is a rendering method commonly known as UV mapping, for example.

[0029] On the other hand, when assigning color to vertices P on the subject 32 from a virtual viewpoint position using view-dependent rendering based on multi-view images of the subject 32 from viewpoint positions cam0 to cam2, the process is as shown in the right side of Figure 3.

[0030] In other words, as shown in the upper right of Figure 3, the pixel values ​​corresponding to vertices P in the multi-view images of viewpoint positions cam0 to cam2 are combined by considering the angle between the line of sight direction to vertex P at each viewpoint position cam0 to cam2 and the normal direction at vertex P, as well as the angle between the line of sight direction from the virtual viewpoint VC and the normal direction of vertex P, and this is expressed as the pixel value Pd viewed from the virtual viewpoint VC, as shown in the lower right of Figure 3.

[0031] As shown in Figure 3, view-dependent rendering, compared to view-independent rendering, takes into account the angle between the line of sight from the virtual viewpoint and the normal direction at vertex P when assigning color to vertex P, making it possible to assign more natural colors.

[0032] <Data Structure Required for View-Dependent Rendering> Next, we will explain the data structure required for view-dependent rendering.

[0033] The VDDS data structure required for view-dependent rendering includes mesh data 71, camera parameters + rectangle placement information 72, and packing textures 73, as shown in Figure 4.

[0034] The mesh data 71 is information that shows the correspondence between the coordinates of the vertex positions of the object represented by the 3D mesh and the vertex positions of the triangular patches that form the surface of the object on the 3D mesh.

[0035] The camera parameters + rectangular placement information 72 consists of camera parameters that show the relationship between the position in three dimensions and the coordinate position in the image captured by each camera 31, and placement coordinate information of a rectangular image containing the subject among the images captured by the multiple cameras 31 packed in the packing texture 73.

[0036] The packing texture 73 is texture information of a subject, formed by packing images (textures) that are extracted from images captured by multiple cameras 31, specifically those containing the subject.

[0037] In other words, in view-dependent rendering, the coordinate positions of each vertex constituting the 3D model of the subject are determined based on the mesh data, which in turn are the positions of the three vertices of the triangular patch that makes up the 3D model of the subject.

[0038] Then, by using camera parameters according to the viewpoint position, the coordinate position within the image is determined based on the three vertices of each triangular patch. Furthermore, the placement coordinates of each rectangular image within the packing texture are determined based on the rectangular placement information. Rendering is then achieved by using the texture corresponding to the specified coordinate position within the image for each rectangular image with the specified placement coordinates.

[0039] <Data Structures Required for View-Independent Rendering> Next, we will explain the data structures required for view-independent rendering.

[0040] The VIDS data structure required for view-independent rendering includes mesh data + UV coordinates 81 and UV texture 82, as shown in Figure 5.

[0041] Mesh data + UV coordinates 81 is composed of information on UV coordinates, which are two-dimensional coordinates within the UV texture 82 corresponding to each of the three vertices of the triangular patches of the mesh data 71, in addition to the mesh data 71 described above.

[0042] The UV texture 82 is a map in which the three-dimensional mesh of the subject is two-dimensionally developed and the texture is arranged.

[0043] That is, view-independent rendering is substantially UV mapping itself. Based on the mesh data + UV coordinates 81, the positions of the three vertices of the triangular patch are specified from the mesh data, and further, the UV coordinates on the UV texture 82 for each of the three vertices are specified, and rendering is performed using the specified texture.

[0044] As described above, in view-dependent rendering, when the UV texture is not provided and the vertex positions of the triangular patches constituting the subject are specified based on the mesh data 71, the texture on the packing texture 73 is used for each viewpoint position using the camera parameters, and rendering is realized.

[0045] Therefore, view-dependent rendering can achieve high-precision color reproduction, but since the processing load is large, the available playback devices are limited according to their performance.

[0046] In contrast, view-independent rendering is substantially UV mapping itself. When the vertex positions of the triangular patches constituting the subject are specified based on the mesh data + UV coordinates 81, only the coordinate positions on the corresponding UV texture 82 are specified, and the texture at the coordinate positions is read out and used.

[0047] Therefore, although view-independent rendering may cause distortion when the UV texture 82 is unfolded from a three-dimensional mesh to two dimensions and may not be able to reproduce appropriate colors, it has a small processing load and can be reproduced on most playback devices.

[0048] Also, in the packing texture 73, an image composed of a rectangular area including a subject was cut into a plurality of pieces and arranged to be aggregated as one image data. However, in the image composed of a rectangular area including a subject that constitutes the packing texture 73, since many areas other than the subject surface remain, in order to aggregate these into one image, the area efficiency is poor and the overall size cannot be reduced. For this reason, it unnecessarily increases the processing load related to the reproduction of the virtual viewpoint image and causes a reduction in transmission efficiency.

[0049] Therefore, in the data structure used for the rendering of the present disclosure, as shown in FIG. 6, based on the data structure of view-dependent rendering, by adding information on UV coordinates in the data structure of view-independent rendering, it is possible to switch between view-dependent rendering and view-independent rendering according to the processing load of the playback device.

[0050] As a result, it is possible to eliminate the limitations according to the performance of the playback device and switch between view-dependent rendering and view-independent rendering according to the processing load of the playback device to reproduce the virtual viewpoint image.

[0051] As a result, it becomes possible to realize the optimal drawing and reproduction of the virtual viewpoint image in playback devices with various performances.

[0052] Furthermore, as shown in Figure 6, the aggregated texture 91 is formed by extracting only the silhouette of the subject from a rectangular area containing the subject, so that the image containing the subject is reduced to a minimum image and then densely arranged and aggregated.

[0053] In this process, offset information indicating the position on the aggregated texture is encoded and transmitted together as a two-dimensional code DC. From this offset position, a rectangular image Ph is extracted, for example, as an image relating to subject H in Figure 6, and playback processing is performed as a projected image for view-dependent rendering.

[0054] As a result, the aggregated texture 91 in Figure 6 is formed by arranging and aggregating images of only the subject surface so that they are denser compared to the packing texture 73 in Figure 4. This improves area efficiency and reduces the size of the aggregated texture 91 itself, thereby suppressing an increase in processing load and improving transmission efficiency.

[0055] As a result, it becomes possible to achieve efficient rendering and playback of virtual viewpoint images.

[0056] Furthermore, the data structure DS in Figure 6 of this disclosure consists of a view-dependent rendering part VDP used in view-dependent rendering and a view-independent rendering part VIP used in view-independent rendering, with each sharing some parts.

[0057] The View-Dependent Rendering Part (VGP) is a data structure (CVDS) required for View-Dependent Rendering, as explained with reference to Figure 4. In this part, camera parameters 72' and aggregated texture 91 are provided instead of camera parameters + rectangle placement information 72 and packing texture 73. Camera parameters 72' are the same as camera parameters + rectangle placement information 72, but with the rectangle placement information removed. Aggregated texture 91 is formed by arranging and aggregating the rectangular images of each subject so that only the images of the subject surface are densely packed. Aggregated texture 91 also has a two-dimensional code DC at its lower end, which encodes offset information indicating the position of each image, and also includes a UV texture Pt.

[0058] Furthermore, the view-independent rendering part VIP consists of a aggregated texture 91 which includes mesh data 71 and a UV texture Pt composed of UV coordinates 81 associated with the triangular patches of the mesh data 71. Note that the UV coordinates 81 in Figure 6 indicate that the UV coordinates of the UV texture Pt are associated with the triangular patches of the mesh data 71 and are not included in the data structure, but it can be considered that the UV texture Pt itself contains the information of the UV coordinates 81 associated with the triangular patches of the mesh data 71.

[0059] In the view-independent rendering part VIP, as explained with reference to Figure 5, the UV texture 82 is not provided among the data structure VIDS required for view-independent rendering, and instead an aggregated texture 91 is provided.

[0060] Furthermore, in the view-independent rendering part VIP, the data structure DS in Figure 6 is such that the UV texture 82 and the corresponding UV texture Pt are included in the aggregated texture 91.

[0061] Furthermore, the UV texture Pt is configured to have UV coordinates 81 associated with the triangular patches of the mesh data 71.

[0062] Based on this configuration, the data structure DS in Figure 6 is based on the data structure VDDS required for view-dependent rendering, as explained with reference to Figure 4. It has a structure comprising a aggregated texture 91 which has mesh data 71 and UV coordinates (two-dimensional coordinates of UV texture) 81 associated with the triangular patches constituting the mesh data 71. Furthermore, the aggregated texture 91 is constructed by arranging and aggregating images so that only the surface of the subject is densely contained.

[0063] <<2. Preferred Embodiments>> <Overview of the Information Processing System of the Disclosure> Figure 7 shows an overview of an information processing system to which the technology of the Disclosure is applied.

[0064] The information processing system 101 in Figure 7 includes a data acquisition unit 111, a 3D model generation unit 112, an encoding unit 113, a transmission unit 114, a reception unit 115, a decoding unit 116, a rendering unit 117, and a display unit 118.

[0065] The data acquisition unit 111 acquires image data for generating a 3D model of the subject. For example, as shown in Figure 8, it acquires multiple viewpoint images as image data from a multi-view imaging system 120, which is composed of multiple cameras 121-1 to 121-n arranged to surround the subject 131.

[0066] In the following, unless there is a need to distinguish between cameras 121-1 to 121-n, they will simply be referred to as camera 121, and the same will apply to other configurations. Furthermore, multiple viewpoint images will also be referred to as multi-viewpoint images.

[0067] In this case, the multiple viewpoint images are preferably images captured synchronously by multiple cameras 121.

[0068] Alternatively, the data acquisition unit 111 may acquire image data by moving a single camera 121 and capturing images of the subject from multiple viewpoints.

[0069] Furthermore, the data acquisition unit 111 may acquire a single captured image of the subject as image data by using machine learning in the 3D model generation unit 112, which will be described later.

[0070] Furthermore, the data acquisition unit 111 may perform calibration based on the image data and acquire the internal and external parameters of each camera 121. The camera parameters, which will be described later, are composed of a combination of the internal and external parameters of the camera 121, and the details will be described later.

[0071] Furthermore, the data acquisition unit 111 may acquire, for example, multiple depth information (depth maps) indicating the distance from multiple viewpoints to the subject.

[0072] The 3D model generation unit 112 generates a model having three-dimensional information of the subject 131 based on image data for generating a 3D model of the subject 131.

[0073] The 3D model generation unit 112 generates a 3D model of the subject by using images from multiple viewpoints (for example, silhouette images from multiple viewpoints) to remove the three-dimensional shape of the subject, based on image data and camera parameters, for example, using a so-called Visual Hull.

[0074] In this case, the 3D model generation unit 112 can further deform the 3D model generated using Visual Hull with high precision using multiple depth information points indicating the distance from multiple viewpoints to the subject.

[0075] Alternatively, for example, the 3D model generation unit 112 may generate a 3D model of the subject 131 from a single captured image of the subject 131.

[0076] The 3D models generated by the 3D model generation unit 112 can be described as 3D model videos, as they are generated in time-series frame units.

[0077] Furthermore, since the 3D model is generated using images captured by camera 121, it can also be described as a 3D model based on real-life images.

[0078] A 3D model can represent the shape information of the surface shape of the object 131 in the form of mesh data, such as a polygon mesh, which is represented by the connections between vertices. Hereafter, the shape information representing the surface shape of the object 131 that represents the 3D model will also be referred to as mesh data (or 3D shape data).

[0079] The methods of representing 3D models are not limited to these; they may also be described using the so-called point cloud representation method, which uses point position information.

[0080] Color information data is also generated as a texture, linked to this mesh data. Here, a UV map texture is generated, which is the color information data for a View Independent texture that has a constant color regardless of the viewing direction. The 3D model generation unit 112 then outputs the generated mesh data and UV map texture, as well as the acquired image data and camera parameters, directly to the encoding unit 113.

[0081] The encoding unit 113 generates mesh encoded data by converting the mesh data, which is the data of the 3D model generated by the 3D model generation unit 112, into a format suitable for transmission and storage. At this time, the encoding unit 113 generates color information data for View Independent textures based on the mesh data, UV map texture, and image data, and also aggregates the UV texture map, which is the color information data for View Dependent textures, to generate an aggregated texture. Furthermore, it generates texture encoded data by converting this to a format suitable for transmission and storage. The encoding unit 113 then outputs the mesh encoded data, texture encoded data, and camera parameters to the transmission unit 114.

[0082] The transmitting unit 114 transmits the mesh encoded data, texture encoded data, and camera parameters, which are data of the 3D model formed by the encoding unit 113, to the receiving unit 115 as transmission data. The transmitting unit 114 transmits the transmission data to the receiving unit 115 after performing a series of processes by the data acquisition unit 111, the 3D model generation unit 112, and the encoding unit 113 offline.

[0083] Furthermore, the transmitting unit 114 may transmit the transmission data generated from the series of processes described above to the receiving unit 115 in real time.

[0084] The receiving unit 115 receives the transmission data transmitted from the transmitting unit 114 and outputs it to the decoding unit 116.

[0085] The decoding unit 116 reconstructs mesh data, UV map textures, image data, and camera parameters based on the transmission data received by the receiving unit, and outputs them to the rendering unit 117.

[0086] The rendering unit 117 projects the mesh of the 3D model as an image of the viewpoint position where it should be drawn, based on the mesh data, UV map texture, image data, and camera parameters. It then performs texture mapping by applying textures that represent color and pattern, and outputs it to the display unit 118 for display. The drawing at this time can be set arbitrarily, regardless of the viewpoint position of the camera 121 at the time of shooting, and can be viewed from any viewpoint. Hereafter, an image with an arbitrarily set free viewpoint will also be referred to as a virtual viewpoint image or free viewpoint image.

[0087] Texture mapping can be done using two methods: View Dependent, which takes into account the user's viewing perspective, and View Independent, which does not.

[0088] The View Dependent method has the advantage of achieving higher quality rendering than the View Independent method because it changes the texture applied to the 3D model according to the viewer's viewpoint.

[0089] On the other hand, the View Independent method does not take into account the viewer's viewpoint, which has the advantage of reducing processing load compared to the View Dependent method.

[0090] Furthermore, viewing viewpoint data is obtained when the display device detects the user's viewing location (Region of Interest) and inputs it from the display device to the rendering unit 117.

[0091] The display unit 118 displays the results rendered by the rendering unit 117 on the display surface of the display device. The display device may be a 2D monitor or a 3D monitor, for example, a head-mounted display, a spatial display, a mobile phone, a television, or a PC (Personal Computer).

[0092] Figure 7 shows the information processing system 101, illustrating a series of steps from the data acquisition unit 111, which acquires captured images that are the raw materials for generating content, to the display unit 118, which controls the display device that the user views.

[0093] However, this does not mean that all functional blocks are required to implement this disclosure; rather, this disclosure can be implemented using individual functional blocks or combinations of functional blocks.

[0094] For example, the information processing system 101 in Figure 7 includes a transmission unit 114 and a reception unit 115 to illustrate a series of steps from the content creator to the content viewer via content data distribution. However, if the entire process from content creation to viewing is performed on the same information processing device (e.g., a personal computer), it is not necessary to include an encoding unit 113, a transmission unit 114, a decoding unit 116, or a reception unit 115.

[0095] In implementing the information processing system 101 shown in Figure 7, the same person may perform all the tasks, or different people may perform tasks for each functional block.

[0096] One example is when business operator A generates 3D content through a data acquisition unit 111, a 3D model generation unit 112, and an encoding unit 113. Then, the 3D content is distributed through business operator B's transmission unit (platform) 114, and business operator C's display unit 118 receives, renders, and controls the display of the 3D content.

[0097] Furthermore, each functional block can be implemented on the cloud. For example, the rendering unit 117 may be implemented within the display device or on a server. In the latter case, information will be exchanged between the display device and the server.

[0098] Figure 7 illustrates the data acquisition unit 111, 3D model generation unit 112, encoding unit 113, transmission unit 114, reception unit 115, decoding unit 116, rendering unit 117, and display unit 118 as a combined information processing system 101.

[0099] However, in this specification, the information processing system 101 is defined as an information processing system 101 if it involves two or more functional blocks. For example, the data acquisition unit 111, 3D model generation unit 112, encoding unit 113, transmission unit 114, reception unit 115, decoding unit 116, and rendering unit 117 can be collectively referred to as the information processing system 101, without including the display unit 118.

[0100] <Virtual Viewpoint Image Display Processing> Next, an example of the flow of virtual viewpoint image display processing using the information processing system 101 in Figure 7 will be explained with reference to the flowchart in Figure 9.

[0101] In step S101, the data acquisition unit 111 acquires image data and camera parameters for generating a 3D model of the subject 131 and outputs them to the 3D model generation unit 112.

[0102] In step S102, the 3D model generation unit 112 executes a 3D model generation process, described later, and generates mesh data and a UV map texture that represent a model having three-dimensional information of the subject 131, based on image data and camera parameters for generating a 3D model of the subject 131. The image data and camera parameters are then output together to the encoding unit 113.

[0103] In step S103, the encoding unit 113 performs an encoding process described later, encoding mesh data representing the shape of the 3D model generated by the 3D model generation unit 112, a UV map texture, image data, and camera parameters into a format suitable for transmission and storage, and outputs them to the transmission unit 114.

[0104] In step 104, the transmission unit 114 transmits the encoded data.

[0105] In step 105, the receiving unit 115 receives the transmitted data and outputs it to the decoding unit 116.

[0106] In step 106, the decoding unit 116 performs a decoding process (decoding process) described later, converting the data into mesh data, UV map textures, image data, and camera parameters that represent the shapes necessary for display, and outputs them to the rendering unit 117.

[0107] In step 107, the rendering unit 117 performs a rendering process described later to render a virtual viewpoint image using mesh data representing the shape of the 3D model, a UV map texture, image data, and camera parameters, and outputs the rendered result to the display unit 118.

[0108] In step 108, the display unit 118 displays the virtual viewpoint image which is the result of rendering.

[0109] When the process in step S108 is completed, the virtual viewpoint image display process by the information processing system 101 is terminated.

[0110] <3D Model Generation Unit> Next, with reference to Figure 10, the detailed configuration of the 3D model generation unit 112 will be described.

[0111] The 3D model generation unit 112 generates mesh data and UV map textures based on the image data and camera parameters supplied from the data acquisition unit 111 using general volumetric video generation processing, and outputs them together with the image data and camera parameters to the encoding unit 113.

[0112] The camera parameters are the perspective projection parameters of camera 121, and are represented by a matrix such as the following equation (1).

[0113] ... (1)

[0114] Here, X, Y, and Z are coordinate positions in three-dimensional space based on mesh data, respectively; r11 to r33 are rotational components of camera external parameters; t1 to t3 are translational components of camera external parameters; fx, fy and cx, cy are camera internal parameters that can be expressed as fx, fy and cx, cy; P and Q are two-dimensional coordinates of the projection plane; and s is the depth.

[0115] In other words, as shown in equation (1), the two-dimensional coordinates P and Q of the projection plane and the depth s are obtained by taking the coordinates in three-dimensional space based on mesh data as input and multiplying the matrix of camera external parameters by the matrix of camera intrinsic parameters.

[0116] The 3D model generation unit 112 consists of a silhouette generation unit 151, a stereo matching unit 152, a modeling unit 153, a meshing unit 154, and a UV map generation unit 155.

[0117] The silhouette generation unit 151 extracts the silhouette of the subject 131 based on the image data supplied by the data acquisition unit 111 and outputs it as a silhouette image to the modeling unit 153.

[0118] More specifically, the image data supplied by the data acquisition unit 111 is supplied as image data consisting of multiple image groups, for example, as shown in Figure 11. Figure 11 shows an example of image data Pg consisting of an image group P1-1 to P1-4 relating to the same subject 131. Although Figure 11 shows an example where the image group consists of four images, the number of images in the image group constituting the image data Pg may be more than four.

[0119] The silhouette generation unit 151 generates a silhouette image P12, for example, as shown on the right side of Figure 12, by extracting a silhouette from the image P11 shown on the left side of Figure 12, which is part of the image group that makes up the image data Pg.

[0120] The stereo matching unit 152 generates a depth map (depth image), which is information about the pixel-level distance to the subject, by performing stereo matching processing based on image data captured by multiple cameras 121, and outputs the generated depth map to the modeling unit 153.

[0121] The modeling unit (Visual Hull and Depth Fusion) 153 uses the silhouette image generated by the silhouette generation unit 151 and the depth map generated by the stereo matching unit 152 to integrate three-dimensional information using a technique called Visual Hull and depth map integration processing to generate a 3D model in voxel data format, which is then supplied to the meshing unit 154.

[0122] The meshing unit 154 generates mesh data by converting the 3D model into data of a group of triangular patches.

[0123] Here, the mesh data is the surface data of the subject object, which is formed by a large number of triangular patches, as shown in image P21 of Figure 13, for example.

[0124] In Figure 13, the subject is a woman wearing a skirt, and the surface of the woman is depicted as being covered with triangular patches based on mesh data.

[0125] The data structure of the mesh data corresponds to the mesh data in Figure 4, for example, the mesh data 160 in Figure 14.

[0126] Specifically, on the left side of the mesh data 160 in Figure 14, there is an array of three-dimensional coordinates of the vertices that make up the surface of the subject, indicated by "vertex" in the figure, and on the right side of the mesh data 160, there is an array of vertex numbers of triangles representing the graph structure of the triangular patches, indicated by "face" in the figure.

[0127] In other words, in the array on the left side of the mesh data 160 in Figure 14, the vertex numbers are denoted from top to bottom as 1, 2, 3, 4, ... and the coordinate positions are denoted from top to bottom as (x1, y1, z1), (x2, y2, z2), (x3, y3, z3), (x4, y4, z4), ...

[0128] Furthermore, in the right-hand arrangement of the mesh data 160 in Figure 14, the notations from top to bottom are (2,1,3), (2,1,4), ...

[0129] This shows that the triangular patch indicated by (2,1,3) in the array on the right consists of three points with vertex numbers 2, 1, and 3, and that the coordinate positions of each vertex are (x2,y2,z2), (x1,y1,z1), and (x3,y3,z3).

[0130] Furthermore, the triangle patch indicated by (2,1,4) in the array on the right consists of three points with vertex numbers 2, 1, and 4, and the coordinate positions of each vertex are (x2,y2,z2), (x1,y1,z1), and (x4,y4,z4).

[0131] The UV map generation unit 155 calculates the color information to be assigned to the triangular patch, generates a UV map texture linked to the mesh data in UV coordinates, and outputs it.

[0132] A UV map texture is, for example, like the UV map texture P41 shown in Figure 15, and is basically composed of multiple camera images taken by multiple cameras 121 shown in Figure 8.

[0133] In Figure 15, the UV map texture P41 is shown as an example in which a rectangular image containing the subject woman is extracted from three camera images taken from three different directions of the woman corresponding to the subject 131 in Figure 11. Note that the number of camera images constituting the UV map texture P41 corresponds to the number of cameras 121, so it may be composed of a number of camera images other than three.

[0134] Furthermore, although not shown in the diagram, each of the multiple camera images that make up the UV map texture P41 has undergone lens distortion correction and color correction in the preceding stage, and is configured to correspond to the 3D model through perspective projection transformation using camera parameters.

[0135] The UV coordinates have a relationship as shown in Figure 16, for example. That is, the UV coordinate 161 in Figure 16 is the two-dimensional coordinate of the UV map texture, represented by uv, corresponding to each triangular patch in the mesh data 160.

[0136] In other words, in Figure 16, the UV coordinate 161 in the UV map texture indicated by uv corresponds to the array showing the graph structure of the triangular patch indicated by face.

[0137] For example, the triangular patch (2 / 2, 1 / 1, 3 / 3) in the array shown as the central face in Figure 16 consists of three points with vertex numbers 2, 1, and 3 in the mesh data, and the coordinate positions of each vertex are (x2, y2, z2), (x1, y1, z1), and (x3, y3, z3). These three points correspond to the three points with vertex numbers 2, 1, and 3 on the texture surface, and the coordinate positions of each on the texture surface are (u2, v2), (u1, v1), and (u3, v3).

[0138] Furthermore, the triangular patch shown in the array on the right, (2 / 4, 1 / 1, 4 / 5), is composed of three points with vertex numbers 2, 1, and 4 in the mesh data, and the coordinate positions of these vertices are (x2, y2, z2), (x1, y1, z1), and (x4, y4, z4). These three points correspond to three points with vertex numbers 4, 1, and 5 on the texture surface, and their coordinate positions on the texture surface are (u4, v4), (u1, v1), and (u5, v5).

[0139] Furthermore, the UV coordinate data format shown in Figure 16 is common as CG mesh data and can be rendered using general-purpose CG libraries.

[0140] <3D Model Generation Process> Next, the 3D model generation process by the 3D model generation unit 112 will be explained with reference to the flowchart in Figure 17.

[0141] In step S111, the silhouette generation unit 151 extracts the silhouette of the subject 131 based on the image data supplied by the data acquisition unit 111 and outputs it as a silhouette image to the modeling unit 153.

[0142] In step S112, the stereo matching unit 152 generates a depth map (depth image), which is distance information for each pixel that makes up the image of the subject, by performing stereo matching processing based on the image data captured by the multiple cameras 121, and outputs the generated depth map to the modeling unit 153.

[0143] In step S113, the modeling unit 153 uses the silhouette image generated by the silhouette generation unit 151 and the depth map generated by the stereo matching unit 152 to integrate three-dimensional information using a technique called Visual Hull and depth map integration processing to generate a 3D model in voxel data format, which is then supplied to the meshing unit 154.

[0144] In step S114, the meshing unit 154 generates and outputs mesh data by converting the 3D model in voxel data format into data of a group of triangular patches.

[0145] In step S115, the UV map generation unit 155 calculates color information to be assigned to the triangular patches based on the mesh data, image data, and camera parameters, and generates and outputs a UV map texture linked to the mesh data in UV coordinates.

[0146] In step S116, the 3D model generation unit 112 outputs the acquired image data to the encoding unit 113.

[0147] In step S117, the 3D model generation unit 112 outputs the acquired camera parameters to the encoding unit 113.

[0148] Through the above processing, mesh data representing a 3D model of the subject and a UV map texture are generated based on the image data and camera parameters, and these are supplied to the encoding unit 113 along with the image data and camera parameters.

[0149] <Encoding Unit> Next, the detailed configuration of the encoding unit 113 will be described with reference to Figure 18.

[0150] The encoding unit 113 generates mesh encoded data and texture encoded data based on the mesh data and UV map texture supplied from the 3D model generation unit 112, as well as image data and camera parameters, and outputs them along with the camera parameters. The camera parameters are output as is.

[0151] The encoding unit 113 consists of a mesh encoder 171, a reprojection silhouette generation unit 172, a texture packing unit 173, an offset information addition unit 174, and a texture encoder 175.

[0152] The mesh encoder 171 compresses the mesh data, as described with reference to Figure 14, using a general-purpose mesh compression codec to convert it into mesh encoded data, and outputs it to the transmission unit 114.

[0153] The type of mesh compression codec doesn't matter as long as it can efficiently compress the mesh data.

[0154] The reprojection silhouette generation unit 172 takes the three-dimensional information, mesh data and camera parameters, as input and performs a projection process to project them onto a two-dimensional plane. It calculates the silhouette region (the region on which the subject is projected) where the mesh is projected and outputs a silhouette image consisting of the resulting silhouette region. The processing performed by the reprojection silhouette generation unit 172 is almost the same as generating a depth map in CG rendering, but instead of a depth value, it stores only two values: whether or not the line of sight hits the subject. The silhouette image is output to the subsequent texture packing unit 173 in binary image data format.

[0155] In other words, the 3D model generation unit 112 in Figure 10, as shown in the upper part of Figure 19, is a process that extracts the silhouette of the subject shown in image P12 from image P11 independently for each camera using pure image processing.

[0156] In response, the reprojection silhouette generation unit 172 generates a two-dimensional image data, a silhouette image P62, by projection processing using the three-dimensional information, mesh data P61, and camera parameters as inputs, as shown in the lower part of Figure 19.

[0157] Comparing image P12 and image P62 on the right side of Figure 19, we can see that in image P12, the original silhouette of the subject is preserved along with some of the background remaining as noise, whereas in image P62, almost only the silhouette of the subject has been extracted.

[0158] This is because the silhouette image P12 generated by the silhouette generation unit 151 is generated by image processing using only one image data, which can result in noise, such as the inclusion of areas that are not the subject and are therefore misclassified.

[0159] On the other hand, the image P62, which is the silhouette image of the reprojection silhouette generation unit 172, is generated using mesh data, which is information of a 3D model generated by integrating information from all cameras 121, and is therefore considered to be image data with less noise.

[0160] By utilizing these low-noise silhouette images and consolidating the textures, it becomes possible to increase the infill density.

[0161] The texture packing unit 173 generates an aggregated texture by aggregating the textures of multiple silhouette images by cutting out image data consisting of multiple silhouette images into the shape of the silhouettes and arranging them densely on the aggregated texture so that the silhouette areas do not overlap. At this time, the texture packing unit 173 also includes the UV map texture as one of the areas of the aggregated texture.

[0162] More specifically, the texture packing unit 173 aggregates a single UV map texture and the images cut out by the silhouettes of the cameras 121 into a single texture so that they do not overlap and are as small as possible. The texture packing unit 173 always places the UV map texture as a fixed-size rectangular image in the upper right corner as a fixed position on the aggregated texture. The texture packing unit 173 moves the images of only the cut-out silhouette regions on the aggregated texture by changing their coordinates, and determines whether they overlap with other images of only silhouette regions, and places them in positions where it is confirmed that they do not overlap.

[0163] As mentioned above, it is possible to determine the placement using the cut-out silhouette image as is, but the texture packing unit 173 uses a reduced silhouette image obtained by scaling down the cut-out silhouette image to determine the placement and generate a reduced placement map.

[0164] This processing reduces the density of the images when the silhouette image is reduced, but it also reduces the number of pixels used for overlap detection. This reduces the processing load associated with determining the placement of elements, further lowering the processing load for placement and enabling faster processing speeds.

[0165] More specifically, the texture packing unit 173 reduces the silhouette image P113 to a size of 1 / 2, as shown in Figure 20, to generate a reduced silhouette image P113'. For example, when the silhouette image P113 is reduced to a reduced silhouette image P113' that is 1 / 4 the size in both width and height, the number of pixels in the silhouette image becomes 1 / 16, and a reduction in processing load close to area-proportional can be expected. Furthermore, the offset coordinates of the silhouette image at this time become 4-pixel units, and the lower 2 bits of the offset information are also reduced in both width and height, which has the advantage of reducing the area that needs to be processed in the offset information addition unit 174 in the subsequent stage.

[0166] Then, as shown in Figure 20, the texture packing unit 173 cuts out a silhouette image from the rectangular image P112 and places it based on the reduced placement map to generate the aggregated texture P101. At this time, as shown in Figure 20, the texture packing unit 173 also places the UV map texture P111 so that it is included in the aggregated texture P101.

[0167] However, UV map textures do not necessarily have to be included in aggregated textures, and examples of cases where UV map textures are not included in aggregated textures will be explained in detail later.

[0168] The offset information addition unit 174 adds an offset value as pixel data to the aggregated texture, indicating the position where the silhouette image was drawn onto the aggregated texture. The decoding unit 116, described later, uses this offset information to restore the image data from the aggregated texture.

[0169] More specifically, the offset information addition unit 174 generates offset information, such as that shown in Figure 21, as positional information of the rectangularly cropped image within the aggregated texture.

[0170] In Figure 21, the offset information 221 is labeled "packSize(3840,2160)" at the top, indicating that the resolution of the aggregated texture 211 is 3840 pixels × 2160 pixels.

[0171] In the second and subsequent rows, the offset information for each rectangularly cut texture is recorded.

[0172] More specifically, for example, in the second row, it is written from left to right as "Cam 101:res(4096,2160)src(1040,324,2016,1836)offset(672,612)", indicating that this is a rectangular image extracted from an image captured by camera 121 identified as Cam 101, with a resolution of 4096 pixels × 2160 pixels before extraction, and that the image was extracted in a rectangular shape from the coordinate position (left edge, top edge, width, height) (= (1040,324,2016,1836)) on the original image, and that it is pasted with an offset position (672,612) on the aggregated texture P101 so that the top left position of the extracted rectangular image is at that point. In addition, although not shown in the diagram, information such as the reduction ratio in the case of reduction processing may also be included.

[0173] In the process of packing silhouette images into a aggregated texture, the placement of the silhouette images basically changes every frame, so the offset information will also be different for each frame. However, to improve video compression efficiency, it is possible to adjust the offset information to be the same every few frames, thereby increasing the temporal correlation of the textures during that period.

[0174] The offset information addition unit 174 encodes the offset information 221 and writes it as a two-dimensional code P121 to the edge of the aggregated texture P101, for example, as shown in Figure 22.

[0175] The 2D code, which encodes the offset information, is handled in binary, and the values ​​0 and 1 are written using the white and black values ​​of the pixel (255 and 0 in an 8-bit image), for example, as information for a 4x4 pixel area per bit. Since the aggregated texture P101 is expected to undergo image compression such as AVC or HEVC when passing through the transmission path, it may be made to have a certain size to prevent misjudgment due to compression distortion. Also, in the case of video codecs, color information is prone to distortion, so it may be better to use only white and black in the bit representation and not use color information.

[0176] Furthermore, the offset information 221 does not necessarily need to be encoded and written onto the aggregated texture P101 as a two-dimensional code; the offset information 221 itself may be transmitted via a separate transmission path.

[0177] However, since the offset information 221 encoded in the aggregated texture P101 is written as a two-dimensional code, there is no need to prepare a separate transmission channel, synchronization problems do not occur, and a simple configuration is possible.

[0178] The texture encoder 175 is a general-purpose module also used for transmitting conventional volumetric video. It encodes the aggregated texture P101 with a codec and outputs it to the transmission unit 114. Because the aggregated texture 211 has a high temporal correlation, it can be efficiently compressed with a general-purpose video codec, and common formats such as AVC and HEVC can be used.

[0179] <Encoding Process> Next, the encoding process performed by the encoding unit 113 will be explained with reference to the flowchart in Figure 18.

[0180] In step S131, the reprojection silhouette generation unit 172 generates a silhouette image by cutting out the silhouette of the subject from the image captured by each camera 121 based on the mesh data and camera parameters, and outputs it to the texture packing unit 173.

[0181] In step S132, the texture packing unit 173 performs texture packing processing based on the image data captured by each camera 121, the UV map texture, and the reprojected silhouette image supplied by the reprojection silhouette generation unit 172 to generate an aggregated texture and output it to the offset information addition unit 174. The texture packing processing will be described in detail later with reference to the flowchart in Figure 24.

[0182] In step S133, the offset information addition unit 174 generates offset information 221 based on the aggregated texture, encodes the generated offset information, and generates a two-dimensional code. Then, the offset information addition unit 174 draws the generated two-dimensional code onto, for example, the lower end of the aggregated texture P101 and outputs it to the texture encoder 175.

[0183] In step S134, the texture encoder 175 encodes the aggregated texture P101 on which the two-dimensional code is written using a codec and outputs it to the transmission unit 114.

[0184] In step S135, the mesh encoder 171 compresses the mesh data using a general-purpose mesh compression encoder to convert it into mesh encoded data and outputs it to the transmission unit 114.

[0185] In step S136, the encoding unit 133 outputs camera parameters to the transmission unit 114.

[0186] Based on the mesh data, UV map texture, image data, and camera parameters, mesh encoded data and texture encoded data are generated and output to the transmission unit 114 along with the camera parameters.

[0187] <Texture Packing Process> Next, the texture packing process will be explained with reference to the flowchart in Figure 24.

[0188] In step S151, the texture packing unit 173 adds the UV map texture to the aggregated texture P101. Since the UV map texture is a fixed size and square, it is not included in the position search loop, and may be initially placed at a fixed position in the upper left corner, for example.

[0189] In step S152, the texture packing unit 173 rearranges the silhouette images (reprojected silhouette images) of each camera 121, generated by the reprojected silhouette generation unit 172, in descending order of area. This is because larger area silhouette images have less flexibility in placement, so the placement is determined first, and then the smaller area silhouette images are placed into the gaps of the larger area silhouette regions where placement has been determined, thereby efficiently filling the aggregated texture with silhouette images.

[0190] In step S153, the texture packing unit 173 sets the largest unprocessed reprojected silhouette image as the silhouette image to be processed, and reduces the size of the silhouette image to be processed to generate a reduced silhouette image.

[0191] In step S154, the texture packing unit 173 temporarily places the position of the reduced silhouette image at a new coordinate position (new coordinate) that is different from the position where it was previously placed in the position search, using the coordinates set in the reduced aggregate texture which is composed of reduced silhouette images.

[0192] In step S155, the texture packing unit 173 determines whether the temporarily placed reduced silhouette image overlaps with other already placed reduced silhouette images.

[0193] If, in step S155, it is determined that the temporarily placed reduced silhouette image overlaps with another already placed reduced silhouette image, the process returns to step S154, and the subsequent processing is repeated.

[0194] In other words, in step S155, the process of sequentially changing the position where the temporarily placed reduced silhouette image is placed to the new coordinates and determining whether or not it overlaps with other already placed reduced silhouette images is repeated until it is determined that the temporarily placed reduced silhouette image does not overlap with other already placed reduced silhouette images.

[0195] If, in step S155, it is determined that the temporarily placed reduced silhouette image does not overlap with any other already placed reduced silhouette images, the process proceeds to step S156.

[0196] In step S156, the texture packing unit 173 places the original size silhouette image to be processed before reduction at a coordinate position on the aggregated texture of the original size that corresponds to the position on the new coordinates where the reduced silhouette image is placed without overlapping with other reduced silhouette images.

[0197] In step S157, the texture packing unit 173 checks whether all images have been processed and whether there are any unprocessed reprojected silhouette images.

[0198] If, in step S157, it is determined that not all images have been processed and there are unprocessed reprojected silhouette images, the process returns to step S153 and the subsequent processing is repeated.

[0199] In other words, steps S153 to S157 are repeated until all images have been processed in step S157 and there are no unprocessed reprojected silhouette images left.

[0200] Then, in step S157, if all images have been processed and it is determined that there are no unprocessed reprojection silhouette images, the process proceeds to step S158.

[0201] In step S158, the texture packing unit 173 determines whether the size of the generated aggregated texture is within the target size.

[0202] If it is determined in step S158 that the size of the aggregated texture is not within the target size, the process proceeds to step S159.

[0203] In step S159, the texture packing unit 173 reduces the generated aggregated texture to the target size while maintaining the same aspect ratio in both width and height, and the process ends.

[0204] Furthermore, if it is determined in step S158 that the size of the aggregated texture is within the target size, the process in step S159 is skipped and the process ends.

[0205] Through the above process, multiple textures can be efficiently aggregated at a high density, making it possible to generate a fixed-size aggregated texture regardless of the size of the subject or the number of cameras.

[0206] This makes it possible to aggregate multiple image data into a highly area-efficient aggregated texture, thereby reducing the processing load on the aggregated texture and improving transmission efficiency.

[0207] As a result, it becomes possible to achieve efficient playback processing of virtual viewpoint images.

[0208] In the above, we have described an example where, if the size of the generated aggregated texture does not fit within the target size, a fixed-size aggregated texture is generated by scaling it down. However, it is also possible to divide the texture so that the resulting size is smaller than the fixed size.

[0209] In other words, in the generated aggregated texture, multiple textures are aggregated sufficiently efficiently, so even if the aggregated texture is divided into several parts, it is still possible to reproduce a sufficiently efficient virtual viewpoint image.

[0210] <Decoding Unit> Next, an example of the configuration of the decoding unit 116 will be described with reference to Figure 25.

[0211] The decoding unit 116 consists of a mesh decoder 251, a texture decoder 252, an offset information reading unit 253, and a texture extraction unit 254.

[0212] The mesh decoder 251 decodes the mesh encoded data encoded by the mesh encoder 171 of the encoding unit 113 and restores it to the original mesh data.

[0213] The texture decoder 252 decodes the texture encoding data encoded by the texture encoder 175 of the encoding unit 113 and restores the aggregated texture with offset information. The restored aggregated texture data is, for example, the aggregated texture P101 described with reference to Figures 20 and 22.

[0214] The offset information reading unit 253 reads and decodes the offset information encoded in the two-dimensional code contained in the aggregated texture P101, restores the offset information 221, and supplies it to the texture extraction unit 254.

[0215] The texture extraction unit 254 extracts the UV map texture and image data from the aggregated texture P101 based on the offset information 221.

[0216] In other words, the texture extraction unit 254, for example as shown in Figure 26, extracts image data P131' from the aggregated texture P101 as a rectangular image, for example, when the coordinates of the offset are Offset(x1, y1), based on the offset information 221, and the coordinate position of the upper left corner is Offset(x1, y1). In Figure 26, the image data P131' contains images of other subjects centered on the subject, but the other subjects can be treated in the same way as the original image data P131 by processing the background.

[0217] Furthermore, the texture extraction unit 254, for example as shown in Figure 26, extracts image data P132' from the aggregated texture P101 as a rectangular image, for example, when the coordinates of the offset are Offset(x2, y2), based on the offset information 221, and the coordinate position of the upper left corner is Offset(x2, y2). In Figure 26, the image data P132' also contains images of other subjects centered on the subject, but similar to the image data P131, the other subjects can be treated in the same way as the original image data P132 by processing the background.

[0218] In the aggregated texture P101 shown in Figure 26, a UV map texture is not included for the sake of simplicity; however, in reality, the configuration includes a UV map texture.

[0219] The decoding unit 116 outputs the camera parameters, along with the decoded mesh data and packing texture data, directly to the subsequent rendering unit 117.

[0220] <Decryption Process> Next, the decoding process by the decoding unit 116 will be explained with reference to the flowchart in Figure 21.

[0221] In step S201, the mesh decoder 251 decodes and decompresses the mesh encoded data encoded by the mesh encoder 171 of the encoding unit 113, and restores it to the original mesh data.

[0222] In step S202, the texture decoder 252 decodes and decompresses the texture encoded data encoded by the texture encoder 175 of the encoding unit 113 to restore the aggregated texture P101.

[0223] In step S203, the offset information reading unit 253 reads and decodes the offset information encoded in the two-dimensional code contained in the aggregated texture P101, restores the offset information 221, and supplies it to the texture extraction unit 254.

[0224] In step S204, the texture extraction unit 254 extracts the UV map texture and image data from the aggregated texture P101 based on the offset information 221.

[0225] In step S205, the decoding unit 116 outputs the camera parameters, along with the decoded mesh data, UV map texture, and restored image data, directly to the subsequent rendering unit 117.

[0226] Through the above processing, the mesh data, UV map texture, and image data supplied as mesh encoded data and texture encoded data are restored and supplied to the rendering unit 117 along with the camera parameters.

[0227] Furthermore, as mentioned above, since the UV map texture is composed of only the silhouette images of the subject efficiently arranged, the texture extraction unit 254 can speed up the image data search process based on offset information compared to a UV map texture in which rectangular images are arranged.

[0228] <Rendering Unit> Next, the detailed configuration of the rendering unit 117 will be described with reference to Figure 28.

[0229] The rendering unit 117 consists of a depth map generation unit 271, a View Dependent drawing processing unit 272, a UV map drawing processing unit 273, a load determination unit 274, a switch 275, and a buffer 276.

[0230] The depth map generation unit 271 generates a depth map of the image data captured by N cameras 121 in a direction close to the rendering viewpoint, based on the mesh data, and outputs it to the View Dependent rendering processing unit 272.

[0231] The number of cameras 121 closest to the rendering viewpoint, N, is set by the load determination unit 274 according to the processing load of the View Dependent rendering processing unit 272.

[0232] The View Dependent drawing processing unit 272 performs the view-dependent drawing process described with reference to Figure 3 based on the mesh data, depth map, image data, and UV map texture, and outputs the drawing result to the buffer 276 via terminal 275a of the switch 275.

[0233] The UV map drawing processing unit 273 draws the pixel of interest using a UV map drawing process, which is a view-dependent drawing process as described with reference to Figure 3, and outputs the drawing result to the buffer 276 via terminal 275b of the switch 275.

[0234] The load determination unit 274 determines whether the processing load of the View Dependent rendering processing unit 272 is in a state where it can handle real-time processing. If it exceeds a predetermined threshold ThA, which is the upper limit at which real-time processing is considered impossible, it reduces N, the number of cameras 121 in the direction closest to the rendering viewpoint, by a predetermined value.

[0235] On the other hand, when the processing load of the View Dependent rendering processing unit 272 falls below a predetermined threshold ThB, which is the lower limit at which it can be considered that there is sufficient capacity, the load determination unit 274 increases N, which is the number of cameras 121 in the direction closest to the rendering viewpoint, by a predetermined value.

[0236] Switch 275 is controlled by the View Dependent drawing processing unit 272. When connected to terminal 275a, the View Dependent drawing processing unit 272 outputs the pixel of interest drawn by the view-dependent drawing process to the buffer 276.

[0237] When switch 275 is connected to terminal 275b, the UV map drawing processing unit 273 outputs the pixel of interest drawn by the UV map drawing process, which is a view-independent drawing process, to buffer 276.

[0238] The buffer 276 sequentially stores the drawing results supplied via the switch 275, and after storing the drawing results for one frame, it outputs the drawing results for one frame.

[0239] In other words, the rendering unit 117 renders the pixel of interest using a view-dependent rendering process when it can be considered that the pixel of interest to be drawn is captured in a camera image taken by any of the N cameras, and renders the pixel of interest using a view-independent rendering process, which is a UV map rendering process, when it can be considered that the pixel of interest is not captured in any of the camera images.

[0240] More specifically, the View Dependent drawing processing unit 272, for each of the camera images captured by the N cameras 121 in the direction close to the drawing viewpoint, checks if there is a camera image in which the pixel of interest is considered to be captured, where the absolute difference between the depth, which is the distance between the camera 121 and the pixel of interest to be drawn based on mesh data, and the depth of the pixel of interest in the depth map corresponding to the camera image is smaller than a predetermined value, and the pixel of interest is considered to be captured in the camera image, connects the switch 275 to terminal 275a, and performs its own View Dependent drawing process to draw the pixel of interest by mixing it so that a larger weight is set for the color information of the camera image in the position close to the drawing viewpoint, and outputs and stores it in the buffer 276. This process is called Visibility Check, and for details, please refer to Patent Document 2 (International Publication No. 2024 / 053371).

[0241] On the other hand, the View Dependent drawing processing unit 272, when the absolute difference between the depth, which is the distance between the camera 121 and the pixel of interest to be drawn based on mesh data, and the depth of the pixel of interest in the depth map corresponding to the camera image is greater than a predetermined value, and there is no camera image in which the pixel of interest can be considered to be captured in the camera image, connects the switch 275 to terminal 275b and controls the UV map drawing processing unit 273 to draw the pixel of interest using a UV map drawing process, which is a view-independent drawing process, and outputs the drawing result to the buffer 276 via terminal 275b of the switch 275.

[0242] Furthermore, the processing performed by the depth map generation unit 271, which generates the depth map used by the View Dependent drawing processing unit 272, involves writing all triangular patches to the depth buffer using perspective projection. This process needs to be performed for each camera 121 used for drawing, resulting in a high processing load.

[0243] Furthermore, the View Dependent rendering processing unit 272 determines color values ​​by weighting and averaging the color information from multiple valid cameras 121 at a mixing ratio suitable for the rendering viewpoint, which also results in a high processing load for the View Dependent rendering processing unit 272.

[0244] On the other hand, the processing by the UV map drawing processing unit 273 can determine the color to be drawn simply by accessing the UV texture map with UV coordinates obtained by internally interpolating the UV coordinates recorded for each vertex within the triangular patch, thus requiring less processing.

[0245] Furthermore, the processing performed by the UV map rendering unit 273 is an established general CG technique and is often provided as a standard function in rendering libraries. This rendering method can be processed even by playback devices with relatively low processing power, such as mobile devices.

[0246] This disclosure allows for adjustment of the processing load by selecting View Dependent rendering, which has a high processing load, when the playback device can process the data in real time, and selecting UV map rendering, which has a lower load, when the processing power is insufficient.

[0247] Furthermore, playback processing on the receiving end is often implemented in software, and in such cases, it is desirable for a single program to run on hardware with various processing capabilities.

[0248] Furthermore, the software needs to perform various other processes in the background, requiring a function to constantly adjust the processing dynamically in real time while monitoring the CPU and GPU load.

[0249] The rendering unit 117 can dynamically switch between processing by the View Dependent rendering unit 272 and processing by the UV map rendering unit 273. When the UV map rendering unit 273 is selected, the View Dependent rendering unit 272, along with the depth map generation unit 271 necessary for that processing, can be stopped.

[0250] Because the rendering unit 117 can switch its operation in this way, it is possible to greatly adjust the processing load. This adjustment can be finely controlled in several stages by increasing or decreasing the number of cameras 121 used for View Dependent rendering processing for each frame.

[0251] <Rendering Process> Next, the rendering process by the rendering unit 117 in Figure 28 will be explained with reference to the flowchart in Figure 29.

[0252] In step S221, the View Dependent drawing processing unit 272 sets the unprocessed frame as the frame of interest.

[0253] In step S222, the depth map generation unit 271 and the View Dependent drawing processing unit 272 acquire the drawing viewpoint and the drawing line of sight direction.

[0254] In step S223, the depth map generation unit 271 generates depth maps corresponding to N camera images in the direction closest to the rendering viewpoint based on the mesh data, and outputs them to the View Dependent rendering processing unit 272.

[0255] In step S224, the View Dependent drawing processing unit 272 sets one of the unprocessed pixels of the frame of interest as the pixel of interest.

[0256] In step S225, the View Dependent drawing processing unit 272 determines whether or not the pixel of interest is captured in any of the N camera images.

[0257] If it is determined in step S226 that the pixel of interest is captured in any of the N camera images, the process proceeds to step S206.

[0258] In step S226, the View Dependent drawing processing unit 272 connects the switch 275 to terminal 275a and, using its own View Dependent drawing process, draws the pixel of interest by mixing the color information of the camera image at a position close to the drawing viewpoint so that a larger weight is set for that position, and outputs it to the buffer 276 for storage.

[0259] On the other hand, if it is determined in step S225 that the pixel of interest is not captured in any of the N camera images, the process proceeds to step S227.

[0260] In step S227, the View Dependent drawing processing unit 272 connects the switch 275 to terminal 275b and controls the UV map drawing processing unit 273 to draw the pixel of interest using the UV map drawing process, which is a view-independent drawing process, and outputs the drawing result to the buffer 276 via terminal 275b of the switch 275.

[0261] In step S208, the View Dependent drawing processing unit 272 determines whether or not there are unprocessed pixels in the frame of interest. If there are unprocessed pixels, the process returns to step S224.

[0262] In other words, steps S224 to S228 are repeated until all pixels of the frame of interest have been drawn.

[0263] Then, in step S228, if it is determined that all pixels of the frame of interest have been drawn and there are no unprocessed pixels, the process proceeds to step S229.

[0264] In step S229, the buffer 276 outputs a drawn image for all pixels of the frame of interest that it has stored.

[0265] In step S230, the load determination unit 274 determines whether the processing load of the View Dependent drawing processing unit 272 exceeds a predetermined upper threshold ThA that can be considered to be the limit for which real-time processing is impossible.

[0266] If, in step S230, it is determined that the processing load of the View Dependent drawing processing unit 272 exceeds a predetermined upper threshold ThA that can be considered to be impossible for real-time processing, the process proceeds to step S231.

[0267] In step S231, the load determination unit 274 controls the View Dependent drawing processing unit 272 to reduce the number of cameras 121 in the direction closest to the drawing viewpoint, N, by a predetermined value, thereby reducing the processing load.

[0268] Furthermore, if it is determined in step S230 that the processing load of the View Dependent drawing processing unit 272 does not exceed a predetermined upper threshold ThA which can be considered to be the limit for which real-time processing is not possible, the process proceeds to step S232.

[0269] In step S232, the load determination unit 274 determines whether the processing load of the View Dependent drawing processing unit 272 has fallen below a predetermined lower threshold ThB, which indicates that there is sufficient capacity to handle the processing load.

[0270] In step S232, if it is determined that the processing load of the View Dependent drawing processing unit 272 has fallen below a predetermined lower threshold ThB, which indicates that there is sufficient processing load available, the process proceeds to step S233.

[0271] In step S233, the load determination unit 274 controls the View Dependent rendering processing unit 272 to increase the number of cameras 121 in the direction closest to the rendering viewpoint, N, by a predetermined value, thereby increasing the processing load. Note that the increase in N does not need to be up to the maximum value of the camera images extracted as reprojected silhouette images in the aggregated texture; for example, a limit up to a predetermined value may be imposed.

[0272] Furthermore, if it is determined in step S232 that the processing load of the View Dependent drawing processing unit 272 is not below a predetermined lower threshold ThB, which indicates that there is sufficient processing load available, the processing in step S233 is skipped.

[0273] In step S234, the View Dependent drawing processing unit 272 determines whether there are any unprocessed frames. If it is determined that there are unprocessed frames, the process returns to step S201, and the subsequent processing is repeated until there are no more unprocessed frames.

[0274] Then, if it is determined in step S234 that there are no unprocessed frames, the process ends.

[0275] Through this process, for example, if the processing performance of the playback device is sufficiently high, it becomes possible to generate depth maps of a number of camera images (N) that are closer to the rendering viewpoint. This increases the likelihood of finding a camera image that can be considered to contain the pixel of interest, allowing the View Dependent rendering process to render more pixels, thus enabling high-precision rendering.

[0276] Furthermore, for example, by adjusting the number of camera images (N) in a direction close to the rendering viewpoint according to the processing performance of the playback device, it becomes possible to reduce the processing load by limiting the number of depth maps that are computationally intensive to generate, and then implement view-dependent rendering.

[0277] Furthermore, if the processing performance of the playback device is low, for example, the number of camera images (N) in the direction close to the rendering viewpoint can be set to 0, so that the depth map generation process can be effectively stopped by the depth map generation unit 271.

[0278] Furthermore, because a depth map is not generated in this way, the process becomes one in which there is no camera image showing the pixel of interest. Therefore, it becomes possible to switch to a view-independent rendering process, namely UV map rendering, to render the pixel of interest. This also allows the processing of the view-dependent rendering unit 272 to be stopped, thereby reducing the processing load and enabling the rendering of an appropriate virtual viewpoint image according to the processing performance of the playback device.

[0279] As a result, it becomes possible to add color information according to the processing performance of the playback device, enabling the playback of optimal virtual viewpoint images (free-viewpoint images, volumetric images) on playback devices with various performance levels.

[0280] <<3. Modification>> In the above, we have described an example in which the UV map texture generated by the 3D model generation unit 112 is included in the aggregated texture, but the UV map texture may be excluded from the aggregated texture.

[0281] The upper part of Figure 30 shows an example of aggregated texture P151 that does not include a UV map texture. That is, as shown in aggregated texture P151 in Figure 30, when a UV map texture is not included, the UV coordinate information in the UV map texture that was originally linked to the vertices of the mesh data becomes unusable. Instead, the pixel coordinates of the image used for View Dependent rendering, which is drawn onto the aggregated texture, are reassigned to correspond to the UV coordinates and used for UV map rendering.

[0282] In other words, this method searches for the optimal location for applying a UV map texture within the image data of the aggregated texture P151, at the triangular patch level of the mesh data of the 3D model constituting the subject, and assigns UV coordinates. Since the image data is a perspective projection and the application using a UV map texture is an affine transformation, a slight error occurs, but under typical volumetric data imaging conditions, the error is sufficiently small and does not pose a problem. Furthermore, even if the coordinates of the pixels in the image used for View Dependent rendering, which are drawn on the aggregated texture and correspond to the UV coordinates, are reassigned at the triangular patch level of the mesh data of the 3D model constituting the subject, the data format remains the same, so the decoding playback process can be performed as is.

[0283] More specifically, pixels that are assigned to UV coordinates in units of triangular patches of mesh data of the 3D model constituting the subject, and that can be used for UV map rendering, exist on multiple image data on the aggregated texture P151. Then, from among the pixels that can be used for UV map rendering that exist on multiple image data, the optimal pixel is assigned, for example, as shown in aggregated texture P151' in Figure 30, and the subject is rendered using the assigned pixel, as shown in image P152 of Figure 30.

[0284] In other words, in Figure 30, pixels Px201 to Px204 that constitute the subject in image P152 are each assigned pixels P201 to P204 in aggregated texture P151'. Therefore, in image P152, it is shown that pixels Px201 to Px204 that constitute the subject are rendered using pixels P201 to P204 assigned in aggregated texture P151'.

[0285] Thus, the aggregated texture does not need to include a UV map texture, which reduces the area of ​​the aggregated texture. Furthermore, there is no need to create a UV map texture in the 3D model generation unit 112; it is only necessary to use the image data for the View Dependent rendering process to assign coordinates to pixels that can be used for UV map rendering, thus reducing the overall processing load.

[0286] As a result, it becomes possible to achieve more efficient rendering and playback of virtual viewpoint images.

[0287] <<4. Description of a computer using this technology>>

[0288] The series of processes described above can be executed by hardware or by software. When the series of processes are executed by software, the programs that make up that software are installed on a computer. Here, "computer" includes computers built into dedicated hardware, as well as general-purpose personal computers, for example, that can perform various functions by installing various programs.

[0289] Figure 31 is a block diagram showing an example of the hardware configuration of a computer that executes the series of processes described above using a program.

[0290] In a computer, the processing circuit 1001, ROM (Read Only Memory) 1002, and RAM (Random Access Memory) 1003 are interconnected by a bus 1004.

[0291] An input / output interface 1005 is further connected to the bus 1004. An input / output interface 1005 is connected to an input unit 1006, an output unit 1007, a storage unit 1008, a communication unit 1009, and a drive 1010.

[0292] The input unit 1006 may include physical or virtual operating means that the user operates to input information, such as a keyboard, mouse, or touch panel, as well as means that the user inputs information through voice, eye gaze, etc. Furthermore, the input unit 1006 may include sensors for inputting various physical quantities into the computer. For example, the input unit 1006 may include sensors that acquire physical quantities such as light (including infrared light other than visible light) or sound, such as a camera or microphone. Also, for example, the input unit 1006 may include sensors that acquire other physical quantities such as temperature, moisture content, acceleration, and distance. The output unit 1007 may include means that present information to the user by stimulating the user's perception, such as a display, speaker, or haptic device. The storage unit 1008 is composed of a hard disk, non-volatile or volatile memory, etc., and stores various information (including programs). The communication unit 1009 is a network interface, etc., and performs wired or wireless communication with the outside. The drive 1010 drives removable media 1011 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0293] The processing circuit 1001 includes a processor that executes programs such as a CPU (Central Processing Unit) and a DSP (Digital Signal Processor). The processing circuit 1001 (its processor) performs the series of processes described above by loading the program stored in the memory unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004 and executing it. The processing circuit 1001 can output the processing results of the series of processes from the output unit 1007, for example, via the bus 1004 and the input / output interface 1005, as needed. The processing circuit 1001 can also store the processing results in the memory unit 1008 or transmit them from the communication unit 1009.

[0294] The program executed by the computer (processing circuit 1001) can be provided by recording it on a removable medium 1011, such as a package medium. The program can also be provided via wired or wireless transmission media, such as a local area network, the internet, or digital satellite broadcasting.

[0295] In a computer, a program can be installed in the storage unit 1008 via the input / output interface 1005 by inserting the removable media 1011 into the drive 1010. Alternatively, a program can be received by the communication unit 1009 from another device, such as a server, via a wired or wireless transmission medium, and installed in the storage unit 1008. Furthermore, programs can be pre-installed in the ROM 1002 or the storage unit 1008.

[0296] The programs executed by the computer may be programs that are processed chronologically in the order described herein, or they may be programs that are processed in parallel or at necessary times, such as when a call is made.

[0297] The processes that a computer performs according to a program do not necessarily have to follow the order described in the flowchart. In other words, the processes that a computer performs according to a program include processes that are executed in parallel or individually (e.g., parallel processing and object-based processing).

[0298] The program may be processed by a single computer (processor), or it may be processed in a distributed manner by multiple computers. Furthermore, the program may be transferred to a remote computer and executed there.

[0299] When the above-described series of processes are performed by a computer executing a program, the processing circuit 1001 (its processor) functions as a 3D model generation unit 112, an encoding unit 113, a decoding unit 116, and a rendering unit 117 by executing the program.

[0300] In this specification, a system means one component or a collection of multiple components (devices, modules (parts), etc.). Therefore, one or more components of a computer, for example, only the processor, or a combination of the processor and memory, for example, only the processing circuit 1001, or a combination of the processing circuit 1001 to the bus 1004, etc., constitute a system. Regarding a collection of multiple components, it is not necessary whether all components reside in the same enclosure or not. Therefore, multiple devices housed in separate enclosures and connected via a network, or a single device containing multiple modules within a single enclosure, are all systems. Furthermore, for example, the entire computer, or a combination of a computer and other devices such as a server (not shown), also constitute a system.

[0301] <<5. Application Examples>> The technology relating to this disclosure can be applied to a variety of products and services.

[0302] (5-1. Content Creation) For example, new video content may be created by combining the 3D model of the subject generated in this embodiment with 3D data managed on another server. Also, for example, if background data acquired by a camera 121 such as a Lidar exists, content can be created in which the subject appears to be in the location indicated by the background data by combining the 3D model of the subject generated in this embodiment with the background data.

[0303] Furthermore, the video content may be three-dimensional video content or two-dimensional video content converted to two dimensions. The 3D model of the subject generated in this embodiment may be, for example, a 3D model generated by the 3D model generation unit or a 3D model reconstructed by the rendering unit 117 (Figure 7).

[0304] (5-2. Experience in Virtual Space) For example, in a virtual space where users communicate as avatars, the subjects (e.g., performers) generated in this embodiment can be placed. In this case, users can view live-action subjects in the virtual space as their avatars.

[0305] (5-3. Application to communication with remote locations) For example, by transmitting a 3D model of a subject generated by the 3D model generation unit 112 (Figure 7) to a remote location via the transmission unit 114 (Figure 7), a user in the remote location can view the 3D model of the subject through a playback device located there. For example, by transmitting this 3D model of the subject in real time, the subject and the user in the remote location can communicate in real time. For example, the subject may be a teacher and the user a student, or the subject may be a doctor and the user a patient.

[0306] (5-4. Others) For example, it is possible to generate free-viewpoint videos of sports and other events based on the 3D models of multiple subjects generated in this embodiment, and individuals can also distribute their own 3D models generated in this embodiment to a distribution platform. In this way, the contents of the embodiments described herein can be applied to various technologies and services.

[0307] Furthermore, for example, the program described above may be executed on any device. In that case, the device should have the necessary functional blocks and be able to obtain the necessary information.

[0308] Furthermore, for example, each step of a flowchart may be executed by one device, or it may be divided among multiple devices. Additionally, if a single step includes multiple processes, these processes may be executed by one device, or they may be divided among multiple devices. In other words, multiple processes included in one step can be executed as multiple steps. Conversely, processes described as multiple steps can be combined and executed as a single step.

[0309] Furthermore, for example, a program executed by a computer may be structured so that the steps of the program are executed chronologically in the order described herein, or they may be executed in parallel or individually at necessary times, such as when a call is made. In other words, the steps may be executed in an order different from the order described above, as long as no inconsistencies arise. Moreover, the steps of this program may be executed in parallel with the processing of other programs, or in combination with the processing of other programs.

[0310] Furthermore, for example, the various technologies relating to this disclosure can be implemented independently, as long as they do not conflict with each other. Of course, any combination of the disclosures can also be implemented. For example, some or all of the disclosures described in one embodiment can be implemented in combination with some or all of the disclosures described in another embodiment. Also, some or all of the aforementioned disclosures can be implemented in combination with other technologies not described above.

[0311] Furthermore, this disclosure may also take the following configurations: <1> An information processing system comprising a texture packing unit that generates an aggregated texture by packing silhouette images consisting of the silhouette of the subject, which are generated from each of the multiple image data based on camera parameters and mesh data of a three-dimensional model of the subject used when imaging the subject with multiple cameras and generating a virtual viewpoint image from multiple image data captured by the multiple cameras. <2> The information processing system according to <1>, further comprising a silhouette image generation unit that projects the subject onto a two-dimensional plane based on the mesh data and the camera parameters, and generates the silhouette image by cutting out the silhouette region on which the subject is projected from the image data, wherein the texture packing unit densely packs the silhouette images generated from each of the multiple image data to generate the aggregated texture. <3> The information processing system according to <2>, wherein the texture packing unit generates the silhouette images from each of the plurality of image data, and in order of increasing area, searches for a position within the aggregated texture where the silhouette images do not overlap with other silhouette images, changing their positions, and places them at the searched positions, thereby densely packing the silhouette images and generating the aggregated texture. <4> The information processing system according to <3>, wherein the texture packing unit reduces the silhouette images by a predetermined magnification to generate reduced silhouette images, and within the reduced aggregated texture, which is reduced by the predetermined magnification, searches for a position within the aggregated texture where the silhouette images do not overlap with other reduced silhouette images, changing their positions, and places the silhouette images at a position within the aggregated texture corresponding to the searched position, thereby densely packing the silhouette images and generating the aggregated texture. <5> The information processing system according to <4>, wherein the predetermined magnification is a power of 2. <6> The information processing system according to <1>, further comprising an offset information writing unit that writes offset information for each silhouette image within the aggregated texture to the aggregated texture.<7> The information processing system according to <6>, wherein the offset information writing unit encodes the offset information for each silhouette image within the aggregated texture into a two-dimensional code and writes it to a predetermined position in the aggregated texture. <8> The information processing system according to <1>, further comprising a UV map generation unit that generates a UV map texture based on the mesh data, the image data, and the camera parameters, wherein the texture packing unit packs the UV map texture in addition to the silhouette images to generate the aggregated texture. <9> The information processing system according to <1>, wherein the texture packing unit reduces the aggregated texture to the predetermined size or smaller when the generated aggregated texture is larger than a predetermined size. <10> The information processing system according to <1>, wherein the texture packing unit divides the aggregated texture into the predetermined size or smaller when the generated aggregated texture is larger than a predetermined size. <11> The information processing system according to <8>, further comprising: an offset information reading unit that reads offset information for each silhouette image written in the aggregated texture; a texture cutting unit that cuts out the image data including the silhouette image and the UV map texture from the aggregated texture based on the offset information read by the offset information reading unit; and a rendering unit that generates the virtual viewpoint image based on the image data, mesh data, camera parameters, and UV map texture cut out from the aggregated texture by the texture cutting unit.<12> The rendering unit includes a view-dependent rendering unit that renders a pixel of interest in the virtual viewpoint image by view-dependent rendering, and a view-independent rendering unit that renders the pixel of interest in the virtual viewpoint image by view-independent rendering, and the system renders the virtual viewpoint image by switching between the view-dependent rendering unit and the view-independent rendering unit depending on whether the pixel of interest exists in the image data captured by the camera that is close to the rendering viewpoint direction of the virtual viewpoint image, and rendering the pixel of interest. <13> The rendering unit includes a depth map generation unit that generates a depth map corresponding to the image data captured by N cameras close to the drawing viewpoint direction based on the mesh data, and the viewpoint-dependent drawing unit determines whether the pixel of interest exists in the image data captured by the cameras close to the drawing viewpoint direction of the virtual viewpoint image based on whether there exists any image data among the image data of the N cameras in which the absolute difference between the distance between the camera and the pixel of interest based on the mesh data and the distance at the pixel of interest in the depth map corresponding to the image data is smaller than a predetermined difference threshold. <14> The viewpoint-dependent drawing unit determines that if there is image data in which the distance difference is smaller than the predetermined difference threshold, the pixel of interest is in the image data captured by the camera that is close to the drawing viewpoint direction of the virtual viewpoint image, and determines that if there is no image data in which the distance difference is smaller than the predetermined difference threshold, the pixel of interest is not in the image data captured by the camera that is close to the drawing viewpoint direction of the virtual viewpoint image, as described in <13>.<15> The information processing system according to <12>, wherein the viewpoint-dependent drawing unit draws the pixel of interest by viewpoint-dependent drawing itself if the pixel of interest exists in the image data captured by the camera near the drawing viewpoint direction, and controls the non-viewpoint-dependent drawing unit to draw the pixel of interest by non-viewpoint-dependent drawing if the pixel of interest does not exist in the image data captured by the camera near the drawing viewpoint direction. <16> The rendering unit includes a processing load determination unit that determines the processing load of the viewpoint-dependent drawing unit, wherein if the processing load determination unit determines that the processing load of the viewpoint-dependent drawing unit exceeds an upper limit that can be considered impossible to process in real time, it decreases N, which is the number of depth maps generated in the depth map generation unit, by a predetermined value, and if the processing load of the viewpoint-dependent drawing unit falls below a lower limit that can be considered sufficient to handle real-time processing, it increases N, which is the number of depth maps generated in the depth map generation unit, by a predetermined value. <17> The information processing system according to <12>, wherein the view-independent drawing is UV map drawing. <18> The information processing system according to <11>, wherein the offset information is encoded in a two-dimensional code and written in the aggregated texture, and the offset information reading unit decodes the two-dimensional code written in the aggregated texture and reads the offset information for each of the silhouette images. <19> A method for operating an information processing system, which includes a texture packing process to generate an aggregated texture by packing silhouette images consisting of the silhouette of the subject, which are generated from each of the multiple image data based on mesh data of a three-dimensional model of the subject and camera parameters, used when imaging a subject with multiple cameras and generating a virtual viewpoint image from multiple image data captured by the multiple cameras.<20> A program that causes a computer to function as a texture packing unit that generates an aggregated texture by packing silhouette images consisting of the silhouette of the subject, which are generated from each of the multiple image data, based on mesh data of a three-dimensional model of the subject and camera parameters, when an image of the subject is captured by multiple cameras and a virtual viewpoint image is generated from the multiple image data captured by the multiple cameras.

[0312] 101 Information processing system, 111 Data acquisition unit, 112 3D model generation unit, 113 Encoding unit, 114 Transmission unit, 115 Receiving unit, 116 Decoding unit, 117, 117' Rendering unit, 118 Display unit, 121, 121-1 to 121-n Camera, 151 Silhouette generation unit, 152 Stereo matching unit, 153 Modeling unit, 154 Meshing unit, 155 UV map generation unit, 171 Mesh encoder, 172 Reprojection silhouette generation unit, 173 Texture packing unit, 174 Offset information addition unit, 175 Texture encoder, 251 Mesh encoder, 252 Texture decoder, 253 Offset information reading unit, 254 Texture extraction unit, 271 Depth map generation unit, 272 View Dependent drawing processing unit, 273 UV map drawing processing unit, 274 Load determination unit, 275 Switch, 276 Buffer

Claims

1. An information processing system comprising a texture packing unit that generates an aggregated texture by packing mesh data, which is a mesh of a 3D model of the subject, and silhouette images, which are generated from each of the multiple image data, based on camera parameters, and which are used when imaging a subject with multiple cameras and generating a virtual viewpoint image from multiple image data captured by the multiple cameras.

2. The information processing system according to claim 1, further comprising a silhouette image generation unit that projects the subject onto a two-dimensional plane based on the mesh data and the camera parameters, and generates the silhouette image by cutting out the silhouette region on which the subject is projected from the image data, wherein the texture packing unit densely packs the silhouette images generated from each of the plurality of image data to generate the aggregated texture.

3. The information processing system according to claim 2, wherein the texture packing unit packs the silhouette images, which are generated from each of the plurality of image data, in order of increasing area, into the aggregated texture, searching for a position within the aggregated texture where they do not overlap with other silhouette images, and placing them in the searched position, thereby densely packing the silhouette images and generating the aggregated texture.

4. The information processing system according to claim 3, wherein the texture packing unit reduces the silhouette image by a predetermined magnification to generate a reduced silhouette image, the aggregated texture searches for a position within the reduced aggregated texture, which has been reduced by the predetermined magnification, that does not overlap with other reduced silhouette images, and places the silhouette image at a position within the aggregated texture that corresponds to the searched position, thereby densely packing the silhouette images and generating the aggregated texture.

5. The information processing system according to claim 4, wherein the predetermined magnification is a power of two.

6. The information processing system according to claim 1, further comprising an offset information writing unit that writes offset information for each silhouette image within the aggregated texture to the aggregated texture.

7. The information processing system according to claim 6, wherein the offset information writing unit encodes the offset information for each silhouette image within the aggregated texture into a two-dimensional code and writes it to a predetermined position in the aggregated texture.

8. The information processing system according to claim 1, further comprising a UV map generation unit that generates a UV map texture based on the mesh data, the image data, and the camera parameters, wherein the texture packing unit packs the UV map texture in addition to the silhouette image to generate the aggregated texture.

9. The information processing system according to claim 1, wherein the texture packing unit reduces the aggregated texture to the predetermined size or smaller when the generated aggregated texture is larger than a predetermined size.

10. The information processing system according to claim 1, wherein the texture packing unit divides the generated aggregated texture into the predetermined size or a smaller size when the aggregated texture is larger than a predetermined size.

11. The information processing system according to claim 8, further comprising: an offset information reading unit that reads offset information for each silhouette image written in the aggregated texture; a texture extraction unit that extracts the image data including the silhouette image and the UV map texture from the aggregated texture based on the offset information read by the offset information reading unit; and a rendering unit that generates the virtual viewpoint image based on the image data, mesh data, camera parameters, and UV map texture extracted from the aggregated texture by the texture extraction unit.

12. The rendering unit includes a view-dependent rendering unit that renders a pixel of interest in the virtual viewpoint image by view-dependent rendering, and a view-independent rendering unit that renders the pixel of interest in the virtual viewpoint image by view-independent rendering, and the system renders the virtual viewpoint image by switching between the view-dependent rendering unit and the view-independent rendering unit depending on whether the pixel of interest exists in the image data captured by the camera near the rendering viewpoint direction of the virtual viewpoint image, thereby rendering the pixel of interest.

13. The information processing system according to claim 12, wherein the rendering unit includes a depth map generation unit that generates a depth map corresponding to the image data captured by N cameras close to the drawing viewpoint direction based on the mesh data, and the viewpoint-dependent drawing unit determines whether the pixel of interest exists in the image data captured by the cameras close to the drawing viewpoint direction of the virtual viewpoint image, based on whether there exists image data among the image data of the N cameras in which the distance difference, which is the absolute difference between the distance between the camera and the pixel of interest based on the mesh data and the distance at the pixel of interest in the depth map corresponding to the image data, is smaller than a predetermined difference threshold.

14. The information processing system according to claim 13, wherein the viewpoint-dependent drawing unit determines that if there is image data in which the distance difference is smaller than the predetermined difference threshold, the pixel of interest is in the image data captured by the camera close to the drawing viewpoint direction of the virtual viewpoint image, and determines that if there is no image data in which the distance difference is smaller than the predetermined difference threshold, the pixel of interest is not in the image data captured by the camera close to the drawing viewpoint direction of the virtual viewpoint image.

15. The information processing system according to claim 12, wherein the viewpoint-dependent drawing unit draws the pixel of interest by viewpoint-dependent drawing itself when the pixel of interest is located in the image data captured by the camera located near the drawing viewpoint direction, and controls the non-viewpoint-dependent drawing unit to draw the pixel of interest by non-viewpoint-dependent drawing when the pixel of interest is not located in the image data captured by the camera located near the drawing viewpoint direction.

16. The information processing system according to claim 13, wherein the rendering unit includes a processing load determination unit that determines the processing load of the viewpoint-dependent rendering unit, and the processing load determination unit determines that the processing load of the viewpoint-dependent rendering unit exceeds an upper limit that can be considered impossible to process in real time, and decreases the number of depth maps, N, generated in the depth map generation unit by a predetermined value, by a predetermined value, and determines that the processing load of the viewpoint-dependent rendering unit falls below a lower limit that can be considered sufficient to handle the real-time processing, and increases the number of depth maps, N, generated in the depth map generation unit by a predetermined value.

17. The information processing system according to claim 12, wherein the view-independent drawing is UV map drawing.

18. The information processing system according to claim 11, wherein the offset information is encoded in a two-dimensional code and written in the aggregated texture, and the offset information reading unit decodes the two-dimensional code written in the aggregated texture and reads the offset information for each silhouette image.

19. A method for operating an information processing system that includes a texture packing process to generate an aggregated texture by packing silhouette images consisting of the silhouette of the subject, which are generated from each of the multiple image data, based on mesh data of a three-dimensional model of the subject and camera parameters, used when imaging a subject with multiple cameras and generating a virtual viewpoint image from multiple image data captured by the multiple cameras.

20. A program that causes a computer to function as a texture packing unit that generates an aggregated texture by packing silhouette images consisting of the silhouette of the subject, which are generated from each of the multiple image data, based on mesh data of a three-dimensional model of the subject and camera parameters, when imaging a subject with multiple cameras and generating a virtual viewpoint image from the multiple image data captured by the multiple cameras.

Images