Generation device, reproduction device, information processing method, and program

Abstract

This generation device generates, for a zone including part of a three-dimensional model of an object, a depth image having a pixel value corresponding to the distance between each point indicating the surface of the shape of the three-dimensional model included in the zone when viewed from a virtual viewpoint and the virtual viewpoint. When a point at the boundary surface of the zone as viewed from the virtual viewpoint is present inside the shape of the three-dimensional model in the generation of the depth image, the generation device complements the pixel value of the depth image corresponding to the point by using a complementary value and generates a depth image for a plurality of virtual viewpoints.

Description

Generation device, playback device, information processing method, and program

[0001] The present invention relates to a technique for encoding the three-dimensional shape of an object.

[0002] A technology for constructing virtual viewpoint images is known that enables a subject to be observed from any virtual viewpoint based on images captured synchronously by multiple imaging devices, each located at a different position. For example, data representing the three-dimensional shape of the subject is generated based on the captured images, the subject is reconstructed based on the data, and the virtual viewpoint image is generated by converting the virtual space into a two-dimensional image. Computer-generated objects may also be placed in the virtual space. Patent Document 1 describes a technology for capturing images using multiple imaging devices to acquire depth information and color information corresponding to the position and line of sight of each imaging device, and then acquiring a three-dimensional model showing the three-dimensional shape and color of the subject based on the information. Patent Document 1 also describes extracting a portion of a virtual space and generating three-dimensional shape data for a subject residing in that portion of the space.

[0003] Special Publication No. 2020-522803

[0004] The three-dimensional shape of an object placed in a virtual space can be represented as a block of voxels. Here, when voxels with a side length of 3 millimeters (mm) are used and each side is represented with a bit depth of 10 bits, the three-dimensional shape of an object up to 3072 mm on a side can be represented. That is, if the size of an object is limited to approximately 3 meters (m) per side, the shape of the object can be represented with a resolution of 3 mm using 30-bit data. On the other hand, objects larger than this size can be represented by, for example, increasing the voxel size, but this reduces the resolution and may make it impossible to represent a detailed three-dimensional shape. Furthermore, increasing the bit depth may have a significant impact on the entire system, such as requiring equipment replacement.

[0005] The present invention provides a technique for appropriately representing the three-dimensional shape of a large-sized object.

[0006] A generating device according to one aspect of the present invention has a generating means for generating a depth image for a zone including a portion of a three-dimensional model of an object, the depth image having pixel values ​​corresponding to the distance between each point indicating the surface of the shape of the three-dimensional model included in the zone when viewed from a virtual viewpoint and the virtual viewpoint, and if a point on the boundary surface of the zone when viewed from the virtual viewpoint is located inside the shape of the three-dimensional model, the generating means complements the pixel value of the depth image corresponding to the point using an complement value, and generates the depth images for multiple virtual viewpoints.

[0007] According to the present invention, it is possible to appropriately represent the three-dimensional shape of a large-sized object.

[0008] Other features and advantages of the present invention will become apparent from the following description taken in conjunction with the accompanying drawings, in which the same or similar elements are designated by the same reference numerals.

[0009] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments of the present invention, and together with the description, are used to explain the principles of the present invention. FIG. 1 is a diagram illustrating an example of the configuration of an information processing system. FIG. 2 is a diagram illustrating an example of the hardware configuration of an apparatus. FIG. 3 is a block diagram illustrating an example of the functional configuration of a data generation apparatus. FIG. 4 is a diagram illustrating an example of the flow of a data generation process. FIG. 5A is a diagram illustrating division of a 3D model and setting of a virtual viewpoint. FIG. 5B is a diagram illustrating division of a 3D model and setting of a virtual viewpoint. FIG. 6 is a diagram illustrating an example of the flow of a depth information generation process. FIG. 7 is a diagram illustrating an example of the flow of a depth information complementation process. FIG. 8 is a diagram illustrating an example of a depth image. FIG. 9 is a diagram illustrating an example of the flow of a texture information generation process. FIG. 10 is a diagram illustrating an example of the flow of a texture information complementation process. FIG. 11 is a diagram illustrating example of encoded data. FIG. 12 is a block diagram illustrating an example of the functional configuration of a playback apparatus. FIG. 13 is a diagram illustrating an example of the flow of a 3D model generation process. FIG. 14 is a diagram illustrating an example of the flow of a zone integration process. FIG. 15 is a diagram illustrating an example of the flow of a depth information generation process. FIG. 16A is a diagram illustrating an example of a depth information complementation method. FIG. 16B is a diagram illustrating an example of a depth information complementing method. FIG. 17 is a diagram illustrating an example of encoded data. FIG. 18 is a diagram illustrating an example of a depth image. FIG. 19A is a diagram illustrating an example of encoded data. FIG. 19B is a diagram illustrating an example of encoded data. FIG. 20A is a diagram illustrating an example of the flow of a three-dimensional model generation process. FIG. 20B is a diagram illustrating an example of the flow of a three-dimensional model generation process. FIG. 21 is a diagram illustrating an example of the flow of a data generation process. FIG. 22A is a diagram illustrating an example of the flow of a process for generating depth information and texture information. FIG. 22B is a diagram illustrating an example of the flow of a process for generating depth information and texture information. FIG. 23 is a diagram illustrating an example of the flow of a process for determining the presence or absence of data before a zone. FIG. 24A is a diagram illustrating an example of the flow of a three-dimensional model generation process. FIG. 24B is a diagram illustrating an example of the flow of a three-dimensional model generation process. FIG. 25 is a diagram illustrating an example of encoded data. FIG. 26 is a block diagram illustrating another example of the functional configuration of a data generation device. FIG. 27 is a diagram illustrating an example of a boundary surface determination image. FIG. 28 is a diagram illustrating the difference between points inside and outside a three-dimensional model.Fig. 29A is a diagram showing another example of the flow of depth information generation processing. Fig. 29B is a diagram showing another example of the flow of depth information generation processing. Fig. 30A is a diagram showing another example of the flow of texture information generation processing. Fig. 30B is a diagram showing another example of the flow of texture information generation processing.

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] (System Configuration) FIG. 1 shows an example configuration of an information processing system according to this embodiment. This information processing system includes, for example, a data generating device 102 and a playback device 122. In this information processing system, the data generating device 102, which may be, for example, a server, converts 3D model information into a format that can be transmitted over a network and outputs the converted information. In this embodiment, the data generating device 102 extracts a space containing objects in a 3D scene from data of the scene based on images captured using, for example, multiple image capturing devices. Note that, hereinafter, the extracted space may be referred to as a "zone." The data generating device 102 then represents the objects included in the extracted space using depth images and texture images, encodes the information, and outputs the image. A depth image is, for example, an image having pixel values ​​that can identify the distance between a viewpoint (image capturing device) and a point on the surface of the object. A texture image is, for example, an image that can identify the color of an object as viewed from the viewpoint. For example, a zone containing a player may be extracted from the space captured by capturing an image of a sports game, and a depth image and texture image that can identify the player's 3D shape data may be generated and encoded. The playback device 122 acquires the encoded data and reconstructs the three-dimensional shape data of the object. Generally, the shape of a three-dimensional shape model is estimated using known viewpoint position information and depth information obtained from a depth image, and coloring can be performed using texture information specified by a texture image. Note that, hereinafter, information on the distance between a specific point on a three-dimensional model and a virtual viewpoint is referred to as depth information, and an image obtained by two-dimensionally mapping the depth information at each point on the three-dimensional model is referred to as a depth image. Note that the term depth information may also refer to the pixel value of a specific pixel in a depth image. Note that texture information on a specific point on a three-dimensional model is referred to as texture information, and an image obtained by two-dimensionally mapping the texture information at each point on the three-dimensional model is referred to as a texture image. Note that the pixel value of a specific pixel in a texture image may also be referred to as texture information.

[0012] The playback device 122 can recreate a three-dimensional scene in the virtual space by placing the reconstructed three-dimensional shape data at an appropriate position within the virtual space. Based on the scene recreated in the virtual space, a virtual viewpoint image can be generated, which is an image observed from any viewpoint (virtual viewpoint) within the space, for example, by tracking objects, analyzing the scene, and rendering a two-dimensional image. The playback device 122 of this embodiment can be included, for example, in an information terminal used to observe the virtual viewpoint image. That is, instead of transmitting the rendered virtual viewpoint image to the information terminal, encoded data can be transmitted by the data generation device 102, and the information terminal can recreate a three-dimensional model and generate a virtual viewpoint image based on the data. This is just one example, and the entire configuration of FIG. 1 can be included in a single device such as a server. In this case, for example, from among captured objects, only the three-dimensional shape data of only the objects to be used in generating the virtual viewpoint image is extracted, and corresponding depth images and texture images are generated and encoded and stored, for example, in the storage unit 111. The encoded data can then be used in subsequent playback of the three-dimensional scene. In this case, data in the form of depth images and texture images before encoding for transmission may be stored in the storage unit 111. Furthermore, for example, the storage unit 111 can store encoded information in order to provide encoded data to an information terminal or the like in a timely manner.

[0013] The data generating device 102 generates depth images and texture images that enable identification of three-dimensional shape data of objects present within the extracted zones, and encodes the information. In this case, for example, when objects are densely packed, it is difficult to individually extract the densely packed objects, and it is assumed that the densely packed objects are treated as a single large object. Furthermore, the size of a single object may be large. When the three-dimensional shape of an object is represented at a constant bit depth, it is assumed that if the size of the object increases beyond a certain level, the resolution of the depth information and texture information corresponding to the object will decrease. Furthermore, as the resolution of the depth information and texture information decreases, the resolution of the reproduced three-dimensional shape data will also decrease accordingly. In this embodiment, in consideration of such circumstances, when a large object is present, the data generating device 102 extracts a space containing the object into multiple zones. The data generating device 102 then generates and outputs depth images and texture images that enable identification of a portion of the three-dimensional shape data of the object for each zone. On the other hand, the playback device 122 restores the 3D shape data of the corresponding part of the object for each of the multiple zones based on the generated depth image and texture image, thereby making it possible to suppress a decrease in resolution due to the large size of the object.

[0014] The data generating device 102 includes, as its functions, a data generating unit 103 and an encoding unit 104. The data generating unit 103 acquires three-dimensional model data from the three-dimensional model storage unit 101. The three-dimensional model storage unit 101 stores three-dimensional model data of a scene generated based on image information and distance information obtained by computer graphics or imaging using multiple imaging devices. The format of the stored data is not particularly limited and may be point cloud or mesh data. The data generating unit 103 specifies, for example, a zone from which an object is to be extracted from the acquired three-dimensional model data and sets a virtual viewpoint surrounding the zone. The data generating unit 103 then acquires depth information representing the distance from the virtual viewpoint to a point where the object exists and texture information representing color information of the object's surface. The data generating unit 103 generates data including the depth information and texture information, as well as virtual viewpoint information obtained when the information was acquired. The virtual viewpoint information is information that enables the position, direction, posture, angle of view, etc. of the set virtual viewpoint to be identified. The encoding unit 104 encodes the data generated by the data generating unit 103. Hereinafter, the encoded data may be referred to as "encoded data." The functions and operations of the data generating device 102 will be described in detail below. The generated encoded data is transmitted via a transmitting unit 105. Note that while FIG. 1 shows an example in which the three-dimensional model storage unit 101 and the transmitting unit 105 exist outside the data generating device 102, the data generating device 102 may include at least one of the three-dimensional model storage unit 101 and the transmitting unit 105.

[0015] The encoded data transmitted by the transmitting unit 105 is received by the receiving unit 121. The encoded data transmitted by the transmitting unit 105 may be stored in, for example, the storage unit 111. The encoded data received by the receiving unit 121 is input to the playback device 122.

[0016] The playback device 122 includes, as its functions, a data playback unit 123 and a 3D model reconstruction unit 124. The data playback unit 123 decodes received encoded data to play back the data, thereby acquiring virtual viewpoint information, a depth image, and a texture image. The 3D model reconstruction unit 124 then reconstructs 3D shape data of the object from the acquired information. The reconstructed 3D shape data is supplied to the output unit 125 and is used, for example, to generate a virtual viewpoint image in the output unit 125. Details of the function and operation of the playback device 122 will be described later. Note that while FIG. 1 shows an example in which the receiving unit 121 and the output unit 125 are located outside the playback device 122, the playback device 122 may include at least one of the receiving unit 121 and the output unit 125.

[0017] (Hardware Configuration) FIG. 2 shows an example of the hardware configuration of an apparatus (at least one of the data generation apparatus 102 and the playback apparatus 122) according to this embodiment. The apparatus is implemented using, for example, a general-purpose computer such as a server or an information terminal, and is configured to include a CPU 201, RAM 202, ROM 203, an operation unit 204, an output unit 205, an external storage device 206, and an interface (I / F 207). These elements are interconnected, for example, via a bus 208. Note that CPU stands for Central Processing Unit, RAM stands for Random Access Memory, and ROM stands for Read Only Memory. Note that the configuration of FIG. 2 is an example, and the apparatus may have only some of the above-described elements, or may include additional elements in addition to the above-described elements. Furthermore, the apparatus may be configured without using a general-purpose computer, for example, using a dedicated processing circuit.

[0018] The CPU 201 controls the entire device using computer programs and data stored in the RAM 202 and the ROM 203, and executes the various processes described below. The CPU 201 is one type of processor, and an MPU (Micro Processing Unit) may be used instead of or in addition to the CPU 201. A multi-core processor may also be used as the CPU 201. That is, in one example, the device is configured to include one or more processors. The RAM 202 has an area for temporarily storing computer programs and data loaded from the ROM 203 or the external storage device 206, data acquired from the outside via the I / F 207, and the like. Furthermore, the RAM 202 provides a work area used by the CPU 201 when executing various processes. That is, the RAM 202 may be allocated as, for example, a frame memory, and may also be used to store various data such as zone setting information and virtual viewpoint information. The RAM 202 can also manage depth information and texture information (depth images and texture images). The ROM 203 stores setting data for the device, a boot program, etc. The device is not limited to the RAM 202 and the ROM 203, and may be configured to include one or more storage devices such as memories.

[0019] The operation unit 204 includes a keyboard, a mouse, and the like, and accepts operation inputs from a user. The operation unit 204 may supply various instructions corresponding to the accepted operation inputs to the CPU 201. The output unit 205 includes, for example, a visual or auditory information output mechanism and is configured to provide various information from the device to the user. The output unit 205 may present information visually via a display or light-emitting diode (LED), or audibly via a speaker. The output unit 205 may also present information in another form, such as vibration. The operation unit 204 and the output unit 205 may be implemented as an integrated device, for example, using a touch panel display. The operation unit 204 and the output unit 205 may also be an interface for accepting inputs via an external operation device and presenting information via an external information output device. In other words, the device does not need to include a device for inputting or presenting information.

[0020] The external storage device 206 is, for example, a large-capacity information storage device such as a hard disk drive. The external storage device 206 can store an OS (operating system) and computer programs for causing the CPU 201 to execute the various functions described below. The external storage device 206 may also store image data to be processed. The external storage device 206 may also store imaging information. The I / F 207 is an interface for connecting the device to a network such as a LAN (local area network) or the Internet. In one example, the device can be connected to an external device such as a projection device or a display device via the I / F 207. The device provides various information to the external device or acquires various information from the external device via the I / F 207.

[0021] (Configuration and processing flow of data generating device) Next, an example of the functional configuration of the data generating device 102 will be described with reference to Fig. 3. The data generating device 102 is configured to include, for example, a three-dimensional model acquisition unit 301, a zone setting unit 302, a virtual viewpoint setting unit 303, a depth information generation unit 304, and a texture information generation unit 305 as functions of the data generating unit 103. Each of these functions can be implemented, for example, by the CPU 201 executing a program stored in the external storage device 206 or the ROM 203. However, this is just one example, and dedicated hardware corresponding to at least some of these functions may be provided.

[0022] The three-dimensional model acquisition unit 301 is connected to the three-dimensional model storage unit 101. The three-dimensional model acquisition unit 301 acquires desired three-dimensional model data from the three-dimensional model storage unit 101. The zone setting unit 302 sets zones, which are units for encoding the acquired three-dimensional model. Information about the set zones is input to the virtual viewpoint setting unit 303, the depth information generation unit 304, the texture information generation unit 305, and the encoding unit 104. The virtual viewpoint setting unit 303 sets virtual viewpoints in units of zones set by the zone setting unit 302. The virtual viewpoint setting unit 303 generates virtual viewpoint information, such as the three-dimensional position, orientation (direction of the virtual camera), angle of view, and principal point (center of the virtual camera image), of the virtual viewpoint for each zone. The virtual viewpoint setting unit 303 then outputs the virtual viewpoint information to the depth information generation unit 304, the texture information generation unit 305, and the encoding unit 104.

[0023] The depth information generation unit 304 acquires three-dimensional model data divided into zones based on the data acquired from the three-dimensional model acquisition unit 301 and the information about the zones set by the zone setting unit 302. The depth information generation unit 304 then generates depth information from the three-dimensional model data for each zone based on the virtual viewpoint information set by the virtual viewpoint setting unit 303, and outputs the depth information to the encoding unit 104. The texture information generation unit 305 also acquires three-dimensional model data divided into zones based on the data acquired from the three-dimensional model acquisition unit 301 and the information about the zones set by the zone setting unit 302. The texture information generation unit 305 then generates texture information from the three-dimensional model data for each zone based on the virtual viewpoint information set by the virtual viewpoint setting unit 303, and outputs the texture information to the encoding unit 104.

[0024] The zone setting information, virtual viewpoint information, depth information, and texture information generated by the data generating device 102 are input to the encoding unit 104. The encoding unit 104 encodes this information to generate encoded data. Details of this encoded data will be described later.

[0025] Next, an example of the flow of data generation processing executed by the data generating device 102 will be described with reference to Figure 4. First, for example, the data generating device 102 accepts a user operation, causing the 3D model acquisition unit 301 to select data to be processed from among the 3D models stored in the 3D model storage unit 101 (S401). Then, the 3D model acquisition unit 301 acquires information about the 3D model to be processed from the 3D model storage unit 101 (S402). Here, information about the position and size of the 3D model is acquired. Then, the zone setting unit 302 determines the size of the zone, which is the unit of encoding, based on the size of the 3D model.

[0026] Here, the size of a zone is determined depending on the voxel resolution and the bit depth of the depth information. For example, assume that the bit depth is 10 bits and the voxel resolution is 0.5 cm. Here, the voxel resolution is defined by the diagonal length of the cube that constitutes the voxel. That is, here, the longest distance between two points in the space occupied by one voxel is referred to as the resolution. As described above, if the voxel resolution is 0.5 cm and the bit depth is 10 bits, a cube with a diagonal length of 512 cm (i.e., a cube of approximately 295 cm x 295 cm x 295 cm) can be set as a zone. Note that in this embodiment, an expression such as x x y x z is used to indicate the size of a space in which the lengths of the horizontal axes, the X-axis and the Y-axis, are x and y, and the vertical length is z. Assuming a zone of this size, it is assumed that a three-dimensional model corresponding to, for example, one person, can be represented by one zone. On the other hand, when people are crowded together, such as in a rugby scrum, it is not always easy to separate the people and generate separate 3D models, and it is assumed that they will be treated as a single large object. In such cases, it is assumed that the 3D model of the object will not fit into a single zone. It is also assumed that the 3D model of an inherently large object, such as a vehicle, will not fit into a single zone. When the size of the 3D model exceeds the size of the zone, it is necessary to divide the zone. The zone setting unit 302 determines how many zones need to be set based on the size of the 3D model to fit the 3D model (S403). For example, if the 3D model of the object is 6m x 4m x 2m, the zone setting unit 302 determines that 2 x 2 x 1 = 4 zones, each approximately 295cm x 295cm x 295cm, as described above, will be set.

[0027] Here, a case where data related to a three-dimensional model 503 is generated will be described with reference to FIG. 5A . When the zone setting unit 302 determines that the size of the three-dimensional model 503 is too large to fit into a single zone, the zone setting unit 302 sets, for example, two zones (zone 501 and zone 502) that fit the three-dimensional model 503. The three-dimensional model 503 is then divided into the two zones. The division may be performed, for example, with the center of the three-dimensional model as the boundary, or so that the left and right sides are the same size. However, these are merely examples, and the division may be performed based on other criteria. For example, the division may be performed so that the cross-sectional area of ​​the divided object is minimized. In this example, the three-dimensional model 503 is divided into two at the position indicated by the dotted line 504. The zone setting unit 302 generates and outputs zone setting information for the set zones. The zone setting information includes, for example, information indicating (or at least identifying) the number of zones, the three-dimensional coordinates representing the position of each zone, and the size of the zone. The position of a zone may be indicated by the position of a point representing the zone as viewed from the origin of the three-dimensional space in which the zone exists. For example, if an arbitrary point in three-dimensional space is expressed using coordinates on the X, Y, and Z axes, the point in the area of ​​the zone that is closest to the origin of the coordinate system (X, Y, Z) = (0, 0, 0) may be determined as the point representing the zone. Alternatively, the center point of the zone may be determined as the point representing the zone. The size of the zone may be indicated, for example, by the lengths of the X, Y, and Z axes of the zone. Alternatively, for example, if the shape of the zone is a rectangular parallelepiped and the point representing the zone is the closest point to the origin, the size of the zone may be indicated by the point farthest from the origin of the coordinate system from the point representing the zone. In other words, the position and size of a zone may be expressed in various formats as long as another device, such as a playback device, can identify the shape of the zone from that information.

[0028] Returning to FIG. 4 , in S404, the virtual viewpoint setting unit 303 acquires information regarding the set zones and the division of the three-dimensional model. Thereafter, the virtual viewpoint setting unit 303 executes a virtual viewpoint determination process for all zones (S405 to S408). In the example of FIG. 4 , a counter for counting the zones for which processing has been completed is initialized to 0 (S405). The virtual viewpoint setting unit 303 increments the counter each time the virtual viewpoint determination process for one zone is completed (S407) (S408). Then, when the incremented counter reaches the number of zones determined in S403 (YES in S406), the virtual viewpoint setting unit 303 ends the loop process of S406 to S408. In the process of S407, the virtual viewpoint setting unit 303 sets virtual viewpoints for virtually capturing images of the target zone from multiple directions. For example, if a rectangular parallelepiped zone is set, six virtual viewpoints may be set, each positioned on a normal line passing through the center of each face of the rectangular parallelepiped, with a line of sight from that position toward the center of the zone. In this case, the angle of view and the position of the virtual viewpoint may be set so as to be able to capture the entire zone or the entire three-dimensional model to be processed. However, this is merely one example, and the virtual viewpoints may be set in any manner as long as it is possible to simulate a state in which an imaging device (camera) is positioned so as to be able to capture the entire surface of the three-dimensional model to be processed without omission.

[0029] FIG. 5B shows an example of setting a virtual viewpoint. FIG. 5B shows a portion of a three-dimensional model 503 included in the zone 501 shown in FIG. 5A. The virtual viewpoint setting unit 303 can set virtual viewpoints 511 to 516, for example, on normals passing through the centers of the respective faces of the rectangular parallelepiped of the zone 501. At this time, the line of sight of each of the virtual viewpoints 511 to 516 is directed toward the center of the zone 501. The angle of view and the distance between the virtual viewpoints 511 to 516 and the zone 501 can be set so that the virtual cameras arranged at the virtual viewpoints 511 to 516 capture the entire zone 501 and the entire three-dimensional model to be processed. The virtual viewpoint setting unit 303 also sets a virtual viewpoint for the zone 502 in a similar manner. The virtual viewpoint setting unit 303 supplies information about the virtual viewpoints set in this manner to the depth information generation unit 304, the texture information generation unit 305, and the encoding unit 104. The position of the virtual viewpoint is expressed, for example, as a relative position from a point representing the zone in which the virtual viewpoint is set. However, this is just one example, and the position of the virtual viewpoint may be expressed, for example, as a position from the origin of the three-dimensional space in which the zone exists, or in any format that allows the position of the virtual viewpoint to be identified on another device such as a playback device.

[0030] 4, after the virtual viewpoints have been determined for all zones, the depth information generating unit 304 generates depth information (S409), and the texture information generating unit 305 generates texture information (S410).

[0031] The process performed by the depth information generation unit 304 when generating depth information in S409 of FIG. 4 will now be described with reference to FIG. 6 . First, the depth information generation unit 304 acquires zone setting information from the zone setting unit 302 (S601). The depth information generation unit 304 then executes the processes of S604 to S616 for each of the set zones. Furthermore, because the following process is repeatedly executed for each virtual viewpoint and pixel, the depth information generation unit 304 executes initialization processing to count these (S602). The depth information generation unit 304 then begins processing for the first zone (NO in S603). The depth information generation unit 304 executes the processes of S605 to S614 for each virtual viewpoint set in each zone. In this process, the depth information generation unit 304 acquires virtual viewpoint information for the virtual viewpoint to be processed (S605). The depth information generation unit 304 also acquires 3D model data of the object included in the target zone from the 3D model acquisition unit 301 (S606). The acquired 3D model data is a 3D model of the entire target object. In other words, no cutting or other processing is performed based on the zone. The acquired 3D model is assumed to be a point cloud, but is not limited to this. In other words, the 3D model may be expressed in any format as long as the 3D shape of the object can be identified. The depth information generation unit 304 then performs the processes of S608 to S612 on all pixel values ​​of the depth image obtained for the target zone and virtual viewpoint. The depth information generation unit 304 acquires the position of a target point, which is a point in the point cloud of the 3D model data that is projected onto the target pixel position (S608). Note that the target point here refers to a point on the surface of the 3D model as seen from the target virtual viewpoint, regardless of whether it is inside or outside the zone. The depth information generation unit 304 then determines whether the target point is located within the target zone (S609).If the depth information generation unit 304 determines that the position of the target point is included in the processing zone (YES in S609), it sets a value corresponding to the distance from the virtual viewpoint to the target point as the pixel value (depth information) of the depth image (S610). On the other hand, if the depth information generation unit 304 determines that the position of the target point is not included in the processing zone (NO in S609), it interpolates the depth information using an interpolation process described below and sets the interpolated value as the pixel value of the depth image (S611). Thereafter, the depth information generation unit 304 increments the counter to proceed to the next pixel value (S612) and returns the process to S607.

[0032] Note that the value corresponding to the distance from the virtual viewpoint to the target point is, for example, a value obtained by subtracting the shortest distance from the virtual viewpoint to the zone from the length of the distance from the virtual viewpoint to the target point. In other words, when the distance from the virtual viewpoint to a target point within a zone is the shortest, the pixel corresponding to the target point is set to a pixel value corresponding to a distance of 0. Hereinafter, unless otherwise specified, the pixel value indicating the distance from the virtual viewpoint to the target point is assumed to be the pixel value corresponding to the value obtained by subtracting the shortest distance from the virtual viewpoint to the zone from the length of the distance from the virtual viewpoint to the target point.

[0033] When the depth information generation unit 304 determines based on the pixel value counter that processing has been completed for all pixel values ​​(YES in S607), the processing for the virtual viewpoint to be processed is completed, and therefore the pixel value counter is reset (S613). The depth information generation unit 304 also increments a counter for managing whether processing has been completed for all virtual viewpoints (S614), and returns the processing to S604. When the depth information generation unit 304 determines based on the virtual viewpoint counter that processing has been completed for all virtual viewpoints (YES in S604), the processing for the zone to be processed is completed, and therefore the virtual viewpoint counter is reset (S615). The depth information generation unit 304 also increments a counter for managing whether processing has been completed for all zones (S616), and returns the processing to S603. Here, when the depth information generation unit 304 determines based on the zone counter that processing has been completed for all zones (YES in S603), this means that processing for all zones, virtual viewpoints, and pixels is completed. For this reason, the depth information generating unit 304 outputs depth information generated for a plurality of virtual viewpoints and each pixel for each zone (S617).

[0034] Next, an example of the flow of the interpolation process executed in S611 will be described with reference to Fig. 7. First, the depth information generation unit 304 identifies the minimum distance to a zone for a target point that exists outside the range of that zone (S701). Then, the depth information generation unit 304 sets the identified distance as a pixel value of the depth image (S702).

[0035] This processing will be described using the example of FIG. 5B . The depth information generation unit 304 identifies, for example, target points 521 to 524 as target points on the surface of the three-dimensional model 503 when viewed from the virtual viewpoint 513. Of these, target points 521 and 522 are points outside the zone 501. Therefore, the depth information generation unit 304 complements the pixel values ​​of the depth image corresponding to these target points based on the distance from each target point to the boundary surface 541 of the nearest zone 501. That is, as viewed from the virtual viewpoint 513, the distance to target points 521 and 522 corresponds to the lengths of dotted lines 531 and 532, and these target points are each projected onto the boundary surface 541, and the pixel values ​​of the depth image are determined based on the projected points. In the example of FIG. 5B , the distance (length of solid line 535) between point 525, which corresponds to the foot of the perpendicular line from target point 521 to boundary surface 541, and the virtual viewpoint 513 is set as the pixel value of the depth image for target point 521. Similarly, the distance (length of solid line 536) between point 526, which corresponds to the foot of the perpendicular line from target point 522 to boundary surface 541, and the virtual viewpoint 513 is set as the pixel value of the depth image for target point 522. On the other hand, target point 523 is a point inside zone 501. Therefore, the distance (length of solid line 533) between the virtual viewpoint 513 and target point 523 is set as the pixel value of the depth image for target point 523. Furthermore, target point 524 is a point on boundary surface 541. In this case, target point 524 is treated as a point inside zone 501, and the distance (length of solid line 534) between the virtual viewpoint 513 and target point 524 is set as the pixel value of the depth image for target point 524. This processing can be said to be processing in which a point on the boundary surface 541 that corresponds to a cross section of the three-dimensional model is treated as if the point of the three-dimensional model were on the boundary surface 541 .

[0036] The determination in S609 may be a determination of whether, when the three-dimensional model is viewed from a virtual viewpoint, the shape of the three-dimensional model is cut at the boundary surface of the zone as viewed from the virtual viewpoint due to the division using the zones. That is, even if a point on the surface of the three-dimensional model exists on a line connecting a point on each boundary surface of the zone as viewed from the virtual viewpoint to the virtual viewpoint and within the zone, if the point on the boundary surface of the zone is included inside the three-dimensional model, the point is determined to be outside the zone. The boundary surface of the zone as viewed from the virtual viewpoint here refers to the surface of the zone that can be observed from the virtual viewpoint when an opaque object of the shape of that zone exists. For example, in FIG. 5B , the boundary surface of zone 501 as viewed from virtual viewpoint 513 is only boundary surface 541. For example, the number of intersections between the line connecting the virtual viewpoint and a point on the boundary surface of the zone and the entire surface of the three-dimensional model may be counted, and if the point on the boundary surface exists between the 2n-1th intersection and the 2nth intersection, the point may be determined to be inside the shape of the three-dimensional model. Here, n is an integer greater than or equal to 1, and the first intersection on a line connecting the virtual viewpoint and a point on the boundary surface of the zone is the first intersection that reaches the surface of the three-dimensional model when viewed from a predetermined point away from the three-dimensional model. The second intersection is the intersection point at which the line passes through the interior of the shape of the three-dimensional model from the first intersection and exits the exterior of the shape of the three-dimensional model. That is, the line enters the interior of the shape of the three-dimensional model at an odd-numbered intersection and exits the exterior of the shape of the three-dimensional model at an even-numbered intersection. Therefore, if a point on the boundary surface of the zone exists between the 2n-1th intersection and the 2nth intersection, the point is determined to exist inside the three-dimensional model. Furthermore, if a point on the boundary surface of the zone exists between the 2nth intersection and the 2n+1th intersection, the point is determined to exist outside the three-dimensional model. In this case, interpolation is not necessary even if the point on the three-dimensional model of the entire object exists when viewed from the virtual viewpoint. Throughout this specification, the process of determining whether the point being processed is within a zone can be read as determining whether a point on the boundary surface of the zone when viewed from a virtual viewpoint is a point within the three-dimensional model.

[0037] FIG. 8 shows an example of a depth image generated by the processing shown in FIGS. 6 and 7 , focusing on the virtual viewpoint 513. In the depth image, region 801 is a region in which interpolated pixel values ​​are specified because the position of the surface of the three-dimensional model as viewed from the virtual viewpoint 513 is outside the zone. In this region, the pixel values ​​indicate the distance from the virtual viewpoint 513 to each point on the boundary surface 541. Regions 802 and 803 indicate the distance from the virtual viewpoint 513 to each point on the surface of the three-dimensional model, which is not interpolated because the position of the surface of the three-dimensional model as viewed from the virtual viewpoint 513 is within the zone. Note that in this example, region 802 includes a portion of the three-dimensional model where a point cloud exists on the boundary surface 541. In this manner, a depth image from one virtual viewpoint is generated for a portion of the three-dimensional model cut out by a zone of a certain size. Note that the depth information generation unit 304 can similarly generate depth images as viewed from other virtual viewpoints, thereby acquiring multiple depth images corresponding to multiple viewpoints. Such multiple depth images allow the three-dimensional shape of the object to be reconstructed, cut out by zones.

[0038] Next, the process performed by the texture information generating unit 305 when generating texture information in S410 of FIG. 4 will be described with reference to FIG.

[0039] First, the texture information generation unit 305 acquires zone setting information from the zone setting unit 302 (S901). The texture information generation unit 305 then sets a texture interpolation value for interpolating points outside the zone in the 3D model (S902). While the value is not particularly limited, it can be set to a value that simplifies encoding, such as 0. The texture information generation unit 305 then executes the processes of S905 to S918 for each of the zones that have been set. Because the following process is repeatedly executed for each virtual viewpoint and pixel, the texture information generation unit 305 executes initialization processing to count these (S903). The texture information generation unit 305 then begins processing for the first zone (NO in S904). The texture information generation unit 305 executes the processes of S906 to S916 for each virtual viewpoint set in each zone. During this process, the texture information generation unit 305 acquires virtual viewpoint information for the virtual viewpoint to be processed (S906). The texture information generation unit 305 also acquires data on the three-dimensional model of the object included in the zone to be processed from the three-dimensional model acquisition unit 301 (S907), and further acquires texture information on the surface of the three-dimensional model (S908). The texture information may be, for example, color information for each point included in the point cloud. However, this is not limited to this, and when the color of the surface of the three-dimensional model is configured using patches or the like, color information for each position on the surface of the three-dimensional shape (for example, positions corresponding to each point included in the point cloud) may be used as texture information.

[0040] The texture information generation unit 305 then performs the processes of S910 to S914 on all pixels of the texture image obtained for the target zone and virtual viewpoint. The texture information generation unit 305 acquires the position of a target point, which is a point in the point cloud of the 3D model data that is projected onto the target pixel position (S910). The texture information generation unit 305 then determines whether the target point is within the target zone (S911). If the texture information generation unit 305 determines that the target point is within the target zone (YES in S911), it sets the texture value of the target point as the pixel value of the texture image (S912). On the other hand, if the texture information generation unit 305 determines that the target point is not within the target zone (NO in S911), it interpolates the texture information using an interpolation process described below and sets the interpolated value as the pixel value of the texture image (S913). Thereafter, the texture information generating unit 305 increments the counter to move the process to the next pixel value (S914), and returns the process to S909.

[0041] When the texture information generation unit 305 determines based on the pixel value counter that processing has been completed for all pixel values ​​(YES in S909), the processing for the virtual viewpoint to be processed is completed, and therefore the pixel value counter is reset (S915). The texture information generation unit 305 also increments a counter for managing whether processing has been completed for all virtual viewpoints (S916), and returns the processing to S905. When the texture information generation unit 305 determines based on the virtual viewpoint counter that processing has been completed for all virtual viewpoints (YES in S905), the processing for the zone to be processed is completed, and therefore the virtual viewpoint counter is reset (S917). The texture information generation unit 305 also increments a counter for managing whether processing has been completed for all zones (S918), and returns the processing to S904. Here, when the texture information generation unit 305 determines based on the zone counter that processing has been completed for all zones (YES in S904), this means that processing for all zones, virtual viewpoints, and pixels has been completed. Therefore, the texture information generating unit 305 outputs texture information generated for a plurality of virtual viewpoints for each zone (S919).

[0042] 10 shows the flow of the process of complementing texture information of target points, which is executed in S913. For target points outside the zone, the texture information generation unit 305 replaces the pixel values ​​of the corresponding texture image with the texture complement value set in S902 (S1001). For example, in the example of FIG. 5B , target points 521 and 522 are outside zone 501, so the pixel values ​​of the texture image corresponding to those target points are complemented with the texture complement value. On the other hand, target points 523 and 524 are inside zone 501, so the texture values ​​of those points in the three-dimensional model are used as the pixel values ​​of the texture image corresponding to those points.

[0043] Next, the coded data generated by the coding unit 104 will be described with reference to Fig. 11. The coded data "Volumetric_video_data" includes header information at a predetermined position, such as the beginning of the data, indicating that the data is coded data.

[0044] The encoded data includes zone setting information "Zone_data." Zone_data includes the number of zones to be encoded "number_of_zone" and information about each zone "Zone_information_data." FIG. 11 shows an example where the number of zones is N. One Zone_information_data is prepared for each zone. Zone_information_data includes, for example, zone identification information "zone_ID." Zone_ID is information that can individually identify multiple zones. For example, consecutive numbers may be assigned to multiple zones as zone_ID, or individual names may be given. Zone_information_data further includes the location of the origin of the target zone "zone_location[3]." In this embodiment, a three-dimensional space including a three-dimensional model is represented by a coordinate system consisting of orthogonal coordinate axes (e.g., X-axis, Y-axis, and Z-axis). zone_location[3] may be a coordinate indicating the location of the zone within this three-dimensional space. For example, the coordinate value of the representative point of the zone (e.g., the point closest to the origin in the three-dimensional space) is represented as zone_location[3]. Here, [3] indicates that three elements (coordinate values) for the three axes, X-axis, Y-axis, and Z-axis, are included. Zone_information_data further includes "zone_size[3]," which indicates the size of the zone. This zone_size[3] also includes values ​​for the three axes, X-axis, Y-axis, and Z-axis, for example. For example, zone_size[3] may store values ​​indicating the respective lengths. The length here may be expressed in units such as millimeters (mm) or other units such as the number of voxels. Furthermore, information that can specify the size of a zone in combination with a representative point of the zone, such as the coordinate value of the point farthest from the origin in three-dimensional space, may be indicated in zone_size[3]. Zone_information_data may also include the bit depth of the depth information, "depth_bit_length."

[0045] The encoded data further includes setting information for the virtual viewpoint, "Virtual_view_point_data." One Virtual_view_point_data is also provided for each zone. The Virtual_view_point_data includes "number_of_virtual_view_point," which indicates the number of virtual viewpoints set in the target zone. FIG. 11 shows an example in which there are M virtual viewpoints. The Virtual_view_point_data includes information for each of the M virtual viewpoints. This information includes, for each virtual viewpoint, an identifier, a position, an orientation, an angle of view, a principal point, and the sizes of the depth image and the texture image to be acquired. Information indicating the identifier is "virtual_view_point_ID" and is expressed, for example, by a number or a character string that identifies the virtual viewpoint. Information indicating the position of the virtual viewpoint is "virtual_view_point_location[3]" and is indicated by coordinate values ​​in a three-dimensional space including a three-dimensional model. Here, similar to the zone information, the three-dimensional space including the three-dimensional model is represented by a coordinate system consisting of orthogonal coordinate axes (e.g., X-axis, Y-axis, and Z-axis), and the position of the virtual viewpoint (virtual camera) is specified by coordinate values. Note that here too, [3] indicates that three elements (coordinate values) for the three axes, e.g., X-axis, Y-axis, and Z-axis, are included. Information indicating the attitude of the virtual viewpoint is "virtual_view_point_attitude[3]" and can be expressed, for example, using a quaternion as Q = (0; x, y, z). In this formula, the part to the left of the semicolon is the real part, and x, y, and z represent the imaginary part. As an example, x, y, and z are used as elements of virtual_view_point_attitude[3], which represents the attitude. Information indicating the angle of view is "angle_of_view", and information indicating the position of the principal point is "center_point[3]". Note that the position of the principal point can be indicated by coordinate values ​​in a three-dimensional space including the three-dimensional model. Information on the size of the acquired depth image and texture image is "virtual_view_point_image_size[2]".This information is configured to include two elements that indicate the vertical and horizontal sizes of the image. Note that the order of the information shown in Fig. 11 is an example, and Virtual_view_point_data may be configured in an order other than this.

[0046] The encoded data includes depth image information "Depth_data" including information on depth images from each virtual viewpoint, and texture image information "Texture_data" including information on texture images from each virtual viewpoint. Depth_data stores information on depth images obtained in each zone and each virtual viewpoint. In FIG. 11, the depth image obtained for the mth virtual viewpoint in the nth zone is represented by "Depth_image_data[n;m]". Furthermore, Texture_data stores information on texture images obtained in each zone and each virtual viewpoint. In FIG. 11, the texture image obtained for the mth virtual viewpoint in the nth zone is represented by "Texture_image_data[n;m]". Note that the depth image and texture image are stored as two-dimensional image data, for example, in an array format, with pixel values ​​for the image size. A general still image encoding method such as JPEG can be used to encode this image data, but any encoding method may be used without limitation. Furthermore, for example, the image data may be stored on a server on a network, and a pointer such as a URL (Uniform Resource Locator) for accessing the server may be included in the encoded data as information related to the image data. In this case, the depth information and texture information generated by the data generation unit 103 may be encoded in the encoding unit 104, for example, into the format of a depth image and a texture image, and stored on the server. The server then notifies the encoding unit 104 of pointer information such as a URL for accessing the depth image and the texture image, and the encoding unit 104 generates encoded data using the notified pointer information. Note that, in this embodiment, the image sizes of the depth image and the texture image are the same, but the sizes of these images may be different from each other. In this case, information separately indicating the respective image sizes is included in the encoded data. Furthermore, "virtual_view_point_image_size[2]" indicating the image size may be included as an element of "Depth_data" or "Texture_data".

[0047] The generated coded data Volumetric_video_data is supplied to the transmitting unit 105. The transmitting unit 105 transmits the coded data to another device, such as the playback device 122. Note that, although the present embodiment has described an example in which transmission data to be transmitted to the playback device 122 is generated, the data does not necessarily have to be transmitted. For example, to simplify management of the three-dimensional model, division by zone as described above may be performed. In this case, the coded data obtained as described above may be supplied to the storage unit 111, and the data may be read from the storage unit 111 for subsequent use. This allows data to be managed by zone, and when using only data related to some zones, there is no need to read data related to other zones, thereby enabling efficient system operation.

[0048] In this manner, pixel values ​​of depth images and texture images are interpolated for portions of the object where the surface is absent as a result of the division into zones, thereby enabling these images to be generated and encoded. Furthermore, in the depth image formed in this embodiment, the distance values ​​between the camera and the point tangent to the boundary surface 541 and the adjacent point where the pixel value is interpolated are very close to each other. As a result, a correlation is created between adjacent non-interpolated and interpolated values, making it possible to achieve high coding efficiency in coding based on prediction from surrounding pixel values. Furthermore, interpolation using a predetermined value in the texture image improves pixel prediction and the probability of code occurrence, thereby improving coding efficiency.

[0049] In the present embodiment, an example in which data is generated using a three-dimensional model once stored in the three-dimensional model storage unit 101 has been described, but this is not limiting. For example, the above-described data generation may be performed in parallel while a three-dimensional model is being generated. If a voxel at a zone boundary is not visible from the outside among the voxels of the generated three-dimensional model, the amount of data in the three-dimensional model may be reduced by deleting that voxel. In addition, in the above-described embodiment, an example in which a virtual viewpoint (virtual camera) is placed on a normal line passing through the center of the boundary surface of a zone and the line of sight is set toward the center of the boundary surface of the zone has been described, but this is not limiting. That is, the virtual viewpoint may be placed at any position in three-dimensional space, and the line of sight may be set arbitrarily under the condition that the entire zone (or, in some cases, a part of the zone) is included within the angle of view. In this case, among the surfaces perpendicular to the line of sight, the surface that contacts the zone at the position closest to the virtual viewpoint may be set as the virtual boundary surface, and a depth image may be generated. This is merely an example, and other methods may be used. For points on the surface of a three-dimensional model viewed from a virtual viewpoint that are outside the zone, the point may be moved parallel to the line of sight to identify the point where it intersects with one of the faces on the boundary of the zone, and a depth image may be generated based on the distance between that point and the virtual viewpoint.

[0050] (Configuration and Processing Flow of Playback Device) Next, an example of the functional configuration of the playback device 122 will be described using FIG. 12 . The playback device 122 has a data playback unit 123 and a 3D model reconstruction unit 124, as described above, in order to reconstruct a 3D model based on the encoded data obtained from the data generation device 102 as described above. The playback device 122 has, as functions of the data playback unit 123, for example, a decoding unit 1201, a zone information acquisition unit 1202, a virtual viewpoint information acquisition unit 1203, a depth information acquisition unit 1204, and a texture information acquisition unit 1205. The playback device 122 also has, as functions of the 3D model reconstruction unit 124, for example, a 3D model generation unit 1211 and a zone integration unit 1212. Each of these functions can be implemented, for example, by the CPU 201 executing a program stored in the external storage device 206 or the ROM 203. However, this is just one example, and dedicated hardware corresponding to at least some of these functions may be provided.

[0051] The decoding unit 1201 receives coded data (Volumetric_video_data in FIG. 11 ) via the receiving unit 121, decodes the received coded data, and reproduces each piece of information contained in the data. The decoding unit 1201 extracts and decodes zone setting information (Zone_data) from the coded data to obtain information such as the number of zones, the ID for each zone, the position of the zone, the size of the zone, and the bit depth. This information is input to the zone information acquisition unit 1202. The decoding unit 1201 also extracts and decodes virtual viewpoint information (Virtual_view_point_data). As a result, the decoding unit 1201 obtains information such as the number of virtual viewpoints for each zone, the ID for each virtual viewpoint, the position of the virtual viewpoint, the attitude of the virtual viewpoint, the angle of view, the position of the principal point, and the size of the image. This information is input to the virtual viewpoint information acquisition unit 1203. The decoding unit 1201 also extracts Depth_data and Texture_data to obtain a depth image and a texture image for each virtual viewpoint of each zone. The depth image data is input to a depth information acquisition unit 1204, and the texture image data is input to a texture information acquisition unit 1205.

[0052] The zone information acquisition unit 1202 outputs the input zone setting information to the virtual viewpoint information acquisition unit 1203, the depth information acquisition unit 1204, and the texture information acquisition unit 1205. The virtual viewpoint information acquisition unit 1203 acquires information on the number of zones, the positions of the zones, and the sizes of the zones from the zone information acquisition unit 1202. The virtual viewpoint information acquisition unit 1203 also acquires and stores the ID, position, orientation, angle of view, principal point position, and image size of each virtual viewpoint for each zone from the decoding unit 1201. The depth information acquisition unit 1204 acquires the number of zones from the zone information acquisition unit 1202 and information such as the number of virtual viewpoints and the size of the image from each virtual viewpoint from the virtual viewpoint information acquisition unit 1203. The depth information acquisition unit 1204 acquires and stores depth images for each virtual viewpoint of each zone from the decoding unit 1201 based on the number of zones, the number of virtual viewpoints, and the size of the image. In response to a request from the 3D model generation unit 1211, the depth information acquisition unit 1204 outputs data (depth information) within a depth image of a corresponding virtual viewpoint for a corresponding zone to the 3D model generation unit 1211. The texture information acquisition unit 1205 acquires the number of zones from the zone information acquisition unit 1202 and information such as the number of virtual viewpoints and the size of images from each virtual viewpoint from the virtual viewpoint information acquisition unit 1203. The texture information acquisition unit 1205 acquires and stores texture image data for each virtual viewpoint for each zone from the decoding unit 1201 based on the number of zones, the number of virtual viewpoints, and the image size. In response to a request from the 3D model generation unit 1211, the texture information acquisition unit 1205 outputs data (texture information) within a texture image of a corresponding virtual viewpoint for a corresponding zone to the 3D model generation unit 1211.

[0053] The three-dimensional model generation unit 1211 generates a three-dimensional model for each zone. For example, after acquiring zone information from the zone information acquisition unit 1202, the three-dimensional model generation unit 1211 identifies the number of zones to be considered when generating a three-dimensional model and identifies the positions and shapes of the zones. Furthermore, the three-dimensional model generation unit 1211, for example, designates a specific zone and acquires virtual viewpoint information for the zone from the virtual viewpoint information acquisition unit 1203, thereby acquiring the position and orientation of the virtual viewpoint surrounding the zone, the angle of view, the position of the principal point, and the image size. The three-dimensional model generation unit 1211 then identifies the characteristics of the virtual camera used when the depth image and texture image for the zone to be processed were acquired. The three-dimensional model generation unit 1211 also acquires depth information and texture information for the zone to be processed from the depth information acquisition unit 1204 and the texture information acquisition unit 1205. The three-dimensional model generation unit 1211 then reconstructs a three-dimensional model of the object in the zone based on the identified zone and virtual viewpoint information and the depth information and texture information corresponding to each virtual viewpoint. This 3D model is generated using interpolated values ​​of depth and texture. In this way, the 3D model generation unit 1211 generates a 3D model for each zone. The zone integration unit 1212 integrates the 3D models for each of the multiple zones generated by the 3D model generation unit 1211 to generate and output a 3D model of one object.

[0054] Here, the flow of the 3D model generation process for each zone, executed by the 3D model generation unit 1211, will be described with reference to FIG. 13 . First, the 3D model generation unit 1211 acquires information on the number of zones set for the 3D model to be generated from the zone information acquisition unit 1202 (S1301). Then, the 3D model generation unit 1211 repeatedly executes the processes of S1304 to S1317 for the number of zones included in the 3D model. Furthermore, because the following process is repeatedly executed for each of the virtual viewpoint, depth image, and texture image pixels, the 3D model generation unit 1211 executes initialization processing to count these (S1302). Then, the 3D model generation unit 1211 starts processing for the first zone (NO in S1303). The 3D model generation unit 1211 acquires information on the position, size, and bit depth of the zone to be processed from the zone information acquisition unit 1202 (S1304). The 3D model generation unit 1211 then executes the processes of S1306 to S1315 for each virtual viewpoint set in each zone. In this process, the 3D model generation unit 1211 acquires virtual viewpoint information for that zone from the virtual viewpoint information acquisition unit 1203 (S1306). The 3D model generation unit 1211 also acquires a depth image corresponding to the virtual viewpoint being processed from the depth information acquisition unit 1204 (S1307). For example, the 3D model generation unit 1211 acquires an image such as that shown in FIG. 8 as a depth image for virtual viewpoint 513, and acquires depths for other virtual viewpoints generated based on the same criteria for those virtual viewpoints. The 3D model generation unit 1211 also acquires a texture image corresponding to the virtual viewpoint being processed from the texture information acquisition unit 1205 (S1308).

[0055] Then, the 3D model generation unit 1211 performs the processes of S1310 to S1313 for all pixels of the depth image (and texture image) obtained for the target zone and virtual viewpoint based on the image size acquired from the virtual viewpoint information acquisition unit 1203. The 3D model generation unit 1211 acquires the pixel value (depth information) of the target pixel from the depth image and acquires distance information from the virtual viewpoint (S1310). Then, based on the distance information from the virtual viewpoint, the 3D model generation unit 1211 sets voxels for target points of the 3D shape corresponding to the pixel value of the target pixel (S1311). For example, using the depth image shown in FIG. 8, voxels are set at positions corresponding to the boundary surface for target points 521 and 522 in FIG. 5B. Furthermore, voxels are set at positions corresponding to the surface of the 3D model to be reproduced (positions behind the boundary surface) for target points 523 and 524. The three-dimensional model generation unit 1211 then colors the voxels of the target points based on the corresponding pixel values ​​(texture information) of the texture image (S1312). For example, texture information of a predetermined value is assigned to target points 521 and 522 in FIG. 5B, and texture information assigned to the three-dimensional model is assigned to target points 523 and 524. By repeatedly executing the processes of S1310 to S1313, colored voxels are set at points of the three-dimensional model visible from the virtual viewpoint being processed.

[0056] When the three-dimensional model generation unit 1211 determines based on the pixel value counter that processing has been completed for all pixel values ​​(YES in S1309), the processing for the virtual viewpoint to be processed is completed, and therefore the pixel value counter is reset (S1314). The three-dimensional model generation unit 1211 also increments a counter for managing whether processing has been completed for all virtual viewpoints (S1315), and returns the processing to S1305. When the three-dimensional model generation unit 1211 determines based on the virtual viewpoint counter that processing has been completed for all virtual viewpoints (YES in S1305), the processing for the zone to be processed is completed, and therefore the virtual viewpoint counter is reset (S1316). The three-dimensional model generation unit 1211 also increments a counter for managing whether processing has been completed for all zones (S1317), and returns the processing to S1303. Through the processing up to this point, a three-dimensional model included in the zone to be processed is reconstructed based on information from all virtual viewpoints corresponding to the zone. In the 3D model corresponding to one zone, voxels not included in the original 3D model are set at the boundary surfaces of the zones. Thereafter, when the 3D model generation unit 1211 determines, based on the zone counter, that processing for all zones has been completed (YES in S1303), this means that processing for all zones, virtual viewpoints, and pixels has been completed. Therefore, the 3D model generation unit 1211 outputs the 3D models reconstructed for each zone to the zone integration unit 1212 (S1318).

[0057] Next, an example of the flow of the zone integrating process executed by the zone integrating unit 1212 will be described using FIG. 14 . The zone integrating unit 1212 first acquires the number of zones from the zone information acquiring unit 1202 (S1401). Note that, if the following process is to be executed in the order of the zone IDs, zone ID information may also be acquired. The zone integrating unit 1212 initializes a counter for counting the number of processed zones (S1402) in order to repeatedly execute the processes of S1404 to S1407 for the number of zones. Note that, if the process is to be executed in the order of the zone IDs, the zone IDs may be expressed as numbers such as 0 to N-1. When the zone integrating unit 1212 starts processing the first zone to be processed (NO in S1403), it acquires zone setting information for that zone from the zone information acquiring unit 1202 (S1404). It should be noted that, although it is assumed here that the zone information acquisition unit 1202 notifies the zone integration unit 1212 of the zone setting information separately from the three-dimensional model reproduced by the three-dimensional model generation unit 1211, this is not limitative. For example, the three-dimensional model generation unit 1211 may add the zone setting information to the three-dimensional model of each zone and transmit it to the zone integration unit 1212.

[0058] The zone integration unit 1212 then acquires the reconstructed 3D model for the target zone from the 3D model generation unit 1211 (S1405). Based on the zone position in the zone setting information acquired in S1404, the zone integration unit 1212 integrates the 3D model for the target zone with the previously integrated and stored 3D model (S1406). The zone integration unit 1212 may initially perform initialization so that there is nothing in the space where the 3D model is to be reproduced. When processing the first target zone, the zone integration unit 1212 only performs the process of arranging the 3D model corresponding to that zone in accordance with the zone's position. From then on, the 3D model for the currently target zone is additionally placed in the space where the 3D model for the previous target zone was placed. In this way, the 3D model is updated the number of times equal to the number of zones, thereby completing the integration process for the 3D models of the object. In other words, the 3D model integration process is performed by arranging the 3D models reconstructed for multiple zones in the same space. When the processing for all zones is completed (YES in S1403), the zone integration unit 1212 completes the integration processing.

[0059] Note that the voxels of the 3D model based on the pixel values ​​of the interpolated depth image are voxels on the boundary between adjacent zones that cannot be observed from the surface. In other words, these voxels are internal voxels of the final reconstructed 3D model, and therefore will not be reflected in the image when the 3D model is used to generate a virtual viewpoint image.

[0060] In this way, the three-dimensional model is divided into zones and coded with high quality and high efficiency, which makes it possible for the playback device 122 that receives the coded information to reconstruct the original large-sized three-dimensional model with high accuracy.

[0061] In the above embodiment, an example has been described in which the depth information generation unit 304 of the data generation device 102 performs interpolation such that, when a point on the surface of a three-dimensional model viewed from a virtual viewpoint is outside a zone, the point is treated as a point on a zone boundary surface. In this variant, a method for performing this interpolation using a different method will be described.

[0062] FIG. 15 shows the flow of the depth information generation process executed in S409 of FIG. 4 in this modification. Note that in FIG. 15, processing steps similar to those described in relation to FIG. 6 are denoted by the same reference numerals, and their description will be omitted. In this process, the zone setting unit 302 sets a depth information complement value for complementing depth information (S1501). Here, a range of values ​​indicating distance is set, and a value outside that range is set as the depth information complement value. That is, the depth information complement value is set to a value that is not used as a value indicating the distance between a point of a three-dimensional model within a zone and the virtual viewpoint. Then, instead of complementing the depth information in the process of S611 of FIG. 6, the depth information generation unit 304 complements the depth information using the depth information complement value set in S1501 (S1502). That is, in this modification, for points outside a zone, the points are not mapped to the boundary surface of the zone, and a predetermined value that is not used as a value indicating distance is set as the pixel value of the depth image corresponding to the point.

[0063] 16A and 16B show examples of setting the depth information complement value in this modification. In FIG. 16A , in an example where the bit depth is 10 bits, the distance between the tent of the three-dimensional model in the zone and the virtual viewpoint is represented by a value between "1" and "1023." For example, the distance from the virtual viewpoint 513 to the closest point in the zone is represented by "1," and the distance to the farthest point in the zone is represented by "1023." Then, "0," which is not used as a distance value, is set as the depth information complement value. The zone setting unit 302 outputs the set depth information complement value to the encoding unit 104 together with the zone setting information. Then, as shown in FIG. 17 , the encoding unit 104 generates encoded data including, for example, "default_out_of_zone," which represents the depth information complement value, as an element of Zone_data. Note that, unlike the symbols in Fig. 11, Depth_data and Texture_data are not provided separately in Fig. 17, but a single element called Depth_Texture_data is provided. However, the contents are the same as in Fig. 11 in that information on depth images and texture images for each zone and each virtual viewpoint set in that zone is included.

[0064] For example, when the zone 501 and virtual viewpoint 531 in FIG. 5B are the processing targets, since target points 521 and 522 are not included in zone 501, the pixel values ​​corresponding to these target points in the generated depth image are complemented with a depth information complement value of "0." On the other hand, since target points 523 and 524 are included in zone 501, the pixel values ​​corresponding to these target points are values ​​between 1 and 1023. For example, target point 524, which is located on boundary surface 541, has a pixel value of "1" for the corresponding pixel in the depth image. Target point 523 is a point within the zone away from boundary surface 541, so the pixel value of the corresponding pixel in the depth image is greater than 1. An example of a depth image generated in this manner is shown in FIG. 18. For comparison with FIG. 8, the same reference numerals are used to denote corresponding regions in the depth image. Region 801 is a pixel region corresponding to a region where a point in the three-dimensional model exists outside the zone, and the pixel value for this region is set to "0," which is a depth information complement value. Areas 802 and 803 are pixel areas corresponding to areas where points of the three-dimensional model exist within the zone, and pixel values ​​corresponding to the distance from the virtual viewpoint to the points are set.

[0065] In the above example, the depth information complement value is set to "0," but this is not limiting. For example, a predetermined value such as the maximum or minimum value that can represent the distance within a zone, or another predetermined value, may be set as the depth information complement value. For example, as shown in FIG. 16B , when the bit depth is 10, "1023" may be set to be used as the depth information complement value, and the distance within the zone to each point on the three-dimensional model may be represented by a value within the range from "0" to "1022." The texture information is generated by the method of FIG. 9 .

[0066] In the above example, the zone setting unit 302 sets the depth information complement value, but this is not limiting. For example, the depth information generation unit 304 may set the depth information complement value for each zone. In this case, as shown in FIG. 19A , default_out_of_zone representing the depth information complement value may be included in Zone_information_data indicating information about each zone. Furthermore, the depth information generation unit 304 may set the depth information complement value for each virtual viewpoint. In this case, as shown in FIG. 19B , default_out_of_zone representing the depth information complement value may be included in Virtual_view_point_information_data indicating information about each virtual viewpoint.

[0067] In this way, by dividing the image into zones, depth information and texture information are complemented for the portions of the three-dimensional object model where the surface no longer exists, making it possible to encode depth images and texture images. Furthermore, for depth images, the complemented portions are set to fixed values, which allows for high encoding efficiency in encoding methods that utilize predictions from surrounding pixel values ​​and code occurrence probabilities.

[0068] Next, the process by which the playback device 122 plays back a three-dimensional model from the coded data generated as described above will be described with reference to Figures 20A and 20B. Note that in Figures 20A and 20B, the same processing steps as in Figure 13 are denoted by the same reference numerals, and detailed description thereof will be omitted. Before this process is performed, the decoding unit 1201 acquires the coded data shown in Figure 17 and decodes the zone setting information (Zone_data) to reproduce the number of zones, the ID of each zone, the position of the zone, the size of the zone, the bit depth, and the depth information complement value. It is assumed that this information has been input to the zone information acquisition unit 1202. Note that the decoding unit 1201 acquires a depth image and a texture image from Depth_Texture_data in the coded data of Figure 17. The decoding unit 1201 then outputs the depth image to the depth information acquisition unit 1204 and the texture image to the texture information acquisition unit 1205.

[0069] 20A and 20B , the 3D model generation unit 1211 acquires, from the zone information acquisition unit 1202, information on the number of zones set for the 3D model to be generated, as well as a depth information complement value (S2001). Then, for each pixel in the depth image, the 3D model generation unit 1211 determines whether the pixel value (depth information) is the depth information complement value acquired in S2001 (S2002). If the pixel value of the depth image corresponding to the point to be processed is not the depth information complement value (NO in S2002), the 3D model generation unit 1211 sets a voxel and texture value for that point (S1311, S1312). On the other hand, if the pixel value of the depth image corresponding to the point to be processed is the depth information complement value (YES in S2002), the 3D model generation unit 1211 does not set a voxel and texture value, and proceeds to processing for the next target point (S1313).

[0070] As described above, in this processing example, voxels are not set in portions corresponding to cross sections of the 3D model when it is divided using zones. Then, when the multiple 3D models reproduced for multiple zones are integrated by the zone integration unit 1212, the regions without voxels become regions within the integrated 3D model. That is, in the above-described embodiment, surfaces divided by zones are set within the integrated 3D model, but in this processing example, such surfaces are not set. However, since such surfaces are not externally observable in the reproduced 3D model, the above-described embodiment and this processing example can reproduce 3D models that are externally identical. Furthermore, in this processing example, unnecessary voxels are not generated at the boundary surfaces of the zones, thereby reducing the data volume of the reconstructed 3D model.

[0071] (Variation 2) Variation 1 described an example in which values ​​that are not used as pixel values ​​for indicating the distance between the virtual viewpoint and points within a zone are used as depth information complement values. In the procedure of Variation 1, for example, in the example of FIG. 16A , values ​​from 1 to 1023 are used as pixel values ​​of the depth image indicating the position of a point cloud within a zone, and 0 cannot be used. Also, in the example of FIG. 16B , values ​​from 0 to 1022 are used as pixel values ​​of the depth image indicating the position of a point cloud within a zone, and 1023 cannot be used. In other words, the depth image is expressed in 1023 levels. In contrast, for example, even if the area is divided into zones, if there are no three-dimensional model points in front of the zone (on the virtual viewpoint side), there is no need to use complement values. Therefore, by not using depth information complement values, the quantization granularity can be made finer. For this reason, in this variation, when generating depth information for each zone, the depth information generation unit 304 determines whether there are three-dimensional model points in front of the zone and determines the quantization granularity accordingly. The depth information generation unit 304 also inputs, for example, information indicating whether or not a point of the three-dimensional model exists in front of the zone to the texture information generation unit 305. The texture information generation unit 305 complements the texture information based on the information indicating whether or not a point of the three-dimensional model exists in front of the zone, and generates texture information. This prevents values ​​that can only be used as depth information or texture information from being set when no point of the three-dimensional model exists in front of the zone, making it possible to represent the three-dimensional model with higher resolution.

[0072] 21 illustrates an example of the flow of data generation processing executed by the data generating device 102 in this modification. In this processing, the zones are divided by processing similar to that in FIG. 4, and a virtual viewpoint is determined for each zone, after which depth information and texture information are generated in conjunction with each other (S2101).

[0073] 22A and 22B show an example of the flow of the depth information and texture information generation process executed in S2101. First, the depth information generation unit 304 acquires information on the number of zones from the zone setting unit 302 (S2201). The depth information generation unit 304 executes the processes of S2204 to S2225 for each of the set zones. Furthermore, because the following process is repeatedly executed for each virtual viewpoint and pixel, the depth information generation unit 304 executes initialization processing to count them (S2202). Then, the depth information generation unit 304 starts processing for the first zone (NO in S2203). The depth information generation unit 304 acquires zone setting information for the zone to be processed from the zone setting unit 302 (S2204). For example, the depth information generation unit 304 acquires information "d" indicating the bit depth of the depth information. The depth information generation unit 304 then executes steps S2206 to S2223 for each virtual viewpoint set in each zone. In this process, the depth information generation unit 304 acquires virtual viewpoint information for the virtual viewpoint being processed (S2206). The depth information generation unit 304 also acquires 3D model data of the object included in the zone being processed from the 3D model acquisition unit 301 (S2207). Based on the acquired 3D model, the depth information generation unit 304 then executes a process of determining whether or not a point of the 3D model exists in front of the zone being processed (S2208).

[0074] The processing of S2208 will now be described with reference to FIG. 23 . The processing of FIG. 23 is processing for determining whether at least one point of a three-dimensional model exists in front of the zone to be processed, as viewed from the virtual viewpoint to be processed. The depth information generation unit 304 first acquires virtual viewpoint information for the virtual viewpoint to be processed from the virtual viewpoint setting unit 303 (S2301). The depth information generation unit 304 then initializes a flag indicating whether or not a point of a three-dimensional model exists in front of the zone, as viewed from the virtual viewpoint (S2302). A value of “0” for this flag indicates that no point of a three-dimensional model exists in front of the zone, as viewed from the virtual viewpoint, and a value of “1” indicates that a point of a three-dimensional model exists in front of the zone, as viewed from the virtual viewpoint. In S2302, the value of this flag is set to 0. The depth information generation unit 304 then initializes a counter so that the following confirmation processing can be performed for all pixels of the depth image to be generated for the zone to be processed and the virtual viewpoint (S2303). The depth information generation unit 304 then acquires the position of the target point of the three-dimensional model that is mapped to each pixel (S2305) and determines whether the target point is within the zone being processed (S2306). If the target point is within the zone being processed (YES in S2306), the depth information generation unit 304 increments the counter (S2307) and processes the next pixel. On the other hand, if the target point is outside the zone being processed (NO in S2306), the depth information generation unit 304 determines whether the position is in front of the zone as seen from the virtual viewpoint (S2308). If the target point is not in front of the zone as seen from the virtual viewpoint (NO in S2308), the depth information generation unit 304 increments the counter (S2307) and processes the next pixel. On the other hand, if the position of the target point is in front of the zone as seen from the virtual viewpoint (YES in S2308), the depth information generation unit 304 sets the flag that was set to 0 in S2302 to 1 (S2309). The depth information generation unit 304 then sets a depth information complement value and a texture information complement value (S2310) and ends the processing. Note that if no corresponding point in the three-dimensional model is located in front of the zone for any pixel (YES in S2304), the depth information generation unit 304 ends the processing.In this case, the value of the flag remains 0. Furthermore, the depth information complement value and the texture information complement value are not set because there are no pixels to be complemented by the complement value. Note that the depth information generation unit 304 can notify the texture information generation unit 305 of the above flag, for example.

[0075] 22A and 22B, the depth information generation unit 304 repeatedly executes the processes from S2210 to S2221 for each pixel of the depth image to be generated. First, the depth information generation unit 304 identifies the position of a point on the three-dimensional model corresponding to the pixel to be processed (S2210). Then, the depth information generation unit 304 assigns subsequent processing depending on whether a point on the three-dimensional model exists in front of the zone to be processed (i.e., whether the flag value is 1 or 0) (S2211).

[0076] If there is no point of the three-dimensional model in front of the zone to be processed (i.e., if the flag is 0, NO in S2211), the depth information generation unit 304 makes it possible to express the distance between the virtual viewpoint and the target point using all values ​​of the set bit depth. This quantization using all ranges of values ​​that can be expressed with the set bit depth is called first quantization (S2212). In the first quantization, the value obtained by subtracting the minimum distance between the virtual viewpoint and the zone from the distance between the target point in the zone and the virtual viewpoint is multiplied by 2. ｄ The depth information generation unit 304 then uses the first quantization to obtain depth information corresponding to the target point (S2213). The texture information generation unit 305 sets the texture value of the target point as the pixel value of the texture image (S2214).

[0077] On the other hand, if a point of the three-dimensional model exists in front of the zone to be processed (i.e., if the flag is 1, YES in S2211), the depth information generation unit 304 expresses the distance between the virtual viewpoint and the target point using a range of values ​​excluding the values ​​used for the depth information complement value. This quantization, which uses a part of the range of values ​​expressible at the set bit depth excluding the values ​​used for the depth information complement value, is called second quantization (S2215). In the second quantization, the value obtained by subtracting the minimum distance between the virtual viewpoint and the zone from the distance between the target point in the zone and the virtual viewpoint is calculated as (2 ｄ In the second quantization, if there are k values ​​that are not used to express the distance to the target point, the quantization is performed in (2 ｄ Quantization may be performed at a step of 1-k. Thus, the granularity of the first quantization when there is no point in the 3D model before the target zone (when the flag is 0) is finer than the granularity of the second quantization when there is a point in the 3D model before the target zone (when the flag is 1). In other words, by using a finer quantization granularity when it is not necessary to use an interpolated value, the depth information generation unit 304 obtains depth information corresponding to the distance from the virtual viewpoint to the target point using the second quantization for a target point within the zone (YES in S2216) (S2217). The texture information generation unit 305 sets the texture value of the target point as the pixel value of the texture image (S2218). On the other hand, for a target point outside the zone (NO in S2216), the depth information generation unit 304 sets the interpolated depth information value as the depth information of the pixel corresponding to the target point (S2219). Furthermore, the texture information generating unit 305 sets the texture value corresponding to the target point as the texture information of the pixel corresponding to that target point (S2220). After completing the setting of the depth information and texture information for the target point to be processed, the counter is incremented (S2221).

[0078] Thereafter, when the depth information generation unit 304 completes processing for target points corresponding to all pixels (YES in S2209), it resets the counter that counts the number of pixels (S2222) and increments the virtual viewpoint counter (S2223). Thereafter, when the depth information generation unit 304 completes processing for all virtual viewpoints in the processing target zone (YES in S2205), it resets the virtual viewpoint counter (S2224) and increments the zone counter (S2225). Then, when the depth information generation unit 304 and the texture information generation unit 305 complete processing for all zones, they output the depth information and texture information to the encoding unit 104 (S2226) and complete the processing. The depth information generation unit 304 also outputs the above-mentioned flag information to the encoding unit 104. Furthermore, when the flag is 1, the depth information generation unit 304 and the texture information generation unit 305 can output the complementary depth information value and the complementary texture information value to the encoding unit 104 .

[0079] 23 may operate to set the flag to 1 if a point on the boundary surface of a zone visible from the virtual viewpoint is inside the shape of the three-dimensional model, i.e., if the three-dimensional model is cut at that boundary surface. This is because even if a point of the three-dimensional model is on the front side of a zone, if the three-dimensional model is not cut by the boundary surface of the zone seen from that virtual viewpoint, interpolation processing is not performed. This makes it possible to change the quantization method depending on whether interpolation processing is performed or not.

[0080] The encoding unit 104 generates encoded data in which the input flag and depth information complement value are included in Zone_information_data for each zone. Fig. 25 shows an example of encoded data in this case. In this modification, Zone_information_data includes flag_out_of_zone as the above-mentioned flag and default_out_of_zone as the depth information complement value when the flag is 1. Note that this information may also include texture information complement values.

[0081] Next, the process by which the playback device 122 reconstructs a 3D model using the coded data obtained in this modification will be described with reference to Figures 24A and 24B. Note that among the processes in Figures 24A and 24B, those common to Figure 13 are assigned common reference numerals and will not be described again. The coded data as shown in Figure 25 is decoded by the decoding unit 1201, and zone setting information including a flag (flag_out_of_zone) and information on the zone information complement value when the flag is 1 is output to the zone information acquisition unit 1202. The depth information acquisition unit 1204 acquires the flag information from the zone information acquisition unit 1202 (S2401), and if it is confirmed that the flag is 1 (YES in S2402), acquires the depth information complement value (S2403). The depth information acquisition unit 1204 then performs processing on each zone, each virtual viewpoint, and each pixel. Then, the depth information acquisition unit 1204 switches the processing for the point of the three-dimensional model corresponding to the pixel to be processed depending on whether the flag is 1 or not (S2404).

[0082] If the flag is 0 (NO in S2404), the depth information acquisition unit 1204 ｄ The depth information acquisition unit 1204 identifies the depth information that has been quantized in the previous step and reconstructs the distance information between the virtual viewpoint and the target point by inverse quantization (S2405). The depth information acquisition unit 1204 then sets a voxel based on the distance information (S2406). The texture information acquisition unit 1205 also sets a texture value at the target point using the pixel value of the pixel to be processed in the texture image (S2407).

[0083] On the other hand, if the flag is 0 (YES in S2404), the depth information acquisition unit 1204 determines whether the pixel value of the pixel to be processed is a depth information complement value (S2408). If the depth information acquisition unit 1204 determines that the pixel value of the pixel to be processed is not a depth information complement value (NO in S2408), the depth information acquisition unit 1204 determines whether the pixel value is (2 ｄThe depth information acquisition unit 1204 then determines that the depth information is quantized at the depth information acquisition stage (S2404-1). The depth information acquisition unit 1204 then performs inverse quantization to reconstruct distance information between the virtual viewpoint and the target point (S2409). The depth information acquisition unit 1204 then sets a voxel based on the distance information (S2406). The texture information acquisition unit 1205 then sets a texture value at the target point using the pixel value of the target pixel in the texture image (S2407). On the other hand, if the depth information acquisition unit 1204 determines that the pixel value of the target pixel is a depth information complement value (YES in S2408), it does not set a voxel.

[0084] As described above, in this modification, the quantization granularity of the depth information and texture information for the 3D model belonging to the zone is changed depending on whether or not there is a point of the 3D model of the object in front of the zone as viewed from the virtual viewpoint. As a result, if there is no point of the 3D model in front of the zone, the bit depth can be fully utilized to generate the depth information for the 3D model. This allows the playback device 122 to determine the distance with higher accuracy, thereby improving the accuracy of reconstructing the 3D model.

[0085] Note that the above-described processing examples can be used in any combination. In one example, a depth image may be generated by switching between the procedures described in FIGS. 6 and 7, the procedures described in FIG. 15, and the procedures described in FIGS. 22A and 22B for each virtual viewpoint. In this case, Virtual_view_point_information_data of the encoded data may include information indicating which procedure was used for encoding. Furthermore, the procedure may be switched for each zone, rather than for each virtual viewpoint. In this case, Zone_information_data of the encoded data may include information indicating which procedure was used for encoding. Based on this information, the playback device 122 may reconstruct a three-dimensional model using the corresponding procedure.

[0086] In the above description, a virtual viewpoint is set for each zone, but this is not limited to this. For example, virtual cameras located at positions surrounding a three-dimensional model of a large object may be used as virtual cameras for multiple zones that divide the three-dimensional model. In other words, a common virtual viewpoint may be used for multiple zones. Furthermore, in the above example, an example in which both depth information and texture information are used has been described, but, for example, only depth information may be used. In this case, only the three-dimensional shape of the object is reproduced.

[0087] Although the above description has been given with respect to a case where multiple zones are set, the above processing can also be applied when a single zone is set that includes a portion of a three-dimensional model of an object. For example, if a user desires to observe only a specific portion of a three-dimensional shape, only one zone corresponding to that portion may be set. Furthermore, the zones may be predetermined. For example, a virtual space may be divided into predefined zones. In this case, when a specific object in the virtual space is specified, a zone that includes at least a portion of the object is identified. The virtual viewpoint for each zone may also be determined in advance. In this case, the data generating device 102 may not include the zone setting unit 302 or the virtual viewpoint setting unit 303. Furthermore, in the above processing example, the coded data includes zone information and virtual viewpoint information, but these information do not necessarily have to be included. For example, if virtual viewpoint information is stored in the playback device 122 in advance, depth images for multiple virtual viewpoints may be acquired to reproduce at least the three-dimensional shape of the object. For this reason, the coded data may include depth images corresponding to at least multiple virtual viewpoints, and other information may be optionally included as needed. Furthermore, when no data is transmitted, only depth images may be generated in the data generating device 102. Note that the above-described technique can be applied to both moving images and still images.

[0088] (Variation 3) In the above-described embodiment, when a zone model is viewed from a virtual viewpoint, target points outside the zone are detected and their values ​​are interpolated to obtain a depth image and a texture image. In this variation, regardless of whether point cloud data exists between the virtual viewpoint and the zone model, data or interpolated data of the model within the zone is directly used.

[0089] Fig. 26 shows an example of the functional configuration of a data generating device 102 according to this modification. Components that perform the same functions as those of the data generating device 102 in Fig. 3 are given the same reference numerals, and descriptions thereof will be omitted. In addition to the functional configuration in Fig. 3, the data generating device 102 includes a boundary surface determination image acquisition unit 2601. Furthermore, the data generating device 102 includes a depth information generating unit 2602 and a texture information generating unit 2603 instead of the depth information generating unit 304 and the texture information generating unit 305.

[0090] The boundary surface determination image acquisition unit 2601 generates a boundary surface determination image that indicates whether each point on the boundary surface of the zone as viewed from the virtual viewpoint is a point inside or outside the three-dimensional model. That is, a boundary surface determination image is generated that indicates whether a cross section of the three-dimensional model cut by the boundary surface of the zone can be observed from the virtual viewpoint. For example, as shown in FIG. 5A , a three-dimensional model is divided into zones 501 and 502. In this case, the surface separating zones 501 and 502 is the boundary surface. When zone 501 is observed from virtual viewpoint 513, for example, the boundary surface is observed, and the boundary surface determination image is an image that indicates whether each point on the boundary surface corresponds to the cross section of the three-dimensional model. FIG. 27 shows an example of a boundary surface determination image generated in this manner. As shown in the example of FIG. 27 , the boundary surface determination image, for example, displays the interior (cross section) range 2701 of the three-dimensional model in black, and the exterior range of the three-dimensional model in white. Note that this is just an example, and an image in a format different from that shown in FIG. 27 may be generated. Furthermore, data other than image data may be generated as long as the position of the cross section of the three-dimensional model cut by the boundary surface of the zone can be identified. In this embodiment, a bitmap image with a 1-bit pixel value is generated, with points in black areas having a value of "1" and points in white areas having a value of "0." The boundary surface determination image indicates whether each point on the boundary surface is located inside the three-dimensional model. Therefore, even if a point of the three-dimensional model exists on the boundary surface, that point is a surface point of the three-dimensional model, not an internal point, and therefore the pixel value of that point is "0." For example, in FIG. 28 , points 525 and 526 are internal points of the three-dimensional model, while point 524 is located on boundary surface 541 but is a surface point of the three-dimensional model, not an internal point. Therefore, in the boundary surface determination image, the pixels corresponding to points 525 and 526 are set to "1," and the pixels of points 524 and 523 are set to "0."

[0091] The depth information generation unit 2602 differs from the depth information generation unit 304 described with reference to FIG. 3 in that it receives a boundary surface determination image and uses it for determination. Here, the processing performed by the depth information generation unit 2602 when generating depth information in S409 of FIG. 4 will be described with reference to FIGS. 29A and 29B . In the following description, steps in which the same processing as in FIG. 6 is performed are denoted by the same reference numerals, and their description will be omitted. The depth information generation unit 2602 acquires a boundary surface determination image from the boundary surface determination image generation unit 2601 (S2901). The depth information generation unit 2602 also calculates a distance d from the virtual viewpoint to a boundary surface at a point inside the three-dimensional model (S2902). For example, when the boundary surface and the virtual viewpoint are directly opposed to each other (the normal direction of the boundary surface coincides with the line of sight from the virtual viewpoint), the distance d can be calculated as the distance between the center of the area of ​​the boundary surface that is inside the three-dimensional model and the virtual viewpoint. When the boundary surface and the virtual viewpoint do not face each other directly, the depth information generating unit 2602 generates a distance image having a different distance d for each pixel of the boundary surface determination image.

[0092] The depth information generation unit 2602 then determines, as a target point, a point in the point cloud of the 3D model data that is projected onto the pixel position being processed, and determines whether the target point is projected onto the interior range 2701 of the 3D model in the boundary surface determination image (S2903). If the depth information generation unit 2602 determines that the target point is projected onto the range 2701 (YES in S2903), it acquires the position of the target point on the boundary surface (S2904). Then, the depth information generation unit 2602 determines, as the depth information for the pixel being processed, a value (distance d) corresponding to the distance from the virtual viewpoint to the target point (S2905). If the depth information generation unit 2602 determines that the target point is projected outside the range 2701 (NO in S2903), it can identify the target point as a point on the surface of the 3D model. For this reason, the depth information generation unit 2602 acquires the position of the point to be processed (S2906), and sets a value corresponding to the distance from the position of the virtual viewpoint to the target point as the depth information of the depth image (S2907).

[0093] The texture information generation unit 2603 differs from the texture information generation unit 305 described with reference to FIG. 3 in that it receives a boundary surface determination image and uses it for determination. The processing performed by the texture information generation unit 2603 to generate texture information in S410 of FIG. 4 will be described with reference to FIGS. 30A and 30B . In the following description, steps in which the same processing as in FIG. 9 is performed are denoted by the same reference numerals, and description thereof will be omitted. The texture information generation unit 2603 acquires a boundary surface determination image from the boundary surface determination image generation unit 2601 (S3001). The texture information generation unit 2603 then determines, as a target point, a point projected onto the pixel position of the processing target among the point cloud of the three-dimensional model data, and determines whether the target point is a point projected onto the interior range 2701 of the three-dimensional model in the boundary surface determination image (S3002). If the texture information generation unit 2603 determines that the target point is a point projected onto the range 2701 (YES in S3002), it acquires the position of the target point on the boundary surface (S3003). Then, the texture information generation unit 2603 replaces the pixel value of the texture image corresponding to the target point with the texture interpolation value set in S902 (S3004). On the other hand, if the texture information generation unit 2603 determines that the target point is a point projected outside the range 2701 (NO in S3003), it can identify the target point as a point on the surface of the three-dimensional model. Therefore, the texture information generation unit 2603 acquires the position of the target point (S3005), acquires texture information of the target point, and sets this as the pixel value of the texture image (S3006).

[0094] In this way, by interpolating pixel values ​​of depth images and texture images for portions (cut portions) where the surface of the object no longer exists as a result of the division by zones, it becomes possible to generate and encode these images. Furthermore, in this modification, it is not necessary to consider target points outside the zone when making a determination, and only points within the zone are processed, thereby reducing the amount of data input and output. Furthermore, in this modification, it is possible to perform processing independently for each zone, which simplifies data reading and facilitates parallel processing. The above-described embodiments and modifications can be used in any combination.

[0095] The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., an ASIC) that realizes one or more of the functions.

[0096] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention.

[0097] This application claims priority based on Japanese Patent Application No. 2023-168863, filed September 28, 2023, the entire contents of which are incorporated herein by reference.

Claims

1. A generating device having a generating means for generating a depth image for a zone including a portion of a three-dimensional model of an object, the depth image having pixel values ​​corresponding to the distance between the virtual viewpoint and each of the points indicating the surface of the shape of the three-dimensional model included in the zone when viewed from the virtual viewpoint, wherein the generating means, when a point on the boundary surface of the zone when viewed from the virtual viewpoint is located inside the shape of the three-dimensional model, complements the pixel value of the depth image corresponding to the point using an complement value, and generates the depth images for a plurality of the virtual viewpoints.

2. The generating device according to claim 1, further comprising setting means for setting a plurality of said zones, each of said zones including a portion of said three-dimensional model.

3. The generating device according to claim 2, wherein said setting means further sets a plurality of said virtual viewpoints for each of a plurality of said zones.

4. A generating device according to any one of claims 1 to 3, wherein the generating means uses, when a point of the boundary surface is present inside the shape of the three-dimensional model, a pixel value corresponding to the distance between the virtual viewpoint and the point as the interpolated value.

5. A generating device according to any one of claims 1 to 3, wherein the generating means uses, as the interpolated value, a predetermined value that is not used as a pixel value corresponding to the distance between the virtual viewpoint and the surface of the shape of the three-dimensional model included in the zone when the boundary surface point is located inside the shape of the three-dimensional model.

6. The generating device according to claim 5, wherein the generating means, when any of the points of the boundary surface are present inside the shape of the three-dimensional model, expresses pixel values ​​corresponding to the distance between each of the points indicating the surface of the shape of the three-dimensional model and the virtual viewpoint in a range of values ​​excluding the specified value, and when any of the points of the boundary surface are present outside the shape of the three-dimensional model, expresses pixel values ​​corresponding to the distance between each of the points indicating the surface of the shape of the three-dimensional model and the virtual viewpoint in a range of values ​​including the specified value.

7. A generating device according to any one of claims 1 to 6, wherein the generating means further generates a texture image indicating a texture for the three-dimensional model of the object seen from the virtual viewpoint, and complements pixel values ​​of the texture image for points in the depth image that are complemented using the complement value using a second complement value.

8. A generating device according to any one of claims 1 to 7, further comprising an encoding means for generating encoded data including at least the depth image, the zone setting information, and the virtual viewpoint setting information.

9. A playback device having: an acquisition means for acquiring, for each of a plurality of zones each including a portion of a three-dimensional model of an object, a depth image having pixel values ​​corresponding to the distance between each of a plurality of virtual viewpoints and each of the points indicating the surface of the shape of the three-dimensional model of the portion included in the zone when viewed from each of a plurality of virtual viewpoints, wherein if a point on the boundary surface of the zone when viewed from the virtual viewpoint is inside the shape of the three-dimensional model of the object, the pixel value of the depth image corresponding to the point is complemented using an complement value; a generation means for generating, for each of the plurality of zones, a three-dimensional model of the portion of the object in that zone from the depth image for each of the plurality of virtual viewpoints; and a reconstruction means for reconstructing the three-dimensional model of the object by integrating the three-dimensional models of the portion of the object for each of the plurality of zones generated for each of the plurality of zones.

10. A playback device as described in claim 9, wherein the generation means generates a three-dimensional model of a portion of the object in the zone by setting voxels based on the pixel values ​​of pixels contained in the depth image.

11. The playback device according to claim 10, wherein the generating means does not set a voxel for a pixel in the depth image to which a value that is not used to indicate the distance is set.

12. A playback device according to any one of claims 9 to 11, wherein the acquisition means acquires, together with the depth image, information indicating whether or not a value not used to indicate the distance in the depth image is used as the complementary value.

13. The playback device described in claim 12, wherein the generation means, when information indicating that a value not used to indicate the distance is used as the interpolation value, determines the distance by assuming that the pixel value corresponding to the distance in the depth image is quantized with a first granularity, when information indicating that a value not used to indicate the distance is not used as the interpolation value, determines the distance by assuming that the pixel value corresponding to the distance in the depth image is quantized with a second granularity finer than the first granularity, and generates a three-dimensional model of a portion of the object based on the determined distance.

14. The playback device according to any one of claims 9 to 13, wherein said acquisition means further acquires a texture image indicative of a texture for said three-dimensional model of said object as viewed from said virtual viewpoint.

15. A playback device according to claim 14, wherein the texture image is an image in which pixel values ​​corresponding to points in the depth image that are interpolated using the interpolated value are interpolated using a second interpolated value.

16. A playback device according to any one of claims 9 to 15, wherein the acquisition means acquires encoded data including at least the depth image, the zone setting information, and the virtual viewpoint setting information.

17. An information processing method executed by a generating device, comprising: generating a depth image for a zone including a portion of a three-dimensional model of an object, the depth image having pixel values ​​corresponding to the distance between the virtual viewpoint and each of the points indicating the surface of the shape of the three-dimensional model included in the zone when viewed from the virtual viewpoint; and in generating the depth image, if a point on the boundary surface of the zone when viewed from the virtual viewpoint is present inside the shape of the three-dimensional model, the pixel value of the depth image corresponding to the point is complemented using an complementation value; and generating the depth images for a plurality of the virtual viewpoints.

18. An information processing method executed by a playback device, comprising: acquiring, for each of a plurality of zones each including a portion of a three-dimensional model of an object, a depth image having pixel values ​​corresponding to the distance between each of a plurality of virtual viewpoints and each of points indicating the surface of the shape of the three-dimensional model of the portion included in the zone when viewed from each of a plurality of virtual viewpoints, wherein when a point on the boundary surface of the zone when viewed from the virtual viewpoint is inside the shape of the three-dimensional model of the object, the pixel value of the depth image corresponding to the point is complemented using an complementation value; generating, for each of the plurality of zones, a three-dimensional model of the portion of the object in the zone from the depth image for each of the plurality of virtual viewpoints; and reconstructing the three-dimensional model of the object by integrating the three-dimensional models of the portion of the object for each of the plurality of zones generated for each of the plurality of zones.

19. A program for causing a computer to function as the generating device according to any one of claims 1 to 8.

20. A program for causing a computer to function as the playback device according to any one of claims 9 to 16.

Images