Generation device, reproduction device, information processing method, and program

Abstract

To appropriately represent the three-dimensional shape of an object of large size.SOLUTION: A generation device generates, for a zone including part of a three-dimensional model of an object, a depth image having a pixel value corresponding to the distance between a virtual viewpoint and each point indicating a surface of the shape of the three-dimensional model included in the zone when viewed from the virtual viewpoint. When a point at a boundary surface of the zone as viewed from the virtual viewpoint is present inside the shape of the three-dimensional model in the generation of the depth image, the generation device complements the pixel value of the depth image corresponding to the point using a complementary value and generates a depth image for a plurality of virtual viewpoints.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to a technique for encoding the three-dimensional shape of an object. [Background technology]

[0002] A technique for constructing a virtual viewpoint image is known that enables a subject to be observed from any virtual viewpoint based on images of the subject captured synchronously by a plurality of imaging devices arranged at different positions. For example, data representing the three-dimensional shape of the subject is formed based on a plurality of captured images, the subject reconstructed based on the data is placed in a virtual space, and the virtual space is converted into a two-dimensional image to generate a virtual viewpoint image. Note that objects generated by a computer may also be placed in the virtual space. Patent Document 1 describes a technique for acquiring depth information and color information corresponding to the position and line of sight of each imaging device by imaging using a plurality of imaging devices, and acquiring a three-dimensional model showing the three-dimensional shape and color of the subject based on the information. Patent Document 1 also describes cutting out a part of the virtual space and generating three-dimensional shape data for a subject existing in the part of the space. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2020-522803 Summary of the Invention [Problem to be solved by the invention]

[0004] The three-dimensional shape of an object placed in a virtual space can be expressed as a block of voxels. Here, when voxels with a side length of 3 millimeters (mm) are used to express each side with a bit depth of 10 bits, the three-dimensional shape of an object with a side length of up to 3072 mm can be expressed. That is, when the size of an object is within a size of approximately 3 meters (m) per side, the shape of the object can be expressed with a resolution of 3 mm units using 30-bit data. On the other hand, for objects exceeding this size, they can be expressed, for example, by increasing the size of the voxels, but the resolution decreases and it may become impossible to express a fine three-dimensional shape. In addition, increasing the bit depth may have a large impact on the entire system, for example, in the case where equipment needs to be replaced.

[0005] The present invention provides a technique for appropriately representing the three-dimensional shape of a large-sized object. [Means for solving the problem]

[0006] A generating device according to one aspect of the present invention has a generating means for generating a depth image for a zone including a portion of a three-dimensional model of an object, the depth image having pixel values ​​corresponding to the distance between the virtual viewpoint and each of the points indicating the surface of the shape of the three-dimensional model included in the zone when viewed from the virtual viewpoint, and the generating means complements the pixel value of the depth image corresponding to a point on the boundary surface of the zone when viewed from the virtual viewpoint if the point is located inside the shape of the three-dimensional model using an complementation value, and generates the depth images for multiple virtual viewpoints. Effect of the Invention

[0007] According to the present invention, it becomes possible to appropriately represent the three-dimensional shape of a large-sized object. [Brief description of the drawings]

[0008] [Figure 1]FIG. 1 illustrates an example of a configuration of an information processing system. [Diagram 2] FIG. 2 illustrates an example of a hardware configuration of the apparatus. [Diagram 3] FIG. 2 is a block diagram illustrating an example of a functional configuration of a data generating device. [Figure 4] FIG. 11 illustrates an example of the flow of a data generation process. [Diagram 5] 1A and 1B are diagrams illustrating division of a three-dimensional model and setting of a virtual viewpoint. [Figure 6] FIG. 11 is a diagram illustrating an example of the flow of a depth information generation process. [Figure 7] FIG. 11 is a diagram illustrating an example of the flow of depth information complementation processing. [Figure 8] FIG. 2 is a diagram showing an example of a depth image. [Figure 9] FIG. 11 is a diagram illustrating an example of the flow of a texture information generation process. [Figure 10] FIG. 13 is a diagram illustrating an example of the flow of texture information complementation processing. [Figure 11] FIG. 11 is a diagram illustrating an example of encoded data. [Figure 12] FIG. 2 is a block diagram showing an example of a functional configuration of a playback device. [Figure 13] FIG. 11 is a diagram illustrating an example of the flow of a three-dimensional model generation process. [Figure 14] FIG. 13 illustrates an example of the flow of a zone integration process. [Figure 15] FIG. 11 is a diagram illustrating an example of the flow of a depth information generation process. [Figure 16] 11A and 11B are diagrams illustrating an example of a method for complementing depth information. [Figure 17] FIG. 11 is a diagram illustrating an example of encoded data. [Figure 18] FIG. 2 is a diagram showing an example of a depth image. [Figure 19] FIG. 11 is a diagram illustrating an example of encoded data. [Figure 20] FIG. 11 is a diagram illustrating an example of the flow of a three-dimensional model generation process. [Figure 21] FIG. 11 illustrates an example of the flow of a data generation process. [Figure 22]FIG. 11 is a diagram showing an example of the flow of a process for generating depth information and texture information. [Diagram 23] 13 is a diagram showing an example of the flow of a process for determining whether or not data exists before a zone. [Figure 24] FIG. 11 is a diagram illustrating an example of the flow of a three-dimensional model generation process. [Diagram 25] FIG. 11 is a diagram illustrating an example of encoded data. [Figure 26] FIG. 13 is a block diagram showing another example of the functional configuration of the data generating device. [Figure 27] FIG. 13 is a diagram showing an example of a boundary surface determination image. [Figure 28] FIG. 2 is a diagram illustrating the difference between internal and external points of a three-dimensional model. [Figure 29] FIG. 11 is a diagram showing another example of the flow of the depth information generation process. [Diagram 30] FIG. 11 is a diagram illustrating another example of the flow of the texture information generation process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0010] (System Configuration) FIG. 1 shows an example of the configuration of an information processing system according to this embodiment. This information processing system includes, for example, a data generating device 102 and a reproducing device 122. In this information processing system, the data generating device 102, which is, for example, a server, converts information on a three-dimensional model into a format that can be transmitted over a network and outputs the converted information. In this embodiment, the data generating device 102 cuts out a space including an object in a three-dimensional scene from data on the scene based on an image obtained by imaging using a plurality of imaging devices. In the following, the space cut out at this time may be called a zone. Then, the data generating device 102 expresses the object included in the cut-out space using a depth image and a texture image, encodes the information, and outputs the information. The depth image is, for example, an image having a pixel value that can specify the distance between the viewpoint (imaging device) and a point on the surface of the object, and the texture image is an image that can specify the color of the object seen from the viewpoint. For example, a zone including a player is cut out from a space obtained by imaging a sports game, and a depth image and a texture image that can specify the three-dimensional shape data of the player are generated and encoded. The playback device 122 acquires the encoded data and reconstructs the three-dimensional shape data of the object. In general, the shape of a three-dimensional shape model is estimated using known viewpoint position information and depth information obtained by a depth image, and coloring can be performed using texture information specified by a texture image. In the following, information on the distance between a virtual viewpoint and a specific point of a three-dimensional model is called depth information, and an image obtained by mapping the depth information at each point of the three-dimensional model two-dimensionally is called a depth image. In addition, the pixel value of a specific pixel in a depth image may be referred to as depth information. In addition, texture information on a specific point of a three-dimensional model is called texture information, and an image obtained by mapping the texture information at each point of the three-dimensional model two-dimensionally is called a texture image. In addition, the pixel value of a specific pixel in a texture image may be called texture information.

[0011] The reproducing device 122 can reproduce a three-dimensional scene in the virtual space by arranging the reconstructed three-dimensional shape data at an appropriate position in the virtual space. Based on the scene reproduced in the virtual space, an image observed from an arbitrary viewpoint (virtual viewpoint) in the space can be generated as a virtual viewpoint image by, for example, tracking an object, analyzing a scene, and drawing a two-dimensional image. The reproducing device 122 of this embodiment can be included in, for example, an information terminal used to observe the virtual viewpoint image. That is, instead of transmitting the virtual viewpoint image after drawing to the information terminal, data encoded by the data generating device 102 can be transmitted, and the information terminal can reproduce a three-dimensional model and generate a virtual viewpoint image based on the data. This is an example, and for example, the entire configuration of FIG. 1 can be included in a device such as a server. In this case, for example, of the captured objects, only the three-dimensional shape data of only the object to be used in generating the virtual viewpoint image is cut out, and the corresponding depth image and texture image are generated and encoded, and stored in, for example, the storage unit 111. Then, in the subsequent reproduction of the three-dimensional scene, the encoded data can be used. In this case, data in the form of a depth image and a texture image before encoding for transmission may be stored in the storage unit 111. Also, for example, the storage unit 111 can store encoded information in order to provide encoded data to an information terminal or the like in a timely manner.

[0012] The data generating device 102 generates a depth image and a texture image for enabling identification of three-dimensional shape data of an object present in the cut-out zone, and encodes the information. At this time, for example, when objects are densely packed, it is not easy to individually cut out the densely packed objects, and it is assumed that the densely packed objects are treated as one huge object. In addition, the size of one object may be large. When the three-dimensional shape of an object is expressed with a certain bit depth, it is assumed that the resolution of the depth information and texture information corresponding to the object decreases when the size of the object increases beyond a certain level. When the resolution of the depth information and texture information decreases, the resolution of the reproduced three-dimensional shape data also decreases accordingly. In this embodiment, in consideration of such circumstances, when a large object exists, the data generating device 102 cuts out a space including the object into multiple zones. Then, the data generating device 102 generates and outputs a depth image and a texture image that enable identification of a part of the three-dimensional shape data of the object for each of the zones. On the other hand, the playback device 122 restores the 3D shape data of a part of the object corresponding to each of the multiple zones based on the generated depth image and texture image. In this way, it is possible to suppress a decrease in resolution due to the large size of the object.

[0013] The data generating device 102 includes a data generating unit 103 and an encoding unit 104 as its functions. The data generating unit 103 acquires data of a three-dimensional model from the three-dimensional model storage unit 101. The three-dimensional model storage unit 101 stores data of a three-dimensional model of a scene generated based on image information and distance information acquired by computer graphics or imaging using a plurality of imaging devices. The format of the stored data is not particularly limited, and may be a point cloud or mesh data. The data generating unit 103 specifies a zone from which an object is cut out from the acquired three-dimensional model data, for example, and sets a virtual viewpoint so as to surround the zone. Then, the data generating unit 103 acquires depth information representing the distance from the virtual viewpoint to a point where an object exists, and texture information representing color information of the surface of the object. The data generating unit 103 generates data including the depth information and texture information, and virtual viewpoint information at the time when the information is acquired. The virtual viewpoint information is information that enables the position, direction, attitude, angle of view, etc. of the set virtual viewpoint to be specified. The encoding unit 104 encodes the data generated by the data generating unit 103. Hereinafter, the encoded data may be referred to as encoded data. Details of the functions and operations of the data generating device 102 will be described later. The generated encoded data is transmitted via a transmitting unit 105. Note that, although an example in which the three-dimensional model saving unit 101 and the transmitting unit 105 exist outside the data generating device 102 is shown in FIG. 1, the data generating device 102 may include at least one of the three-dimensional model saving unit 101 and the transmitting unit 105.

[0014] The encoded data transmitted by the transmitting unit 105 is received by the receiving unit 121. The encoded data transmitted by the transmitting unit 105 may be stored in, for example, the storage unit 111. The encoded data received by the receiving unit 121 is input to the reproduction device 122.

[0015] The reproduction device 122 includes, as its functions, a data reproduction unit 123 and a three-dimensional model reconstruction unit 124. The data reproduction unit 123 decodes the received encoded data to reproduce the data, and acquires virtual viewpoint information, a depth image, and a texture image. The three-dimensional model reconstruction unit 124 then reconstructs three-dimensional shape data of the object from the acquired information. The reconstructed three-dimensional shape data is supplied to the output unit 125, and is used, for example, to generate a virtual viewpoint image in the output unit 125. Details of the function and operation of the reproduction device 122 will be described later. Note that, although FIG. 1 shows an example in which the receiving unit 121 and the output unit 125 exist outside the reproduction device 122, the reproduction device 122 may include at least one of the receiving unit 121 and the output unit 125.

[0016] (Hardware configuration) FIG. 2 shows an example of a hardware configuration of an apparatus (at least one of the data generating apparatus 102 and the reproducing apparatus 122) according to this embodiment. The apparatus is implemented using a general-purpose computer such as a server or an information terminal, and includes a CPU 201, a RAM 202, a ROM 203, an operation unit 204, an output unit 205, an external storage device 206, and an interface (I / F 207). These elements are connected to each other via a bus 208, for example. Note that CPU is an abbreviation for Central Processing Unit, RAM is an abbreviation for Random Access Memory, and ROM is an abbreviation for Read Only Memory. Note that the configuration in FIG. 2 is an example, and the apparatus may have only some of the above-mentioned elements, or may include additional elements in addition to the above-mentioned elements. Also, the apparatus may be configured using, for example, a dedicated processing circuit, without using a general-purpose computer.

[0017] The CPU 201 controls the entire device using computer programs and data stored in the RAM 202 and the ROM 203, and executes each process described below. The CPU 201 is one aspect of a processor, and instead of or in addition to the CPU 201, an MPU (Micro Processing Unit) may be used. A multi-core processor may be used as the CPU 201. That is, in one example, the device includes one or more processors. The RAM 202 has an area for temporarily storing computer programs and data loaded from the ROM 203 or the external storage device 206, data acquired from the outside via the I / F 207, and the like. Furthermore, the RAM 202 provides a work area used when the CPU 201 executes various processes. That is, the RAM 202 is allocated as, for example, a frame memory, and can also be used to store various data such as zone setting information and virtual viewpoint information. The RAM 202 can also manage depth information and texture information (depth images and texture images). The ROM 203 stores setting data and a boot program of the device. It should be noted that the device is not limited to the RAM 202 and the ROM 203, and may be configured to include one or more storage devices such as memories.

[0018] The operation unit 204 includes a keyboard, a mouse, and the like, and accepts operation input by a user. The operation unit 204 may supply various instructions corresponding to the accepted operation input to the CPU 201. The output unit 205 includes, for example, an output mechanism for visual or auditory information, and is configured to provide various information from the device to the user. The output unit 205 may present information visually via a display or a light-emitting diode (LED), and may present information auditorily via a speaker. The output unit 205 may also present information in another form, such as vibration. Note that the operation unit 204 and the output unit 205 may be implemented as an integrated device, for example, using a touch panel display. The operation unit 204 and the output unit 205 may also be an interface for accepting input via an external operation device of the device and presenting information via an external information output device. That is, the device may not include a device for inputting and presenting information.

[0019] The external storage device 206 is, for example, a large-capacity information storage device such as a hard disk drive device. The external storage device 206 can store an OS (operating system) and a computer program for causing the CPU 201 to execute each function described later. The external storage device 206 may also store image data to be processed. The external storage device 206 may also store imaging information. The I / F 207 is an interface for connecting the device to a network such as a LAN (local area network) or the Internet. In one example, the device can be connected to an external device such as a projection device or a display device via the I / F 207. The device provides various information to the external device or obtains various information from the external device via the I / F 207.

[0020] (Configuration of data generating device and processing flow) Next, an example of the functional configuration of the data generating device 102 will be described with reference to Fig. 3. The data generating device 102 is configured to include, for example, a three-dimensional model acquisition unit 301, a zone setting unit 302, a virtual viewpoint setting unit 303, a depth information generating unit 304, and a texture information generating unit 305 as functions of the data generating unit 103. Each of these functions can be implemented, for example, by the CPU 201 executing a program stored in the external storage device 206 or the ROM 203. However, this is just one example, and dedicated hardware corresponding to at least some of these functions may be provided.

[0021] The three-dimensional model acquisition unit 301 is connected to the three-dimensional model storage unit 101. The three-dimensional model acquisition unit 301 acquires desired three-dimensional model data from the three-dimensional model storage unit 101. The zone setting unit 302 sets a zone, which is a unit for encoding the acquired three-dimensional model. Information on the set zone is input to the virtual viewpoint setting unit 303, the depth information generation unit 304, the texture information generation unit 305, and the encoding unit 104. The virtual viewpoint setting unit 303 sets a virtual viewpoint in units of the zone set in the zone setting unit 302. The virtual viewpoint setting unit 303 generates, for each zone, a three-dimensional position, a posture (direction of the virtual camera), an angle of view, a principal point (center on the virtual camera image), and the like of the virtual viewpoint as virtual viewpoint information. Then, the virtual viewpoint setting unit 303 outputs the virtual viewpoint information to the depth information generation unit 304, the texture information generation unit 305, and the encoding unit 104.

[0022] The depth information generating unit 304 acquires data of the three-dimensional model divided for each zone based on the data acquired from the three-dimensional model acquiring unit 301 and the information of the zone set by the zone setting unit 302. Then, the depth information generating unit 304 generates depth information from the data of the three-dimensional model for each zone based on the virtual viewpoint information set by the virtual viewpoint setting unit 303, and outputs the depth information to the encoding unit 104. The texture information generating unit 305 also acquires data of the three-dimensional model divided for each zone based on the data acquired from the three-dimensional model acquiring unit 301 and the information of the zone set by the zone setting unit 302. Then, the texture information generating unit 305 generates texture information from the data of the three-dimensional model for each zone based on the virtual viewpoint information set by the virtual viewpoint setting unit 303, and outputs the texture information to the encoding unit 104.

[0023] The zone setting information, virtual viewpoint information, depth information, and texture information generated by the data generating device 102 are input to the encoding unit 104. The encoding unit 104 encodes this information to generate encoded data. Details of this encoded data will be described later.

[0024] Next, an example of the flow of the data generation process executed by the data generating device 102 will be described with reference to Fig. 4. First, for example, the data generating device 102 accepts a user operation, and the three-dimensional model acquisition unit 301 selects data to be processed from among the three-dimensional models stored in the three-dimensional model storage unit 101 (S401). Then, the three-dimensional model acquisition unit 301 acquires information on the three-dimensional model to be processed from the three-dimensional model storage unit 101 (S402). Here, information on the position and size of the three-dimensional model is acquired. Then, the zone setting unit 302 determines the size of the zone, which is the unit of encoding, based on the size of the three-dimensional model.

[0025] Here, the size of the zone is determined depending on the resolution of the voxel and the bit depth of the depth information. For example, the bit depth is 10 bits and the voxel resolution is 0.5 cm. Here, the resolution of the voxel is defined by the length of the diagonal of the cube that constitutes the voxel. That is, here, the longest distance between two points in the space occupied by one voxel is called the resolution. And, as described above, if the resolution of the voxel is 0.5 cm and the bit depth is 10 bits, a cube with a diagonal length of 512 cm (i.e., a cube of about 295 cm x 295 cm x 295 cm) can be set as a zone. In this embodiment, an expression such as x x y x z is used as information indicating the size of a space in which the lengths of the X-axis and Y-axis, which are the horizontal axes, are x and y, and the length in the vertical direction is z. Assuming a zone of such a size, it is assumed that a three-dimensional model corresponding to one person, for example, can be expressed by one zone. On the other hand, when people are crowded together, for example, in a rugby scrum, it is not necessarily easy to separate the people and generate separate three-dimensional models, and it is assumed that they are treated as one large-sized object. In such a case, it is assumed that the three-dimensional model of the object does not fit into one zone. It is also assumed that the three-dimensional model of an object that is inherently large, such as a vehicle, does not fit into one zone. In this way, when the size of the three-dimensional model exceeds the size of the zone, it is necessary to divide the zone. The zone setting unit 302 specifies how many zones need to be set based on the size of the three-dimensional model to fit the three-dimensional model (S403). For example, when the three-dimensional model of the object is 6m x 4m x 2m, the zone setting unit 302 determines that 2 x 2 x 1 = 4 zones of approximately 295cm x 295cm x 295cm as described above are to be set.

[0026] Here, a case where data related to the three-dimensional model 503 is generated will be described with reference to FIG. 5(A). When the zone setting unit 302 determines that the size of the three-dimensional model 503 is too large to fit into one zone, the zone setting unit 302 sets, for example, two zones (zone 501 and zone 502) into which the three-dimensional model 503 can fit. The three-dimensional model 503 is then divided into the two zones. The division can be performed, for example, with the center of the three-dimensional model as the boundary, or so that the left and right sides are the same size. However, these are only examples, and the division can be performed based on other criteria. For example, the division can be performed so that the cross-sectional area of ​​the object when divided is minimized. Here, the three-dimensional model 503 is divided into two at the position indicated by the dotted line 504. The zone setting unit 302 generates and outputs zone setting information for the set zone. The zone setting information includes, for example, the number of zones, the three-dimensional coordinates indicating the position of each zone, and information indicating (or at least capable of identifying) the size of the zone. The position of a zone may be indicated by the position of a point representing the zone as viewed from the origin of a three-dimensional space in which the zone exists. For example, when an arbitrary point in a three-dimensional space is expressed using coordinates of the X-axis, Y-axis, and Z-axis, a point in the area of ​​the zone that is closest to the origin of the coordinate system (X, Y, Z)=(0, 0, 0) may be determined as the point representing the zone. The center point of the zone may also be determined as the point representing the zone. The size of a zone may be indicated by, for example, the lengths of the X-axis, Y-axis, and Z-axis of the zone. Furthermore, when the shape of the zone is a rectangular parallelepiped and the point representing the zone is the closest point to the origin, the size of the zone may be indicated by the point farthest from the point representing the zone to the origin of the coordinate system. In other words, the position and size of a zone may be expressed in various formats as long as another device such as a playback device can identify the shape of the zone from the information.

[0027] Returning to FIG. 4, in S404, the virtual viewpoint setting unit 303 acquires information on the set zones and division of the three-dimensional model. Thereafter, the virtual viewpoint setting unit 303 executes a process of determining a virtual viewpoint for all zones (S405 to S408). In the example of FIG. 4, a counter for counting the zones for which the process has been completed is initialized to 0 (S405), and the virtual viewpoint setting unit 303 increments the counter (S408) each time the process of determining a virtual viewpoint for one zone is completed (S407). Then, when the incremented counter reaches the number of zones determined in S403 (YES in S406), the virtual viewpoint setting unit 303 ends the loop process of S406 to S408. In the process of S407, the virtual viewpoint setting unit 303 sets virtual viewpoints for virtually photographing the target zone from multiple directions. For example, when a rectangular parallelepiped zone is set, six virtual viewpoints may be set, each located on a normal line passing through the center of each face of the rectangular parallelepiped, with a line of sight from that position toward the center of the zone. At this time, for example, an angle of view and a position of the virtual viewpoint may be set so that the entire zone can be imaged, and the entire three-dimensional model to be processed can be imaged. However, this is merely one example, and the virtual viewpoints may be set in any manner as long as it is possible to simulate a state in which an imaging device (camera) is arranged so that the surface of the three-dimensional model to be processed can be imaged without omission.

[0028] FIG. 5(B) shows an example of setting a virtual viewpoint. FIG. 5(B) shows a part of a three-dimensional model 503 included in the zone 501 shown in FIG. 5(A). The virtual viewpoint setting unit 303 may set virtual viewpoints 511-516 on a normal line passing through the center of each face of the rectangular parallelepiped of the zone 501, for example. At this time, the line of sight of each of the virtual viewpoints 511-516 faces the center of the zone 501. In addition, the angle of view and the distance between the virtual viewpoints 511-516 and the zone 501 may be set so that the virtual cameras arranged at the virtual viewpoints 511-516 capture the entire zone 501 and the entire three-dimensional model to be processed. The virtual viewpoint setting unit 303 sets a virtual viewpoint for the zone 502 in a similar manner. The virtual viewpoint setting unit 303 supplies information on the virtual viewpoints set in this way to the depth information generating unit 304, the texture information generating unit 305, and the encoding unit 104. The position of the virtual viewpoint is expressed, for example, as a relative position from a point that represents the zone in which the virtual viewpoint is set. However, this is just one example, and the position of the virtual viewpoint may be expressed, for example, as a position from the origin of the three-dimensional space in which the zone exists, and the position of the virtual viewpoint may be expressed in any format that allows the position of the virtual viewpoint to be specified in another device such as a playback device.

[0029] 4 again, after the virtual viewpoints have been determined for all zones, the depth information generator 304 generates depth information (S409), and the texture information generator 305 generates texture information (S410).

[0030] Here, the process of the depth information generating unit 304 generating the depth information in S409 of FIG. 4 will be described with reference to FIG. 6. First, the depth information generating unit 304 acquires zone setting information from the zone setting unit 302 (S601). Then, the depth information generating unit 304 executes the processes of S604 to S616 for each of the set zones. In addition, since the following process is repeatedly executed for each of the virtual viewpoints and pixels, the depth information generating unit 304 executes an initialization process for counting them (S602). Then, the depth information generating unit 304 starts the process for the first zone (NO in S603). The depth information generating unit 304 executes the processes of S605 to S614 for each virtual viewpoint set in each zone. In this process, the depth information generating unit 304 acquires virtual viewpoint information for the virtual viewpoint to be processed (S605). The depth information generating unit 304 also obtains data of a three-dimensional model of an object included in the processing target zone from the three-dimensional model obtaining unit 301 (S606). The data of the three-dimensional model obtained here is a three-dimensional model of the entire object to be processed. That is, no cutting by the zone is performed. The three-dimensional model obtained here is a point cloud, but is not limited to this. That is, the three-dimensional model may be expressed in any format as long as the three-dimensional shape of the object can be specified. Then, the depth information generating unit 304 executes the processes of S608 to S612 for all pixel values ​​of the depth image obtained for the processing target zone and virtual viewpoint. The depth information generating unit 304 obtains the position of a target point by taking a point projected onto the processing target pixel position from the point cloud of the data of the three-dimensional model as a target point (S608). Note that the target point here is a point on the surface of the three-dimensional model seen from the processing target virtual viewpoint, regardless of whether it is inside or outside the zone. Then, the depth information generating unit 304 judges whether the position of the target point is included in the zone to be processed (S609). If the depth information generating unit 304 judges that the position of the target point is included in the zone to be processed (YES in S609), the depth information generating unit 304 sets a value corresponding to the distance from the position of the virtual viewpoint to the target point as a pixel value (depth information) of the depth image (S610).On the other hand, if the depth information generating unit 304 determines that the position of the target point is not included in the processing target zone (NO in S609), it complements the depth information by a complementation process described below and sets the complemented value as the pixel value of the depth image (S611). After that, the depth information generating unit 304 increments the counter to move the process to the next pixel value (S612) and returns the process to S607.

[0031] Note that the value corresponding to the distance from the virtual viewpoint position to the target point is, for example, a value obtained by subtracting the shortest distance from the virtual viewpoint position to the zone from the length of the distance from the virtual viewpoint position to the target point. In other words, when the distance from the virtual viewpoint position to a target point in a zone is minimum, a pixel value for a distance of 0 is set for a pixel corresponding to the target point. Hereinafter, unless otherwise specified, the pixel value indicating the distance from the virtual viewpoint position to the target point is assumed to be a pixel value corresponding to a value obtained by subtracting the shortest distance from the virtual viewpoint position to the zone from the length of the distance from the virtual viewpoint position to the target point.

[0032] When the depth information generating unit 304 determines that the processing for all pixel values ​​has been completed based on the pixel value counter (YES in S607), the processing for the virtual viewpoint to be processed has been completed, and therefore the pixel value counter is reset (S613). The depth information generating unit 304 also increments a counter for managing whether the processing for all virtual viewpoints has been completed (S614), and returns the processing to S604. When the depth information generating unit 304 determines that the processing for all virtual viewpoints has been completed based on the virtual viewpoint counter (YES in S604), the processing for the zone to be processed has been completed, and therefore the virtual viewpoint counter is reset (S615). The depth information generating unit 304 also increments a counter for managing whether the processing for all zones has been completed (S616), and returns the processing to S603. Here, when the depth information generating unit 304 determines that the processing for all zones is completed based on the zone counter (YES in S603), the processing for all zones, virtual viewpoints, and pixels is completed. Therefore, the depth information generating unit 304 outputs the depth information generated for multiple virtual viewpoints and each pixel for each zone (S617).

[0033] Next, an example of the flow of the complementation process executed in S611 will be described with reference to Fig. 7. First, the depth information generating unit 304 identifies the minimum distance to a zone for a target point that exists outside the range of the zone (S701). Then, the depth information generating unit 304 sets the identified distance as a pixel value of the depth image (S702).

[0034] This process will be described using the example of FIG. 5B. The depth information generating unit 304 specifies, for example, target points 521 to 524 as target points on the surface of the model when the three-dimensional model 503 is viewed from the virtual viewpoint 513. Of these, the target points 521 and 522 are points outside the zone 501. For this reason, the depth information generating unit 304 complements pixel values ​​of the depth image corresponding to these target points based on the distance from each target point to the boundary surface 541 of the closest zone 501. That is, the distance from the virtual viewpoint 513 to the target points 521 and 522 corresponds to the lengths of the dotted lines 531 and 532, but these target points are projected onto the boundary surface 541, and the pixel values ​​of the depth image are determined based on the points after the projection. In the example of FIG. 5B, the distance (length of solid line 535) between point 525 corresponding to the foot of the perpendicular line from target point 521 to boundary surface 541 and virtual viewpoint 513 is set as the pixel value of the depth image related to target point 521. Similarly, the distance (length of solid line 536) between point 526 corresponding to the foot of the perpendicular line from target point 522 to boundary surface 541 and virtual viewpoint 513 is set as the pixel value of the depth image related to target point 522. On the other hand, target point 523 is a point inside zone 501. Therefore, the distance (length of solid line 533) between virtual viewpoint 513 and target point 523 is set as the pixel value of the depth image related to target point 523. Also, target point 524 is a point on boundary surface 541. In this case, target point 524 is treated as a point inside zone 501, and the distance (length of solid line 534) between virtual viewpoint 513 and target point 524 is set as the pixel value of the depth image related to target point 524. This process can be said to be a process in which a point on boundary surface 541 that corresponds to a cross section of the three-dimensional model is treated as if the point of the three-dimensional model were on boundary surface 541 .

[0035] The determination in S609 may be a determination as to whether or not the shape of the three-dimensional model is cut at the boundary surface of the zone when viewed from the virtual viewpoint by the division using the zone when the three-dimensional model is viewed from the position of the virtual viewpoint. That is, even if a point on the surface of the three-dimensional model exists on a line connecting a point on each boundary surface of the zone when viewed from the position of the virtual viewpoint and the virtual viewpoint and on a line within the zone, if the point on the boundary surface of the zone is included inside the three-dimensional model, the point is determined to be outside the zone. Note that the boundary surface of the zone when viewed from the position of the virtual viewpoint here is the surface of the zone that can be observed from the virtual viewpoint when an opaque object of the shape of the zone exists. For example, in FIG. 5(B), the boundary surface of the zone 501 when viewed from the position of the virtual viewpoint 513 is only the boundary surface 541. For example, it may be determined that the point on the boundary surface exists between the 2n-1th intersection point and the 2nth intersection point by counting the intersection points between the line connecting the virtual viewpoint and the point on the boundary surface of the zone and the entire surface of the three-dimensional model, and that the point is inside the shape of the three-dimensional model when the point exists between the 2n-1th intersection point and the 2nth intersection point. Here, n is an integer equal to or greater than 1, and the first intersection point on the line connecting the virtual viewpoint and a point on the boundary surface of the zone is the first intersection point that reaches the surface of the three-dimensional model when viewed from a predetermined point away from the three-dimensional model. The second intersection point is the intersection point when passing through the inside of the shape of the three-dimensional model from the first intersection point and exiting the outside of the shape of the three-dimensional model. That is, the inside of the shape of the three-dimensional model is entered at an odd-numbered intersection point, and the outside of the shape of the three-dimensional model is exited at an even-numbered intersection point. Therefore, if a point on the boundary surface of the zone exists between the 2n-1th intersection point and the 2nth intersection point, the point is determined to exist inside the three-dimensional model. Also, if a point on the boundary surface of the zone exists between the 2nth intersection point and the 2n+1th intersection point, the point is determined to exist outside the three-dimensional model. In this case, even if a point of the three-dimensional model as the whole object exists when viewed from the virtual viewpoint, it is not necessary to perform interpolation. Throughout this specification, the process of determining whether the point being processed is within a zone can be read as a determination of whether a point on the boundary surface of the zone when viewed from a virtual viewpoint is a point inside the three-dimensional model.

[0036] FIG. 8 shows an example of a depth image generated by the process shown in FIG. 6 and FIG. 7, focusing on the virtual viewpoint 513. In the depth image, an area 801 is an area in which a complemented pixel value is specified because the position of the surface of the three-dimensional model seen from the virtual viewpoint 513 is outside the zone. In this area, the distance from the virtual viewpoint 513 to each point on the boundary surface 541 is indicated by the pixel value. In areas 802 and 803, the position of the surface of the three-dimensional model seen from the virtual viewpoint 513 is inside the zone, so the distance from the virtual viewpoint 513 to the point on the surface of the three-dimensional model is not complemented and is indicated by the pixel value. Note that, in this example, the area 802 includes a part of the three-dimensional model in which a point group exists on the boundary surface 541. In this way, a depth image from one virtual viewpoint is generated for a part of the three-dimensional model cut out by a zone of a certain size. Note that the depth information generating unit 304 can obtain multiple depth images corresponding to multiple viewpoints by generating depth images seen from other virtual viewpoints in the same manner. Such multiple depth images allow the three-dimensional shape of the object to be reconstructed, segmented by zones.

[0037] Next, the process performed by the texture information generating unit 305 to generate texture information in S410 of FIG. 4 will be described with reference to FIG.

[0038] First, the texture information generating unit 305 obtains zone setting information from the zone setting unit 302 (S901). The texture information generating unit 305 also sets a texture complement value for complementing points outside the zone in the three-dimensional model (S902). The value is not particularly limited, but may be set to a value such as 0 that simplifies encoding. Then, the texture information generating unit 305 executes the processes of S905 to S918 for each of the set zones. Since the following processes are repeatedly executed for each of the virtual viewpoints and pixels, the texture information generating unit 305 executes an initialization process for counting them (S903). Then, the texture information generating unit 305 starts the process for the first zone (NO in S904). The texture information generating unit 305 executes the processes of S906 to S916 for each virtual viewpoint set in each zone. In this process, the texture information generating unit 305 acquires virtual viewpoint information for the virtual viewpoint to be processed (S906). Furthermore, the texture information generating unit 305 acquires data of a three-dimensional model of an object included in the zone to be processed from the three-dimensional model acquiring unit 301 (S907), and further acquires texture information of the surface of the three-dimensional model (S908). The texture information can be, for example, color information of each point included in the point cloud. However, this is not limited to the above, and when the color of the surface of a three-dimensional model is formed using patches or the like, color information at each position on the surface of the three-dimensional shape (for example, positions corresponding to each point included in the point cloud) may be used as texture information.

[0039] Then, the texture information generating unit 305 executes the processes of S910 to S914 for all pixels of the texture images obtained for the processing target zone and virtual viewpoint. The texture information generating unit 305 acquires the position of a target point, which is a point projected onto the processing target pixel position among the point group of the data of the three-dimensional model, as a target point (S910). Then, the texture information generating unit 305 judges whether or not the position of the target point is included in the zone to be processed (S911). If the texture information generating unit 305 judges that the position of the target point is included in the zone to be processed (YES in S911), it sets the texture value of the target point as the pixel value of the texture image (S912). On the other hand, if the texture information generating unit 305 judges that the position of the target point is not included in the zone to be processed (NO in S911), it complements the texture information by a complementation process to be described later, and sets the complemented value as the pixel value of the texture image (S913). Thereafter, the texture information generating unit 305 increments the counter to move the process to the next pixel value (S914), and returns the process to S909.

[0040] When the texture information generating unit 305 determines that the processing for all pixel values ​​has been completed based on the pixel value counter (YES in S909), the processing for the virtual viewpoint to be processed has been completed, and therefore the counter for the pixel values ​​is reset (S915). Furthermore, the texture information generating unit 305 increments a counter for managing whether the processing for all virtual viewpoints has been completed (S916), and returns the processing to S905. When the texture information generating unit 305 determines that the processing for all virtual viewpoints has been completed based on the virtual viewpoint counter (YES in S905), the processing for the zone to be processed has been completed, and therefore the counter for the virtual viewpoint is reset (S917). Furthermore, the texture information generating unit 305 increments a counter for managing whether the processing for all zones has been completed (S918), and returns the processing to S904. Here, when the texture information generating unit 305 determines that the processing for all zones has been completed based on the zone counter (YES in S904), the processing for all zones, virtual viewpoints, and pixels is completed. For this reason, the texture information generating unit 305 outputs the texture information generated for a plurality of virtual viewpoints for each zone (S919).

[0041] Fig. 10 shows the flow of the texture information complementation process for target points, which is executed in S913. The texture information generation unit 305 replaces the pixel values ​​of the texture image corresponding to target points outside the zone with the texture complementation value set in S902 (S1001). For example, in the example of Fig. 5(B), since target points 521 and 522 exist outside zone 501, the pixel values ​​of the texture image corresponding to the target points are complemented by the texture complementation value. On the other hand, since target points 523 and 524 exist inside zone 501, the texture values ​​of the points in the three-dimensional model are used as the pixel values ​​of the texture image corresponding to the points.

[0042] Next, the coded data generated by the coding unit 104 will be described with reference to Fig. 11. The coded data "Volumetric_video_data" includes header information indicating that the data is coded data at a predetermined position, such as at the beginning of the data.

[0043] The encoded data includes zone setting information "Zone_data". Zone_data includes the number of zones to be encoded "number_of_zone" and information on each zone "Zone_information_data". Here, FIG. 11 shows an example in which the number of zones is N. Zone_information_data is prepared for each zone. Zone_information_data includes, for example, zone identification information "zone_ID". The zone_ID is information that can individually identify multiple zones. For example, as the zone_ID, consecutive numbers may be assigned to multiple zones, or individual names may be given. Zone_information_data further includes the position of the origin of the target zone "zone_location[3]". In this embodiment, it is assumed that a three-dimensional space including a three-dimensional model is represented by a coordinate system consisting of orthogonal coordinate axes (for example, an X axis, a Y axis, and a Z axis). The zone_location[3] may be a coordinate indicating where the zone is located in this three-dimensional space. For example, the coordinate value of the representative point of the zone (for example, the point closest to the origin in the three-dimensional space) is represented as zone_location[3]. Here, [3] indicates that, for example, three elements (coordinate values) for three axes, the X axis, the Y axis, and the Z axis, are included. Zone_information_data further includes the size of the zone, "zone_size[3]". This zone_size[3] also includes, for example, values ​​for the three axes, the X axis, the Y axis, and the Z axis. For example, a value indicating each length may be stored in zone_size[3]. The length here may be expressed in units such as millimeters (mm), or may be expressed in other units such as the number of voxels. In addition, information that can specify the size of the zone by combination with the representative point of the zone, such as the coordinate value of the point farthest from the origin in the three-dimensional space, may be indicated in zone_size[3]. Zone_information_data may further include the bit depth of the depth information, "depth_bit_length".

[0044] The encoded data further includes setting information "Virtual_view_point_data" about the virtual viewpoint. One Virtual_view_point_data is also prepared for each zone. The Virtual_view_point_data includes the number of virtual viewpoints set in the target zone, "number_of_virtual_view_point". FIG. 11 shows an example in which the number of virtual viewpoints is M. The Virtual_view_point_data includes information about each of the M virtual viewpoints. This information includes information about the identifier, position, attitude, angle of view, principal point, and size of the depth image and texture image to be acquired for each virtual viewpoint. The information indicating the identifier is "virtual_view_point_ID", and is expressed by, for example, a number or a character string that identifies the virtual viewpoint. The information indicating the position of the virtual viewpoint is "virtual_view_point_location[3]", and is indicated by the coordinate value of a three-dimensional space including a three-dimensional model. Here, similarly to the zone information, a three-dimensional space including a three-dimensional model is represented by a coordinate system consisting of orthogonal coordinate axes (for example, X-axis, Y-axis, and Z-axis), and the position of a virtual viewpoint (virtual camera) is specified by coordinate values. Note that, here too, [3] indicates that, for example, three elements (coordinate values) for three axes, X-axis, Y-axis, and Z-axis, are included. Information indicating the attitude of the virtual viewpoint is "virtual_view_point_attitude[3]", which can be expressed, for example, by using a quaternion as Q=(0;x,y,z). Note that in this formula, the left of the semicolon is the real part, and x, y, and z represent the imaginary part. As an example, these x, y, and z are used as elements of virtual_view_point_attitude[3] representing the attitude. Information indicating the angle of view is "angle_of_view", and information indicating the position of the principal point is "center_point[3]". The position of the principal point can be indicated by coordinate values ​​in a three-dimensional space including the three-dimensional model. Information on the size of the acquired depth image and texture image is "virtual_view_point_image_size[2]".This information includes two elements that indicate the vertical and horizontal sizes of the image. Note that the order of the information shown in Fig. 11 is an example, and the Virtual_view_point_data may be configured in an order other than this.

[0045] The encoded data includes depth image information "Depth_data" including information on depth images from each virtual viewpoint, and texture image information "Texture_data" including information on texture images from each virtual viewpoint. Depth_data stores information on depth images obtained in each zone and each virtual viewpoint. In FIG. 11, the depth image obtained for the mth virtual viewpoint in the nth zone is represented by "Depth_image_data[n;m]". Texture_data stores information on texture images obtained in each zone and each virtual viewpoint. In FIG. 11, the texture image obtained for the mth virtual viewpoint in the nth zone is represented by "Texture_image_data[n;m]". The depth image and the texture image are stored as two-dimensional image data, for example, in an array format of pixel values ​​for the image size. The encoding of these image data can be performed using a general still image encoding method such as JPEG, but is not limited thereto, and any encoding method may be used. Also, for example, image data may be stored in a server on a network, and a pointer such as a URL (Uniform Resource Locator) for accessing the server may be included in the encoded data as information related to the image data. In this case, the depth information and texture information generated by the data generating unit 103 may be encoded in the encoding unit 104 in the form of a depth image and a texture image, for example, and stored in the server. Then, the server notifies the encoding unit 104 of pointer information such as a URL for accessing the depth image and the texture image, and the encoding unit 104 generates encoded data using the notified pointer information. Note that, in this embodiment, the image sizes of the depth image and the texture image are the same, but the sizes of these images may be different from each other. In this case, information indicating the respective image sizes separately is included in the encoded data. Also, "virtual_view_point_image_size[2]" indicating the image size may be included as an element of "Depth_data" or "Texture_data".

[0046] The generated coded data Volumetric_video_data is supplied to the transmission unit 105. The transmission unit 105 transmits the coded data to another device such as the playback device 122. Note that, in the present embodiment, an example of generating transmission data to be transmitted to the playback device 122 has been described, but the data does not necessarily need to be transmitted. For example, in order to simplify management of the three-dimensional model, division by zone as described above may be performed. In that case, the coded data obtained as described above may be supplied to the storage unit 111, and the data may be read from the storage unit 111 when it is used thereafter. This allows data to be managed for each zone, and when data related to only a portion of the zones is used, it is not necessary to read data related to other zones, making it possible to operate the system efficiently.

[0047] In this manner, the pixel values ​​of the depth image and texture image are complemented for the portion where the surface of the object does not exist as a result of the division by zones, and these images can be generated and encoded. In addition, in the depth image formed in this embodiment, the distance value between the camera and the point contacting the boundary surface 541 and the point adjacent to that point and whose pixel value is complemented is very close. As a result, a correlation is generated between the non-complemented value and the complemented value adjacent to each other, so that high encoding efficiency can be obtained in encoding by prediction from surrounding pixel values. In addition, by performing complementation using a predetermined value in the texture image, the encoding efficiency can be improved by improving the prediction of pixels and the occurrence probability of codes.

[0048] In the present embodiment, an example in which data is generated using a three-dimensional model once stored in the three-dimensional model storage unit 101 has been described, but the present invention is not limited to this. For example, data generation as described above may be performed in parallel during the process of generating a three-dimensional model. In addition, when a voxel at the boundary of a zone is not visible from the outside in the voxels of the generated three-dimensional model, the amount of data of the three-dimensional model may be reduced by deleting the voxel. In addition, in the above-mentioned embodiment, an example in which a virtual viewpoint (virtual camera) is set on a normal line passing through the center of the boundary surface of a zone, and the line of sight is set to the center direction of the boundary surface of the zone, but the present invention is not limited to this. That is, the virtual viewpoint may be placed at an arbitrary position in three-dimensional space, and the line of sight may be set arbitrarily under the condition that the entire zone (or a part of the zone in some cases) is included within the angle of view. In this case, among the surfaces that intersect perpendicularly with the line of sight direction, the surface that contacts the zone at the position closest to the virtual viewpoint is set as the virtual boundary surface, and a depth image may be generated. In addition, this is just one example, and other methods may be used. For points on the surface of a three-dimensional model viewed from a virtual viewpoint that are outside a zone, the point may be moved parallel to the line of sight to identify the point where it intersects with one of the boundary faces of the zone, and a depth image may be generated based on the distance between the point and the virtual viewpoint.

[0049] (Playback device configuration and processing flow) Next, an example of the functional configuration of the playback device 122 will be described with reference to FIG. 12. The playback device 122 has a data playback unit 123 and a three-dimensional model reconstruction unit 124 as described above in order to reconstruct a three-dimensional model based on the encoded data obtained from the data generation device 102 as described above. The playback device 122 has, for example, a decoding unit 1201, a zone information acquisition unit 1202, a virtual viewpoint information acquisition unit 1203, a depth information acquisition unit 1204, and a texture information acquisition unit 1205 as functions of the data playback unit 123. The playback device 122 also has, for example, a three-dimensional model generation unit 1211 and a zone integration unit 1212 as functions of the three-dimensional model reconstruction unit 124. Each of these functions can be implemented by, for example, the CPU 201 executing a program stored in the external storage device 206 or the ROM 203. However, this is just an example, and dedicated hardware corresponding to at least a part of these functions may be provided.

[0050] The decoding unit 1201 receives the coded data (Volumetric_video_data in FIG. 11) via the receiving unit 121, decodes the received coded data, and reproduces each piece of information included in the data. The decoding unit 1201 extracts and decodes zone setting information (Zone_data) from the coded data to obtain information on the number of zones, IDs for each zone, positions of the zones, size of the zones, and bit depth. This information is input to the zone information acquisition unit 1202. The decoding unit 1201 also extracts and decodes virtual viewpoint information (Virtual_view_point_data). As a result, the decoding unit 1201 obtains information on the number of virtual viewpoints for each zone, IDs for each virtual viewpoint, positions of the virtual viewpoints, attitude of the virtual viewpoints, angle of view, position of the principal point, and size of the images. This information is input to the virtual viewpoint information acquisition unit 1203. The decoding unit 1201 also extracts Depth_data and Texture_data to obtain depth images and texture images for each virtual viewpoint of each zone. The depth image data is input to a depth information acquisition unit 1204 , and the texture image data is input to a texture information acquisition unit 1205 .

[0051] The zone information acquisition unit 1202 outputs the input zone setting information to the virtual viewpoint information acquisition unit 1203, the depth information acquisition unit 1204, and the texture information acquisition unit 1205. The virtual viewpoint information acquisition unit 1203 acquires information on the number of zones, the position of the zone, and the size of the zone from the zone information acquisition unit 1202. The virtual viewpoint information acquisition unit 1203 also acquires and holds the ID, position, attitude, angle of view, position of the principal point, and image size for each virtual viewpoint related to each zone from the decoding unit 1201. The depth information acquisition unit 1204 acquires the number of zones from the zone information acquisition unit 1202 and information such as the number of virtual viewpoints and the size of the image from each virtual viewpoint from the virtual viewpoint information acquisition unit 1203. The depth information acquisition unit 1204 acquires and holds the depth image for each virtual viewpoint of each zone from the decoding unit 1201 based on the number of zones, the number of virtual viewpoints, and the size of the image. The depth information acquisition unit 1204 outputs data (depth information) in a depth image of a corresponding virtual viewpoint of a corresponding zone to the three-dimensional model generation unit 1211 in response to a request from the three-dimensional model generation unit 1211. The texture information acquisition unit 1205 acquires information such as the number of zones from the zone information acquisition unit 1202 and the number of virtual viewpoints and the size of an image from each virtual viewpoint from the virtual viewpoint information acquisition unit 1203. The texture information acquisition unit 1205 acquires texture image data for each virtual viewpoint of each zone from the decoding unit 1201 based on the number of zones, the number of virtual viewpoints, and the size of the image, and holds the data. The texture information acquisition unit 1205 outputs data (texture information) in a texture image of a corresponding virtual viewpoint of a corresponding zone to the three-dimensional model generation unit 1211 in response to a request from the three-dimensional model generation unit 1211.

[0052] The three-dimensional model generating unit 1211 generates a three-dimensional model for each zone. For example, when the three-dimensional model generating unit 1211 acquires information on the zone from the zone information acquiring unit 1202, the three-dimensional model generating unit 1211 specifies the number of zones to be considered when generating a three-dimensional model, and specifies the position and shape of the zone. In addition, the three-dimensional model generating unit 1211, for example, specifies a specific zone, acquires information on the virtual viewpoint for the zone from the virtual viewpoint information acquiring unit 1203, and acquires the position and attitude of the virtual viewpoint surrounding the zone, the angle of view, the position of the principal point, and the size of the image. Then, the three-dimensional model generating unit 1211 specifies the characteristics of the virtual camera used when the depth image and texture image for the processing target zone were acquired. In addition, the three-dimensional model generating unit 1211 acquires depth information and texture information for the processing target zone from the depth information acquiring unit 1204 and the texture information acquiring unit 1205. Then, the three-dimensional model generating unit 1211 restores a three-dimensional model of an object in the zone based on the specified zone and virtual viewpoint information and the depth information and texture information corresponding to each virtual viewpoint. This 3D model is generated using interpolated values ​​of depth and texture. In this manner, the 3D model generation unit 1211 generates a 3D model for each zone. The zone integration unit 1212 integrates the 3D models for each of the multiple zones generated by the 3D model generation unit 1211 to generate and output a 3D model of one object.

[0053] Here, with reference to FIG. 13, the flow of the generation process of the three-dimensional model in each zone executed by the three-dimensional model generation unit 1211 will be described. First, the three-dimensional model generation unit 1211 acquires information on the number of zones set for the three-dimensional model to be generated from the zone information acquisition unit 1202 (S1301). Then, the three-dimensional model generation unit 1211 repeatedly executes the processes of S1304 to S1317 for the number of zones including the three-dimensional model. In addition, since the following process is repeatedly executed for each of the virtual viewpoint and the pixels of the depth image and the texture image, the three-dimensional model generation unit 1211 executes an initialization process for counting them (S1302). Then, the three-dimensional model generation unit 1211 starts the process for the first zone (NO in S1303). The three-dimensional model generation unit 1211 acquires information on the position, size, and bit depth of the zone to be processed from the zone information acquisition unit 1202 (S1304). Then, the three-dimensional model generating unit 1211 executes the process of S1306 to S1315 for each virtual viewpoint set in each zone. In this process, the three-dimensional model generating unit 1211 acquires virtual viewpoint information related to the zone from the virtual viewpoint information acquiring unit 1203 (S1306). In addition, the three-dimensional model generating unit 1211 acquires a depth image corresponding to the virtual viewpoint to be processed from the depth information acquiring unit 1204 (S1307). For example, the three-dimensional model generating unit 1211 acquires an image as shown in FIG. 8 as a depth image related to the virtual viewpoint 513, and acquires depths for the other virtual viewpoints that have been generated based on the virtual viewpoints in the same manner. In addition, the three-dimensional model generating unit 1211 acquires a texture image corresponding to the virtual viewpoint to be processed from the texture information acquiring unit 1205 (S1308).

[0054] Then, the three-dimensional model generating unit 1211 executes the processes of S1310 to S1313 for all pixels of the depth image (and texture image) obtained for the zone to be processed and the virtual viewpoint based on the image size acquired from the virtual viewpoint information acquiring unit 1203. The three-dimensional model generating unit 1211 acquires the pixel value (depth information) of the processing target from the depth image, and acquires distance information from the virtual viewpoint (S1310). Then, the three-dimensional model generating unit 1211 sets a voxel for the target point of the three-dimensional shape corresponding to the pixel value of the processing target based on the distance information from the virtual viewpoint (S1311). For example, voxels are set at the corresponding positions of the boundary surface for the target points 521 and 522 in FIG. 5(B) based on the depth image shown in FIG. 8. Also, voxels are set at the positions corresponding to the surface of the three-dimensional model to be reproduced (positions behind the boundary surface) for the target points 523 and 524. Then, the three-dimensional model generating unit 1211 colors the voxels of the target points based on the corresponding pixel values ​​(texture information) of the texture image (S1312). For example, texture information of a predetermined value is added to the target points 521 and 522 in Fig. 5(B), and the texture information added to the three-dimensional model is added to the target points 523 and 524. By repeatedly executing the processes of S1310 to S1313, colored voxels are set at the points of the three-dimensional model visible from the virtual viewpoint being processed.

[0055] When the three-dimensional model generation unit 1211 determines that the processing for all pixel values ​​has been completed based on the pixel value counter (YES in S1309), the processing for the virtual viewpoint to be processed has been completed, and therefore the three-dimensional model generation unit 1211 resets the pixel value counter (S1314). Furthermore, the three-dimensional model generation unit 1211 increments a counter for managing whether the processing for all virtual viewpoints has been completed (S1315), and returns the processing to S1305. When the three-dimensional model generation unit 1211 determines that the processing for all virtual viewpoints has been completed based on the virtual viewpoint counter (YES in S1305), the processing for the zone to be processed has been completed, and therefore the three-dimensional model generation unit 1211 resets the virtual viewpoint counter (S1316). Furthermore, the three-dimensional model generation unit 1211 increments a counter for managing whether the processing for all zones has been completed (S1317), and returns the processing to S1303. Through the processing up to this point, a three-dimensional model included in the zone to be processed is reconstructed based on information from all virtual viewpoints corresponding to the zone. In the three-dimensional model corresponding to one zone, voxels not included in the original three-dimensional model are set at the boundary surface of the zone. After that, when the three-dimensional model generating unit 1211 determines that the processing for all zones is completed based on the zone counter (YES in S1303), the processing for all zones, virtual viewpoints, and pixels is completed. Therefore, the three-dimensional model generating unit 1211 outputs the three-dimensional model reconstructed for each zone to the zone integration unit 1212 (S1318).

[0056] Next, an example of the flow of the zone integration process executed by the zone integration unit 1212 will be described with reference to FIG. 14. The zone integration unit 1212 first acquires the number of zones from the zone information acquisition unit 1202 (S1401). When the following processes are executed in the order of the zone IDs, the zone ID information may be acquired. The zone integration unit 1212 initializes a counter for counting the number of processed zones in order to repeatedly execute the processes of S1404 to S1407 for the number of zones (S1402). When the processes are executed in the order of the zone IDs, the zone IDs may be expressed by numbers such as 0 to N-1. When the zone integration unit 1212 starts processing the first zone to be processed (NO in S1403), it acquires zone setting information related to the zone from the zone information acquisition unit 1202 (S1404). It should be noted that, in this embodiment, the zone information acquisition unit 1202 notifies the zone integrating unit 1212 of the zone setting information separately from the three-dimensional model reproduced by the three-dimensional model generation unit 1211, but is not limited to this. For example, the three-dimensional model generation unit 1211 may add the zone setting information to the three-dimensional model of each zone and transmit it to the zone integrating unit 1212.

[0057] Then, the zone integration unit 1212 acquires the three-dimensional model reproduced for the zone to be processed from the three-dimensional model generation unit 1211 (S1405). Based on the position of the zone in the zone setting information acquired in S1404, the zone integration unit 1212 integrates the three-dimensional model of the zone to be processed with the three-dimensional model that has already been integrated and stored (S1406). The zone integration unit 1212 may perform initialization processing so that nothing initially exists in the space in which the three-dimensional model is reproduced. In the processing for the first zone to be processed, the zone integration unit 1212 executes only processing for arranging the three-dimensional model corresponding to that zone in accordance with the position of that zone. After that, the three-dimensional model for the currently processed zone is additionally arranged in the space in which the three-dimensional model was arranged in the processing for the previous zone to be processed. In this way, the three-dimensional model is updated the number of times equal to the number of zones, and the integration processing of the three-dimensional model of the object is completed. That is, the integration process of the three-dimensional model is performed by arranging the three-dimensional models reproduced for each of the multiple zones in the same space. When the process for all the zones is completed (YES in S1403), the zone integration unit 1212 completes the integration process.

[0058] Note that the voxels of the 3D model based on the pixel values ​​of the interpolated depth image are voxels on the borders of adjacent zones that cannot be seen from the surface, i.e., they are internal voxels of the final reconstructed 3D model, and therefore will not be reflected in the image when the 3D model is used to generate a virtual viewpoint image.

[0059] In this way, the 3D model is divided into zones and encoded with high quality and high efficiency, making it possible for the playback device 122 that receives the encoded information to reconstruct the original large-sized 3D model with high accuracy.

[0060] (Variation 1) In the above embodiment, an example has been described in which the depth information generating unit 304 of the data generating device 102 performs complementation such that, when a point on the surface of a three-dimensional model seen from a virtual viewpoint is outside a zone, the point is treated as a point on a zone boundary surface. In this modified example, a method of performing this complementation using a different method will be described.

[0061] FIG. 15 shows the flow of the depth information generation process executed in S409 of FIG. 4 in this modification. In FIG. 15, the same processing steps as those described in relation to FIG. 6 are denoted by the same reference numerals, and the description thereof will be omitted. In this process, the zone setting unit 302 sets a depth information complement value for complementing the depth information (S1501). Here, a value range indicating the distance is set, and a value outside the value range is set as the depth information complement value. That is, the depth information complement value is set to a value that is not used as a value indicating the distance between a point of the three-dimensional model in the zone and the virtual viewpoint. Then, the depth information generating unit 304 complements the depth information using the depth information complement value set in S1501 instead of complementing the depth information in the process of S611 of FIG. 6 (S1502). That is, in this modification, for a point outside the zone, the point is not mapped to the boundary surface of the zone, and a predetermined value that is not used as a value indicating the distance is set as the pixel value of the depth image corresponding to the point.

[0062] Here, FIG. 16(A) and FIG. 16(B) show an example of setting the depth information complement value in this modification. In FIG. 16(A), in an example where the bit depth is 10 bits, the distance between the tent of the three-dimensional model in the zone and the virtual viewpoint is expressed by a value between "1" and "1023". For example, the distance from the virtual viewpoint 513 to the closest point in the zone is expressed by "1", and the distance to the farthest point in the zone is expressed by "1023". Then, "0", which is a value not used as a distance value, is set as the depth information complement value. The zone setting unit 302 outputs the set depth information complement value to the encoding unit 104 together with the zone setting information. Then, the encoding unit 104 generates encoded data including, for example, "default_out_of_zone" representing the depth information complement value as an element of Zone_data, as shown in FIG. 17. In Fig. 17, unlike the symbols in Fig. 11, Depth_data and Texture_data are not provided separately, but one element called Depth_Texture_data is provided. However, the contents are the same as those in Fig. 11 in that information on depth images and texture images for each zone and each virtual viewpoint set in the zone is included.

[0063] For example, when the zone 501 and the virtual viewpoint 531 in FIG. 5B are to be processed, the pixel values ​​corresponding to the target points 521 and 522 are complemented with a depth information complement value of "0" in the generated depth image because the target points 521 and 522 are not included in the zone 501. On the other hand, the pixel values ​​corresponding to the target points 523 and 524 are values ​​between 1 and 1023 because the target points are included in the zone 501. For example, the pixel value of the corresponding pixel in the depth image of the target point 524 on the boundary surface 541 is set to "1". The pixel value of the corresponding pixel in the depth image of the target point 523 is a point inside the zone away from the boundary surface 541, so that the pixel value of the corresponding pixel in the depth image is a value greater than 1. An example of a depth image generated in this manner is shown in FIG. 18. For comparison with FIG. 8, the same reference numbers are used for the corresponding regions in the depth image. Region 801 is a pixel region corresponding to a region where a point of the three-dimensional model is outside the zone, and the pixel value of this region is set to "0," which is a depth information complement value. Regions 802 and 803 are pixel regions corresponding to regions where a point of the three-dimensional model is inside the zone, and a pixel value corresponding to the distance from the virtual viewpoint to that point is set.

[0064] In the above example, the depth information complement value is set to "0", but the present invention is not limited to this. For example, a predetermined value such as a maximum or minimum value that can be expressed as the distance within a zone, or other predetermined value, may be set as the depth information complement value. For example, as shown in FIG. 16(B), when the bit depth is 10, "1023" may be set to be used as the depth information complement value, and the distance within the zone to each point of the three-dimensional model may be expressed by a value within the range of "0" to "1022". The texture information is generated by the method of FIG. 9.

[0065] In the above example, the zone setting unit 302 sets the depth information complement value, but the present invention is not limited to this. For example, the depth information generating unit 304 may set the depth information complement value for each zone. In this case, as shown in FIG. 19(A), default_out_of_zone representing the depth information complement value may be included in Zone_information_data indicating information about each zone. In addition, the depth information generating unit 304 may set the depth information complement value for each virtual viewpoint. In this case, as shown in FIG. 19(B), default_out_of_zone representing the depth information complement value may be included in Virtual_view_point_information_data indicating information about each virtual viewpoint.

[0066] In this way, by dividing the image into zones, depth information and texture information are complemented for the portions where the surface of the three-dimensional model of the object no longer exists, making it possible to encode the depth image and the texture image. Furthermore, for the depth image, the complemented portion is set to a fixed value, so that a high encoding efficiency can be obtained in an encoding method that utilizes prediction from surrounding pixel values, code occurrence probability, and the like.

[0067] Next, a process in which the playback device 122 plays back a three-dimensional model from the coded data generated as described above will be described with reference to FIG. 20. In FIG. 20, the same process steps as those in FIG. 13 are denoted by the same reference numerals, and detailed description thereof will be omitted. Before this process is performed, the decoding unit 1201 acquires the coded data shown in FIG. 17, decodes the zone setting information (Zone_data), and reproduces the number of zones, the ID for each zone, the position of the zone, the size of the zone, the bit depth, and the depth information complement value. It is assumed that these pieces of information have been input to the zone information acquisition unit 1202. The decoding unit 1201 acquires a depth image and a texture image from Depth_Texture_data in the coded data in FIG. 17. The decoding unit 1201 outputs the depth image to the depth information acquisition unit 1204, and outputs the texture image to the texture information acquisition unit 1205.

[0068] In the process of FIG. 20, the three-dimensional model generating unit 1211 obtains a depth information complement value from the zone information obtaining unit 1202 in addition to information on the number of zones set for the three-dimensional model to be generated (S2001). After that, the three-dimensional model generating unit 1211 determines whether or not the pixel value (depth information) of each pixel of the depth image is the depth information complement value obtained in S2001 (S2002). Then, if the pixel value of the depth image corresponding to the point to be processed is not the depth information complement value (NO in S2002), the three-dimensional model generating unit 1211 sets a voxel and texture value for that point (S1311, S1312). On the other hand, if the pixel value of the depth image corresponding to the point to be processed is the depth information complement value (YES in S2002), the three-dimensional model generating unit 1211 does not set a voxel and texture value, and proceeds to processing for the next target point (S1313).

[0069] In this manner, in this processing example, voxels are not set in the portion corresponding to the cross section of the three-dimensional model when it is divided using the zone. Then, the multiple three-dimensional models reproduced for the multiple zones are integrated by the zone integration unit 1212, so that the region where no voxels are set becomes the region inside the integrated three-dimensional model. That is, in the above-mentioned embodiment, the surface divided by the zone is set inside the integrated three-dimensional model, but in this processing example, such a surface is not set. On the other hand, such a surface is a surface that is not observed from the outside in the reproduced three-dimensional model, so that the above-mentioned embodiment and this processing example can reproduce a three-dimensional model that is externally the same. In addition, in this processing example, unnecessary voxels are not generated on the boundary surface of the zone, so that the amount of data of the reconstructed three-dimensional model can be reduced.

[0070] (Variation 2) In the first modification, for example, an example was shown in which a value that is not used as a pixel value for indicating the distance between the virtual viewpoint and a point in a zone is used as the depth information complement value. In the procedure of the first modification, for example, in the example of FIG. 16(A), values ​​of 1 to 1023 are used as pixel values ​​of the depth image indicating the position of the point group inside the zone, and 0 cannot be used. Also, in the example of FIG. 16(B), values ​​of 0 to 1022 are used as pixel values ​​of the depth image indicating the position of the point group inside the zone, and 1023 cannot be used. That is, the depth image is expressed in 1023 stages. In contrast, for example, even if the area is divided into zones, if there is no point of the three-dimensional model in front of the zone (on the virtual viewpoint side), there is no need to use the complement value, so the granularity of the quantization can be made finer by not using the depth information complement value. For this reason, in this modification, when the depth information generating unit 304 generates depth information for each zone, it determines whether or not there is a point of the three-dimensional model in front of the zone, and determines the granularity of the quantization. The depth information generating unit 304 also inputs, for example, information indicating whether or not a point of the three-dimensional model exists in front of the zone to the texture information generating unit 305. The texture information generating unit 305 complements the texture information based on the information indicating whether or not a point of the three-dimensional model exists in front of the zone, and generates texture information. This makes it possible to express the three-dimensional model with higher resolution by preventing values ​​that can only be used as depth information or texture information from being set when there is no point of the three-dimensional model on the front side of the zone.

[0071] An example of the flow of data generation processing executed by the data generating device 102 in this modified example will be described with reference to Fig. 21. In this processing, zones are divided and a virtual viewpoint for each zone is determined by processing similar to that in Fig. 4, and then depth information and texture information are generated in association with each other (S2101).

[0072] FIG. 22 shows an example of the flow of the generation process of depth information and texture information executed in S2101. First, the depth information generation unit 304 acquires information on the number of zones from the zone setting unit 302 (S2201). The depth information generation unit 304 executes the processes of S2204 to S2225 for each of the set zones. In addition, since the following processes are repeatedly executed for each of the virtual viewpoints and pixels, the depth information generation unit 304 executes an initialization process for counting them (S2202). Then, the depth information generation unit 304 starts the process for the first zone (NO in S2203). The depth information generation unit 304 acquires zone setting information for the zone to be processed from the zone setting unit 302 (S2204). For example, the depth information generation unit 304 acquires information "d" indicating the bit depth of the depth information. Then, the depth information generation unit 304 executes the process of S2206 to S2223 for each virtual viewpoint set in each zone. In this process, the depth information generation unit 304 acquires virtual viewpoint information for the virtual viewpoint to be processed (S2206). The depth information generation unit 304 also acquires three-dimensional model data of an object included in the zone to be processed from the three-dimensional model acquisition unit 301 (S2207). Here, the depth information generation unit 304 executes a process of determining whether or not a point of the three-dimensional model exists in front of the zone to be processed based on the acquired three-dimensional model (S2208).

[0073] Here, the process of S2208 will be described with reference to FIG. 23. The process of FIG. 23 is a process for determining whether or not at least one point of a three-dimensional model exists in front of a zone to be processed as viewed from a virtual viewpoint to be processed. The depth information generating unit 304 first acquires virtual viewpoint information of a virtual viewpoint to be processed from the virtual viewpoint setting unit 303 (S2301). Then, the depth information generating unit 304 initializes a flag indicating whether or not a point of a three-dimensional model exists in front of a zone as viewed from a virtual viewpoint (S2302). When the value of this flag is "0", it indicates that no point of a three-dimensional model exists in front of a zone as viewed from a virtual viewpoint, and when the value of this flag is "1", it indicates that a point of a three-dimensional model exists in front of a zone as viewed from a virtual viewpoint. In S2302, the value of this flag is set to 0. Then, the depth information generating unit 304 initializes a counter so that the following confirmation process can be performed for all pixels of a depth image to be generated for a zone to be processed and a virtual viewpoint (S2303). Then, the depth information generating unit 304 acquires the position of the target point of the three-dimensional model to be mapped to each pixel (S2305), and judges whether the target point exists within the zone to be processed (S2306). If the target point exists within the zone to be processed (YES in S2306), the depth information generating unit 304 increments the counter (S2307) and processes the next pixel. On the other hand, if the target point exists outside the zone to be processed (NO in S2306), the depth information generating unit 304 judges whether the position is in front of the zone as viewed from the virtual viewpoint (S2308). If the target point is not in front of the zone as viewed from the virtual viewpoint (NO in S2308), the depth information generating unit 304 increments the counter (S2307) and processes the next pixel. On the other hand, if the position of the target point is in front of the zone as viewed from the virtual viewpoint (YES in S2308), the depth information generation unit 304 sets the flag that was set to 0 in S2302 to 1 (S2309). Then, the depth information generation unit 304 sets a depth information complement value and a texture information complement value (S2310) and ends the processing. Note that, if no corresponding point of the three-dimensional model exists in front of the zone for any pixel (YES in S2304), the depth information generation unit 304 ends the processing as it is.In this case, the value of the flag remains 0. Also, the depth information complement value and the texture information complement value are not set because there are no pixels to be complemented by the complement value. Note that the depth information generation unit 304 can notify the texture information generation unit 305 of the above-mentioned flag, for example.

[0074] Returning to Fig. 22, the depth information generation unit 304 repeatedly executes the processes from S2210 to S2221 for each pixel of the depth image to be generated. First, the depth information generation unit 304 identifies the position of a point of a three-dimensional model corresponding to a pixel to be processed (S2210). Then, the depth information generation unit 304 assigns subsequent processes depending on whether or not a point of the three-dimensional model exists in front of the zone to be processed (i.e., whether the value of the flag is 1 or 0) (S2211).

[0075] If there is no point of the three-dimensional model in front of the zone to be processed (i.e., if the flag is 0, NO in S2211), the depth information generating unit 304 makes it possible to express the distance between the virtual viewpoint and the target point using all values ​​of the set bit depth. This quantization using all ranges of values ​​expressible with the set bit depth is called first quantization (S2212). In the first quantization, the value obtained by subtracting the minimum distance between the virtual viewpoint and the zone from the distance between the target point in the zone and the virtual viewpoint is multiplied by 2. d The depth information generating unit 304 then uses the first quantization to obtain depth information corresponding to the target point (S2213). The texture information generating unit 305 sets the texture value of the target point as the pixel value of the texture image (S2214).

[0076] On the other hand, when a point of the three-dimensional model is present in front of the zone to be processed (i.e., when the flag is 1, YES in S2211), the depth information generating unit 304 expresses the distance between the virtual viewpoint and the target point using a range of values ​​excluding the values ​​used for the depth information complement value. This quantization using a part of the range of values ​​expressible at the set bit depth excluding the values ​​used for the depth information complement value is called second quantization (S2215). In the second quantization, the value obtained by subtracting the minimum distance between the virtual viewpoint and the zone from the distance between the target point in the zone and the virtual viewpoint is calculated as (2 d In the second quantization, if there are k values ​​that are not used to represent the distance to the target point, the quantization is performed in steps of (2 d -k) stages of quantization. In this way, the granularity of the first quantization when there is no point of the three-dimensional model in front of the zone to be processed (when the flag is 0) is finer than the granularity of the second quantization when there is a point of the three-dimensional model in front of the zone to be processed (when the flag is 1). In other words, when it is not necessary to use the interpolated value, by making the granularity of the quantization finer, Then, for a target point within the zone (YES in S2216), the depth information generating unit 304 uses the second quantization to obtain depth information corresponding to the distance from the virtual viewpoint to the target point (S2217). The texture information generating unit 305 sets the texture value of the target point as the pixel value of the texture image (S2218). On the other hand, for a target point outside the zone (NO in S2216), the depth information generating unit 304 sets the depth information complement value as the depth information of the pixel corresponding to the target point (S2219). In addition, the texture information generating unit 305 sets the texture value corresponding to the target point as the texture information of the pixel corresponding to the target point (S2220). After completing the setting of the depth information and texture information for the target point to be processed, the counter is incremented (S2221).

[0077] Thereafter, when the depth information generating unit 304 completes the processing for the target points corresponding to all pixels (YES in S2209), it resets the counter that counts the number of pixels (S2222) and increments the counter for the virtual viewpoint (S2223). Thereafter, when the depth information generating unit 304 completes the processing for all virtual viewpoints of the processing target zone (YES in S2205), it resets the counter for the virtual viewpoint (S2224) and increments the counter for the zone (S2225). Then, when the depth information generating unit 304 and the texture information generating unit 305 complete the processing for all zones, they output the depth information and the texture information to the encoding unit 104 (S2226) and complete the processing. Note that the depth information generating unit 304 further outputs the above-mentioned flag information to the encoding unit 104. Furthermore, when the flag is 1, the depth information generation unit 304 and the texture information generation unit 305 can output the depth information complement value and the texture information complement value to the encoding unit 104.

[0078] 23 may operate to set the flag to 1 when a point on the boundary surface of a zone visible from the virtual viewpoint is inside the shape of the three-dimensional model, that is, when the three-dimensional model is cut at that boundary surface. Even if a point of the three-dimensional model is on the near side of the zone, if the three-dimensional model is not cut by the boundary surface of the zone seen from the virtual viewpoint, the interpolation process is not performed. This makes it possible to change the quantization method depending on whether the interpolation process is performed or not.

[0079] The encoding unit 104 generates encoded data in which the input flag and depth information complement value are included in Zone_information_data for each zone. An example of encoded data in this case is shown in Fig. 25. In this modification, the Zone_information_data includes flag_out_of_zone as the above-mentioned flag, and default_out_of_zone as the depth information complement value when the flag is 1. This information may also include a texture information complement value.

[0080] Next, the process of the reproduction device 122 reconstructing a three-dimensional model using the encoded data obtained in this modification will be described with reference to FIG. 24. Note that, among the processes in FIG. 24, the processes common to FIG. 13 are denoted by common reference symbols and will not be described. The encoded data as shown in FIG. 25 is decoded by the decoding unit 1201, and zone setting information including a flag (flag_out_of_zone) and information on a zone information complement value when the flag is 1 is output to the zone information acquisition unit 1202. The depth information acquisition unit 1204 acquires the flag information from the zone information acquisition unit 1202 (S2401), and when it is confirmed that the flag is 1 (YES in S2402), acquires a depth information complement value (S2403). Thereafter, the depth information acquisition unit 1204 performs processing on each zone, each virtual viewpoint, and each pixel. Then, the depth information acquisition unit 1204 switches the process for the point of the three-dimensional model corresponding to the pixel to be processed depending on whether the flag is 1 or not (S2404).

[0081] If the flag is 0 (NO in S2404), the depth information acquisition unit 1204 sets the pixel value to 2 d The depth information acquisition unit 1204 identifies the depth information that was quantized in the previous step and reproduces the distance information between the virtual viewpoint and the target point by inverse quantization (S2405). The depth information acquisition unit 1204 then sets a voxel based on the distance information (S2406). The texture information acquisition unit 1205 also sets a texture value for the target point using the pixel value of the pixel to be processed in the texture image (S2407).

[0082] On the other hand, if the flag is 0 (YES in S2404), the depth information acquisition unit 1204 determines whether the pixel value of the pixel to be processed is a depth information complement value (S2408). If the depth information acquisition unit 1204 determines that the pixel value of the pixel to be processed is not a depth information complement value (NO in S2408), the depth information acquisition unit 1204 dThe depth information acquisition unit 1204 then determines that the acquired depth information is depth information that has been quantized at the first (-1) stage. The depth information acquisition unit 1204 then reproduces distance information between the virtual viewpoint and the target point by inverse quantization (S2409). The depth information acquisition unit 1204 then sets a voxel based on the distance information (S2406). The texture information acquisition unit 1205 also sets a texture value at the target point using the pixel value of the pixel to be processed in the texture image (S2407). On the other hand, if the depth information acquisition unit 1204 determines that the pixel value of the pixel to be processed is a depth information complement value (YES in S2408), it does not set a voxel.

[0083] As described above, in this modification, the quantization granularity of the depth information and texture information related to the three-dimensional model belonging to the zone is changed depending on whether or not there is a point of the three-dimensional model of the object in front of the zone as viewed from the virtual viewpoint. As a result, if there is no point of the three-dimensional model on the front side of the zone, the bit depth can be fully used to generate the depth information of the three-dimensional model. This makes it possible for the playback device 122 to specify the distance with higher accuracy, thereby improving the accuracy of reconstruction of the three-dimensional model.

[0084] The above-mentioned processing examples can be used in any combination. In one example, a depth image may be generated by switching between the procedures described in FIG. 6 and FIG. 7, the procedures described in FIG. 15, and the procedures described in FIG. 22 for each virtual viewpoint. In this case, Virtual_view_point_information_data of the encoded data may include information indicating which procedure was used for encoding. The procedures may be switched not for each virtual viewpoint but for each zone. In this case, Zone_information_data of the encoded data may include information indicating which procedure was used for encoding. The playback device 122 may reconstruct a three-dimensional model using the procedures corresponding to each of the information.

[0085] In the above description, a virtual viewpoint is set for each zone, but this is not limited to the above. For example, a virtual camera located at a position surrounding a three-dimensional model of a large-sized object may be used as a virtual camera for multiple zones that divide the three-dimensional model. That is, a virtual viewpoint common to multiple zones may be used. In the above example, an example in which both depth information and texture information are used has been described, but for example, only depth information may be used. In this case, only the three-dimensional shape of the object is reproduced.

[0086] In the above description, a case where a plurality of zones are set has been described, but the above processing can also be applied when one zone including a part of a three-dimensional model of an object is set. For example, when a user desires to observe only a specific part of a three-dimensional shape, only one zone corresponding to that part may be set. The zones may also be predefined. For example, the virtual space may be divided into predefined zones. In that case, when a specific object in the virtual space is specified, a zone including at least a part of the object is specified. The virtual viewpoint for each zone may also be predefined for each zone. In this case, the data generating device 102 may not have the zone setting unit 302 or the virtual viewpoint setting unit 303. In addition, in the above processing example, the zone information and the virtual viewpoint information are included in the encoded data, but these do not necessarily have to be included. For example, when information on the virtual viewpoint is held in advance in the playback device 122, at least depth images for the plurality of virtual viewpoints are acquired, so that at least the three-dimensional shape of the object can be reproduced. For this reason, the encoded data may include at least depth images corresponding to the plurality of virtual viewpoints, and other information may be included as an option as necessary. Furthermore, when no data is transmitted, only a depth image may be generated in the data generating device 102. Note that the above-described technique can be applied to both moving images and still images.

[0087] (Variation 3) In the above embodiment, when a model of a zone is viewed from a virtual viewpoint, target points outside the zone are detected and their values ​​are complemented to obtain a depth image and a texture image. In this modified example, regardless of whether there is point cloud data between the virtual viewpoint and the model of the zone, data of the model within the zone or complement data is directly used.

[0088] Fig. 26 shows an example of the functional configuration of a data generating device 102 according to this modification. Components that perform the same functions as those of the data generating device 102 in Fig. 3 are given the same reference numerals, and descriptions thereof will be omitted. The data generating device 102 includes a boundary surface determination image acquisition unit 2601 in addition to the functional configuration in Fig. 3. Furthermore, the data generating device 102 includes a depth information generating unit 2602 and a texture information generating unit 2603 instead of the depth information generating unit 304 and the texture information generating unit 305.

[0089] The boundary surface determination image acquisition unit 2601 generates a boundary surface determination image that indicates whether each point on the boundary surface of the zone as viewed from the virtual viewpoint is an inner point or an outer point of the three-dimensional model. That is, a boundary surface determination image is generated that indicates whether a cross section of the three-dimensional model cut by the boundary surface of the zone is observed from the virtual viewpoint. For example, as shown in FIG. 5(A), a three-dimensional model is divided into zones 501 and 502. In this case, the surface that separates zones 501 and 502 is the boundary surface. Then, for example, when zone 501 is observed from virtual viewpoint 513, the boundary surface is observed, and an image that indicates whether each point on the boundary surface corresponds to the cross section of the three-dimensional model is the boundary surface determination image. FIG. 27 shows an example of the boundary surface determination image generated in this way. As shown in the example of FIG. 27, for example, the range 2701 inside (cross section) of the three-dimensional model is displayed in black, and the range outside the three-dimensional model is displayed in white. Note that this is just an example, and an image in a format different from that of FIG. 27 may be generated. Also, data other than image format may be generated as long as the position of the cross section of the three-dimensional model cut by the boundary surface of the zone can be specified. In this embodiment, a bitmap image with one pixel value is generated, in which the value of the points in the black area is "1" and the value of the points in the white area is "0". Here, since the boundary surface determination image indicates whether each point of the boundary surface exists inside the three-dimensional model or not, even if a point of the three-dimensional model exists on the boundary surface, the point is a point on the surface of the three-dimensional model and not an internal point, so the pixel value of this point is "0". For example, in FIG. 28, points 525 and 526 are points inside the three-dimensional model, while point 524 is a point on the boundary surface 541, but is a point on the surface of the three-dimensional model and is not an internal point. Therefore, in the boundary surface determination image, the pixels corresponding to points 525 and 526 are set to "1", and the pixels of points 524 and 523 are set to "0".

[0090] The depth information generating unit 2602 differs from the depth information generating unit 304 described with reference to FIG. 3 in that the depth information generating unit 2602 inputs a boundary surface determination image and uses it for determination. Here, with reference to FIG. 29, a process in which the depth information generating unit 2602 generates depth information in S409 of FIG. 4 will be described. In the following description, the same reference numerals are used for steps in which the same processes as those in FIG. 6 are performed, and description thereof will be omitted. The depth information generating unit 2602 acquires a boundary surface determination image from the boundary surface determination image generating unit 2601 (S2901). In addition, the depth information generating unit 2602 calculates a distance d from the virtual viewpoint to the boundary surface at a point inside the three-dimensional model (S2902). For example, when the boundary surface and the virtual viewpoint are directly opposed to each other (the normal direction of the boundary surface coincides with the line of sight from the virtual viewpoint), the distance d can be calculated as the distance between the center of the range of the boundary surface that is inside the three-dimensional model and the virtual viewpoint. When the boundary surface and the virtual viewpoint do not face each other directly, the depth information generating unit 2602 generates a distance image having a different distance d for each pixel of the boundary surface determination image.

[0091] The depth information generating unit 2602 then determines, as a target point, a point projected onto the pixel position of the processing target among the point group of the data of the three-dimensional model, and determines whether or not the target point is a point projected onto the range 2701 inside the three-dimensional model in the boundary surface determination image (S2903). If the depth information generating unit 2602 determines that the target point is a point projected onto the range 2701 (YES in S2903), it acquires the position of the target point on the boundary surface (S2904). Then, the depth information generating unit 2602 determines a value (distance d) corresponding to the distance from the position of the virtual viewpoint to the target point as depth information for the pixel of the processing target (S2905). If the depth information generating unit 2602 determines that the target point is a point projected outside the range 2701 (NO in S2903), it can specify that the target point is a point on the surface of the three-dimensional model. For this reason, the depth information generating unit 2602 acquires the position of the point to be processed (S2906), and sets a value corresponding to the distance from the position of the virtual viewpoint to the target point as the depth information of the depth image (S2907).

[0092] The texture information generating unit 2603 differs from the texture information generating unit 305 described with reference to FIG. 3 in that the texture information generating unit 2603 receives a boundary surface determination image and uses it for determination. The process of generating texture information in S410 of FIG. 4 by the texture information generating unit 2603 will be described with reference to FIG. 30. In the following description, steps in which the same processes as those in FIG. 9 are performed are given the same reference numerals and description thereof will be omitted. The texture information generating unit 2603 acquires a boundary surface determination image from the boundary surface determination image generating unit 2601 (S3001). Then, the texture information generating unit 2603 sets a point projected onto a pixel position to be processed among the point group of the data of the three-dimensional model as a target point, and determines whether or not the target point is a point projected onto the inner range 2701 of the three-dimensional model in the boundary surface determination image (S3002). When the texture information generating unit 2603 determines that the target point is a point projected onto the range 2701 (YES in S3002), it acquires the position of the target point on the boundary surface (S3003). Then, the texture information generating unit 2603 replaces the pixel value of the texture image corresponding to the target point with the texture complement value set in S902 (S3004). On the other hand, when the texture information generating unit 2603 determines that the target point is a point projected outside the range 2701 (NO in S3003), it can identify the target point as a point on the surface of the three-dimensional model. Therefore, the texture information generating unit 2603 acquires the position of the target point (S3005), acquires texture information of the target point, and sets this as the pixel value of the texture image (S3006).

[0093] In this manner, for the portion (cut portion) where the surface of the object does not exist as a result of the division by zones, the pixel values ​​of the depth image and the texture image are complemented, and these images can be generated and encoded. In addition, in this modified example, it is not necessary to consider target points outside the zone when making a judgment, and only points within the zone are processed, so that the amount of data input and output can be reduced. In addition, in this modified example, since processing can be performed independently for each zone, data reading can be simplified and parallel processing can be easily performed. Note that the above-mentioned embodiment and modified examples can be used in any combination.

[0094] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0095] (Summary of the embodiment) At least some of the above-described embodiments can be summarized as follows. (Item 1) a generating means for generating a depth image for a zone including a part of a three-dimensional model of an object, the depth image having pixel values ​​corresponding to distances between each of points indicating a surface of the shape of the three-dimensional model included in the zone when viewed from a virtual viewpoint and the virtual viewpoint; The generating means includes: When a point on the boundary surface of the zone when viewed from the virtual viewpoint is present inside the shape of the three-dimensional model, a pixel value of the depth image corresponding to the point is complemented with an complement value; generating the depth images for a plurality of the virtual viewpoints; A generating device characterized by: (Item 2) 2. The generating device according to item 1, further comprising a setting means for setting a plurality of said zones, each of said zones including a portion of said three-dimensional model. (Item 3) 3. The generating device according to item 2, wherein the setting means further sets a plurality of the virtual viewpoints for each of the plurality of the zones. (Item 4) 4. The generating device according to any one of items 1 to 3, characterized in that, when a point of the boundary surface exists inside the shape of the three-dimensional model, the generating means uses, as the interpolated value, a pixel value corresponding to the distance between the virtual viewpoint and the point. (Item 5) The generating device according to any one of items 1 to 3, characterized in that when a point of the boundary surface is present inside the shape of the three-dimensional model, the generating means uses, as the interpolated value, a predetermined value that is not used as a pixel value corresponding to the distance between the virtual viewpoint and the surface of the shape of the three-dimensional model included in the zone. (Item 6) The generating means includes: expressing pixel values ​​corresponding to distances between each of the points indicating the surface of the shape of the three-dimensional model and the virtual viewpoint in a value range excluding the predetermined value when any of the points of the boundary surface exists inside the shape of the three-dimensional model; If any of the points of the boundary surface are present outside the shape of the three-dimensional model, pixel values ​​corresponding to the distance between each of the points indicating the surface of the shape of the three-dimensional model and the virtual viewpoint are expressed in a value range including the predetermined value. 6. The generating device according to item 5, characterized in that (Item 7) The generating device described in any one of items 1 to 6, characterized in that the generating means further generates a texture image indicating a texture for the three-dimensional model of the object seen from the virtual viewpoint, and complements pixel values ​​of the texture image for points in the depth image that are complemented using the complement value using a second complement value. (Item 8) The generating device according to any one of items 1 to 7, further comprising an encoding means for generating encoded data including at least the depth image, the setting information of the zone, and the setting information of the virtual viewpoint. (Item 9) an acquisition means for acquiring a depth image for each of a plurality of zones each including a portion of a three-dimensional model of an object, the depth image having pixel values ​​corresponding to the distance between each of a plurality of virtual viewpoints and each of points indicating the surface of the shape of the three-dimensional model of the portion included in the zone when viewed from each of the plurality of virtual viewpoints, the pixel values ​​of the depth image corresponding to points on the boundary surface of the zone when viewed from the virtual viewpoints are complemented using complementation values; a generating means for generating, for each of the plurality of zones, a three-dimensional model of a portion of the object in that zone from the depth image for each of the plurality of virtual viewpoints; a reconstruction means for reconstructing a three-dimensional model of the object by integrating three-dimensional models of a portion of the object for each of the plurality of zones generated for each of the plurality of zones; A playback device comprising: (Item 10) 10. The playback device according to item 9, wherein the generation means generates a three-dimensional model of a portion of the object in the zone by setting voxels based on the pixel values ​​of pixels included in the depth image. (Item 11) Item 11. The playback device according to item 10, wherein the generating means does not set a voxel for pixels in the depth image to which a value that is not used to indicate the distance is set. (Item 12) The playback device described in any one of items 9 to 11, characterized in that the acquisition means acquires, together with the depth image, information indicating whether a value not used to indicate the distance in the depth image is used as the complementary value. (Item 13) The generating means includes: When information indicating that a value not used to indicate the distance is used as the interpolation value is acquired, the pixel value corresponding to the distance in the depth image is quantized with a first granularity to identify the distance; When information indicating that a value not used to indicate the distance is not used as the interpolation value is acquired, the pixel value corresponding to the distance in the depth image is quantized at a second granularity finer than the first granularity to identify the distance; generating a three-dimensional model of a portion of the object based on the determined distance; 13. The playback device according to item 12. (Item 14) 14. The playback device according to any one of items 9 to 13, wherein the acquisition means further acquires a texture image indicating a texture for the three-dimensional model of the object as seen from the virtual viewpoint. (Item 15) Item 15. The playback device according to item 14, characterized in that the texture image is an image in which pixel values ​​corresponding to points in the depth image that are interpolated using the interpolation value are interpolated using a second interpolation value. (Item 16) The playback device according to any one of items 9 to 15, characterized in that the acquisition means acquires encoded data including at least the depth image, the zone setting information, and the virtual viewpoint setting information. (Item 17) An information processing method performed by a generating device, comprising: generating a depth image for a zone including a part of a three-dimensional model of an object, the depth image having pixel values ​​corresponding to distances between the virtual viewpoint and each of points indicating a surface of the shape of the three-dimensional model included in the zone when viewed from the virtual viewpoint; In generating the depth image, When a point on the boundary surface of the zone when viewed from the virtual viewpoint is present inside the shape of the three-dimensional model, a pixel value of the depth image corresponding to the point is complemented with an complement value; generating the depth images for a plurality of the virtual viewpoints; 23. An information processing method comprising: (Item 18) An information processing method executed by a playback device, comprising: acquiring a depth image for each of a plurality of zones each including a portion of a three-dimensional model of an object, the depth image having pixel values ​​corresponding to distances between each of a plurality of virtual viewpoints and each of points indicating a surface of the shape of the three-dimensional model of the portion included in the zone when viewed from each of the plurality of virtual viewpoints, the depth image being interpolated with an interpolation value when a point on the boundary surface of the zone when viewed from the virtual viewpoint is present inside the shape of the three-dimensional model of the object; generating, for each of the plurality of zones, a three-dimensional model of a portion of the object in that zone from the depth images for each of the plurality of virtual viewpoints; reconstructing a three-dimensional model of the object by integrating the three-dimensional models of the portion of the object for each of the plurality of zones generated for each of the plurality of zones; 13. An information processing method comprising: (Item 19) A program for causing a computer to function as the generating device according to any one of items 1 to 8. (Item 20) A program for causing a computer to function as the playback device according to any one of items 9 to 16.

[0096] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0097] 102: data generating device, 103: data generating unit, 104: encoding unit, 122: playback device, 123: data playback unit, 124: three-dimensional model reconstruction unit, 302: zone setting unit, 303: virtual viewpoint setting unit, 304: depth information generating unit, 305: texture information generating unit, 1202: zone information acquiring unit, 1203: virtual viewpoint information acquiring unit, 1204: depth information acquiring unit, 1205: texture information acquiring unit, 1211: three-dimensional model generating unit, 1212: zone integration unit

Claims

1. a generating means for generating a depth image for a zone including a part of a three-dimensional model of an object, the depth image having pixel values ​​corresponding to distances between each of points indicating a surface of the shape of the three-dimensional model included in the zone when viewed from a virtual viewpoint and the virtual viewpoint; The generating means includes: When a point on the boundary surface of the zone when viewed from the virtual viewpoint is present inside the shape of the three-dimensional model, a pixel value of the depth image corresponding to the point is complemented with an complement value; generating the depth images for a plurality of the virtual viewpoints; A generating device characterized by:

2. The generating device according to claim 1 , further comprising setting means for setting a plurality of said zones, each of said zones including a portion of said three-dimensional model.

3. The generating device according to claim 2 , wherein the setting means further sets a plurality of the virtual viewpoints for each of a plurality of the zones.

4. The generating device according to claim 1, characterized in that, when a point of the boundary surface exists inside the shape of the three-dimensional model, the generating means uses, as the interpolated value, a pixel value corresponding to the distance between the virtual viewpoint and the point.

5. The generating device described in claim 1, characterized in that when the boundary surface point is located inside the shape of the three-dimensional model, the generating means uses, as the interpolated value, a predetermined value that is not used as a pixel value corresponding to the distance between the virtual viewpoint and the surface of the shape of the three-dimensional model included in the zone.

6. The generating means includes: expressing pixel values ​​corresponding to distances between each of the points indicating the surface of the shape of the three-dimensional model and the virtual viewpoint in a range excluding the predetermined value when any of the points of the boundary surface exists inside the shape of the three-dimensional model; If any of the points of the boundary surface are present outside the shape of the three-dimensional model, pixel values ​​corresponding to the distance between each of the points indicating the surface of the shape of the three-dimensional model and the virtual viewpoint are expressed in a value range including the predetermined value. The generating device according to claim 5 .

7. The generating device described in claim 1, characterized in that the generating means further generates a texture image indicating a texture for the three-dimensional model of the object viewed from the virtual viewpoint, and complements pixel values ​​of the texture image for points in the depth image that are complemented using the complement value using a second complement value.

8. The generating device according to claim 1 , further comprising an encoding means for generating encoded data including at least the depth image, the setting information of the zone, and the setting information of the virtual viewpoint.

9. an acquisition means for acquiring a depth image for each of a plurality of zones each including a portion of a three-dimensional model of an object, the depth image having pixel values ​​corresponding to the distance between each of a plurality of virtual viewpoints and each of points indicating the surface of the shape of the three-dimensional model of the portion included in the zone when viewed from each of the plurality of virtual viewpoints, the pixel values ​​of the depth image corresponding to points on the boundary surface of the zone when viewed from the virtual viewpoints are complemented using complementation values; a generating means for generating, for each of the plurality of zones, a three-dimensional model of a portion of the object in that zone from the depth image for each of the plurality of virtual viewpoints; a reconstruction means for reconstructing a three-dimensional model of the object by integrating three-dimensional models of a portion of the object for each of the plurality of zones generated for each of the plurality of zones; A playback device comprising:

10. The playback device according to claim 9 , wherein the generating means generates a three-dimensional model of a portion of the object in the zone by setting voxels based on the pixel values ​​of pixels included in the depth image.

11. The playback device according to claim 10 , wherein the generating means does not set a voxel for a pixel in the depth image to which a value that is not used to indicate the distance is set.

12. The playback device according to claim 9, characterized in that the acquisition means acquires, together with the depth image, information indicating whether a value not used to indicate the distance in the depth image is used as the complementary value.

13. The generating means includes: When information indicating that a value not used to indicate the distance is used as the interpolation value is acquired, the pixel value corresponding to the distance in the depth image is quantized with a first granularity to identify the distance; When information indicating that a value not used to indicate the distance is not used as the interpolation value is acquired, the pixel value corresponding to the distance in the depth image is quantized at a second granularity finer than the first granularity to identify the distance; generating a three-dimensional model of a portion of the object based on the determined distance; 13. The playback device according to claim 12.

14. 10. The playback device according to claim 9, wherein the acquisition means further acquires a texture image indicative of a texture for the three-dimensional model of the object as viewed from the virtual viewpoint.

15. The playback device according to claim 14 , wherein the texture image is an image in which pixel values ​​corresponding to points in the depth image that are interpolated using the interpolation value are interpolated using a second interpolation value.

16. 10. The playback device according to claim 9, wherein the acquisition means acquires encoded data including at least the depth image, the setting information of the zone, and the setting information of the virtual viewpoint.

17. An information processing method executed by a generating device, comprising: generating a depth image for a zone including a portion of a three-dimensional model of an object, the depth image having pixel values ​​corresponding to distances between the virtual viewpoint and each of points indicating a surface of the shape of the three-dimensional model included in the zone when viewed from the virtual viewpoint; In generating the depth image, When a point on the boundary surface of the zone when viewed from the virtual viewpoint is present inside the shape of the three-dimensional model, a pixel value of the depth image corresponding to the point is complemented with an complement value; generating the depth images for a plurality of the virtual viewpoints; 23. An information processing method comprising:

18. An information processing method executed by a playback device, comprising: acquiring a depth image for each of a plurality of zones each including a portion of a three-dimensional model of an object, the depth image having pixel values ​​corresponding to distances between each of a plurality of virtual viewpoints and each of points indicating a surface of the shape of the three-dimensional model of the portion included in the zone when viewed from each of the plurality of virtual viewpoints, the depth image being interpolated with an interpolation value when a point on the boundary surface of the zone when viewed from the virtual viewpoint is present inside the shape of the three-dimensional model of the object; generating, for each of the plurality of zones, a three-dimensional model of a portion of the object in that zone from the depth images for each of the plurality of virtual viewpoints; reconstructing a three-dimensional model of the object by integrating the three-dimensional models of the portion of the object for each of the plurality of zones generated for each of the plurality of zones; 13. An information processing method comprising:

19. A program for causing a computer to function as the generating device according to any one of claims 1 to 8.

20. A program for causing a computer to function as the playback device according to any one of claims 9 to 16.

Images