Information processing apparatus, 3D model generation method, and program

Abstract

A first layer depth map generation unit (131) (distance acquisition unit) of an encoding apparatus (40a) (information processing apparatus) acquires a first distance (d1) between a projection point (Q) and a first point (P1) based on a 3D shape of a surface of a subject (90), a straight line extending from the projection point (Q) to the subject (90) contacts the subject (90) at the first point. A second layer depth map generation unit (132) (distance acquisition unit) acquires a second distance (d2) between the projection point (Q) and a second point (P2) based on the 3D shape of the surface of the subject (90), a straight line extending from the first point (P1) and into an interior of the subject (90) comes out of the subject (90) at the second point.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to an information processing apparatus, a 3D model generation method, and a program. BACKGROUND

[0002] Conventionally, a method of generating a 3D object in a viewing space by using information obtained by sensing a real 3D space, for example, by using a multi-view video obtained by imaging a subject from different viewpoints, to generate a video (volume video) in which the object looks as if the object is located in the viewing space has been proposed.

[0003] The generation of a volume video requires coordinate information about a three-dimensional space. For example, in Patent Literature 1, a 3D shape of a subject is obtained based on a depth map indicating a distance from an imaging device to a surface of the subject.

[0004] LIST OF CITATIONS

[0005] PATENT LITERATURE

[0006] Patent Literature 1: WO 2018 / 074252 A SUMMARY

[0007] TECHNICAL PROBLEM

[0008] However, in Patent Literature 1, only a 3D shape of a front surface of a subject, that is, a front surface side of the subject, facing the imaging device can be obtained. Therefore, it is necessary to obtain a 3D shape of a back surface side of the subject based on a depth map captured from the back surface side of the subject.

[0009] Therefore, there is a problem that it is difficult to generate an accurate shape of a subject.

[0010] The present disclosure proposes an information processing apparatus, a 3D model generation method, and a program configured to generate an accurate shape of a subject.

[0011] SOLUTION TO PROBLEM

[0012] To solve the above problem, an information processing apparatus according to an embodiment of the present disclosure includes a distance acquisition unit that acquires a first distance between a projection point and a first point at which a straight line extending from the projection point to a subject is in contact with the subject, and a second distance between the projection point and a second point at which a straight line extending from the first point into an interior of the subject penetrates to an exterior of the subject, based on a 3D shape of a surface of the subject. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 is a block diagram showing a schematic configuration of an information processing system according to a first embodiment.

[0014] Figure 2 is a diagram showing an example of an imaging device.

[0015] Figure 3 is a flowchart showing an example of processing performed by an information processing system.

[0016] Figure 4 is a hardware block diagram showing an example of a hardware configuration of an information processing system.

[0017] Figure 5 is a functional block diagram showing an example of a functional configuration of an encoding unit.

[0018] Figure 6 is a diagram showing a method of generating a depth map.

[0019] Figure 7 is a diagram showing quantized depth values of a first layer depth map and a second layer depth map.

[0020] Figure 8 is a diagram showing an example of a first layer depth map and a second layer depth map.

[0021] Figure 9 is a diagram showing an example of an image generated by a tiling unit.

[0022] Figure 10 is a functional block diagram showing an example of a functional configuration of a decoding unit.

[0023] Figure 11 is a diagram showing a method of reading depth values from each depth map by a depth valid region determination unit.

[0024] Figure 12 is a diagram showing a method of determining validity / invalidity of depth values by a depth valid region determination unit.

[0025] Figure 13 is a flowchart showing an example of processing performed by a depth valid region determination unit.

[0026] Figure 14 is a diagram showing triangular patches set on each depth map.

[0027] Figure 15 is a diagram showing the number of valid points of each triangular patch.

[0028] Figure 16 is a diagram showing a vertex coordinate change processing.

[0029] Figure 17 is a diagram showing how three-dimensional mesh data about a subject is generated.

[0030] Figure 18is a flowchart showing an example of mesh data generation processing performed by a triangle patch generation unit.

[0031] Figure 19 is a functional block diagram showing an example of a functional configuration of a rendering unit.

[0032] Figure 20 is a diagram showing an example of an image to be encoded generated by a tiling unit in the second embodiment.

[0033] Figure 21 is a functional block diagram showing an example of a functional configuration of a decoding unit and a rendering unit of the second embodiment.

[0034] Figure 22 is a diagram showing visibility confirmation.

[0035] Figure 23 is a diagram showing functions of an information processing system according to the third embodiment.

[0036] Figure 24 is a diagram showing functions of an information processing system according to the fourth embodiment. DETAILED DESCRIPTION

[0037] Embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Note that in the following embodiments, the same portions are denoted by the same reference numerals or symbols, and repetitive description thereof will be omitted.

[0038] Further, the present disclosure will be described in the order of items shown below.

[0039] 1. First Embodiment

[0040] 1-1. Functional Configuration of Information Processing System

[0041] 1-2. Processing Performed by Information Processing System

[0042] 1-3. Hardware Configuration of Information Processing System

[0043] 1-4. Detailed Configuration of Encoding Unit

[0044] 1-5. Detailed Configuration of Decoding Unit

[0045] 1-6. Depth Validity Determination Processing

[0046] 1-7. Mesh Data Generation Processing

[0047] 1-8. Detailed Configuration of Rendering Unit

[0048] 1-9. Modified Example of First Embodiment

[0049] 1-10. Effects of First Embodiment

[0050] 2. Second Embodiment

[0051] 2-1. Configuration of Tiled Image

[0052] 2-2. Visibility Confirmation

[0053] 2-3. Effects of Second Embodiment

[0054] 3. Third Embodiment

[0055] 3-1. Functions of Third Embodiment

[0056] 3-2. Effects of Third Embodiment

[0057] 4. Fourth Embodiment

[0058] 4-1. Functions of Fourth Embodiment

[0059] 4-2. Effects of Fourth Embodiment

[0060] 5. Application Examples of Present Disclosure

[0061] 5-1. Production of Content

[0062] 5-2. Experience in Virtual Space

[0063] 5-3. Application to Communication with Remote Location

[0064] 5-4. Other Aspects

[0065] (1. First Embodiment)

[0066] [1-1. Configuration of Information Processing System]

[0067] First, an overview of an information processing system 10a to which the present disclosure is applied will be described with reference to Figure 1 FIG. 1. Figure 1 is a block diagram showing a schematic configuration of an information processing system according to the first embodiment.

[0068] As shown in Figure 1 , the information processing system 10a includes a data acquisition unit 11, a 3D model generation unit 12, an encoding unit 13a, a transmission unit 14, a reception unit 15, a decoding unit 16a, a rendering unit 17a, and a display unit 18.

[0069] The data acquisition unit 11 acquires image data for generating a 3D model 90M of a subject 90 that is an object to be imaged. For example, as shown in Figure 2As shown, the plurality of viewpoint images captured by the plurality of imaging devices 70 (70a, 70b,...) arranged to surround the subject 90 are acquired as image data by the data acquisition unit 11. In this configuration, preferably the plurality of viewpoint images are images captured by the plurality of imaging devices synchronously with each other. Further, for example, the data acquisition unit 11 can acquire image data obtained by moving one imaging device to capture images of the subject from a plurality of viewpoints. Note that the data acquisition unit 11 can perform calibration based on the image data to acquire internal parameters and external parameters of each imaging device. Further, for example, the data acquisition unit 11 can acquire a plurality of pieces of depth information indicating distances to the subject 90 from a plurality of viewpoints.

[0070] The 3D model generation unit 12 generates a model having three-dimensional information on the subject based on the image data used to generate the 3D model 90M of the subject 90. For example, the 3D model generation unit 12 generates a 3D model of the subject by sculpting a three-dimensional shape of the subject from images captured from a plurality of viewpoints (e.g., silhouette images captured from a plurality of viewpoints) using a so-called visual hull. In this configuration, the 3D model generation unit 12 can further deform the 3D model 90M generated with high accuracy using the visual hull by using a plurality of pieces of depth information indicating distances to the subject from a plurality of viewpoints. Further, the 3D model generation unit 12 can generate the 3D model 90M of the subject 90 from one captured image of the subject 90.

[0071] The 3D model 90M generated by the 3D model generation unit 12 in a time series of frames can also be referred to as a moving image of the 3D model. Further, the 3D model 90M is generated using images captured by the imaging device, and thus the 3D model 90M can also be referred to as a live 3D model. In the 3D model, for example, shape information indicating a surface shape of the subject 90 can be expressed in the form of three-dimensional mesh data expressed by connections between vertices called a polygonal mesh. For example, the three-dimensional mesh data includes three-dimensional coordinates of vertices of the mesh and index information indicating whether which vertices are combined to form a triangular mesh. Note that the expression of the 3D model is not limited thereto, and the 3D model can be described by using an expression method of a so-called point cloud expressed by point position information. Color information data is also generated as a texture associated with the above-expressed 3D shape data. For the texture, a viewpoint-independent texture that provides a constant color even when viewed from any direction or a viewpoint-dependent texture that provides a color change depending on a viewing direction is considered.

[0072] The encoding unit 13a converts data on the 3D model 90M generated by the 3D model generation unit 12 into a format suitable for transmission and accumulation of data. In the present embodiment, three-dimensional shape data having a format such as three-dimensional mesh data is converted into a depth information image projected from one or a plurality of viewpoints, that is, a so-called depth map. The depth information and color information as two-dimensional images are compressed and output to the transmission unit 14. The depth information and color information can be transmitted together as one image, or can be transmitted as two separate images. Both are in the form of two-dimensional image data, and can also be compressed using a two-dimensional compression technique such as Advanced Video Coding (AVC).

[0073] The transmission unit 14 transmits transmission data formed by the encoding unit 13a to the reception unit 15. The transmission unit 14 transmits transmission data to the reception unit 15 after performing a series of processes by the data acquisition unit 11, the 3D model generation unit 12, and the encoding unit 13a offline. Furthermore, the transmission unit 14 can transmit transmission data generated according to the series of processes described above to the reception unit in real time.

[0074] Note that the 3D model generation unit 12, the encoding unit 13a, and the transmission unit 14 constitute an encoding apparatus 40a. The encoding apparatus 40a is an example of an information processing apparatus in the present disclosure.

[0075] The reception unit 15 receives transmission data transmitted from the transmission unit 14.

[0076] The decoding unit 16a recovers two-dimensional images from a bitstream received by the reception unit 15, and recovers mesh and texture information for drawing by the rendering unit 17a from the recovered two-dimensional images.

[0077] Note that the reception unit 15, the decoding unit 16a, and the rendering unit 17a constitute a decoding apparatus 50a. The decoding apparatus 50a is an example of an information processing apparatus in the present disclosure.

[0078] The rendering unit 17a performs texture mapping by projecting the mesh of the 3D model 90M with the viewpoint of the drawing camera and mapping a texture representing a color or a pattern. The present embodiment is characterized in that the drawing at this time can be viewed from an arbitrary set viewpoint, regardless of the camera position at the time of image capture. The texture mapping includes a so-called viewpoint-dependent method (VD method) that takes into account the user's viewing viewpoint and a so-called viewpoint-independent method (VI method) that does not take into account the user's viewing viewpoint. In the VD method, the texture to be mapped to the 3D model changes depending on the position of the viewing viewpoint, and thus has the advantage that rendering with higher quality than that of the VI method can be achieved. On the other hand, in the VI method, the position of the viewing viewpoint is not taken into account, and thus has the advantage that the amount of processing required is less than that in the VD method. Note that, based on the detection of the user's viewing viewpoint (region of interest) by the display device, data regarding the viewing viewpoint is input from the display device to the rendering unit 17a.

[0079] The display unit 18 displays the result of rendering by the rendering unit 17a on the display unit of the display device. The display device can be a 2D monitor or a 3D monitor, such as a head-mounted display, a spatial display, a mobile phone, a television, or a PC.

[0080] Note that, Figure 1 The information processing system 10a in FIG. 1 illustrates a series of processing steps from the data acquisition unit 11 that acquires a captured image as material for generating content to the display control unit that controls the display device viewed by the user. However, this does not mean that all the functional blocks are necessary for implementing the present embodiment, but means that the present embodiment can be implemented for each functional block or a combination of a plurality of functional blocks. For example, in the information processing system 10a in FIG. 1, the data acquisition unit 11, the 3D model generation unit 12, and the encoding unit 13a can be implemented by a single information processing device. Figure 1 In FIG. 1, the transmission unit 14 and the reception unit 15 are provided to illustrate a series of processing steps from the side of creating content to the side of viewing content through distribution of content data. However, in a case where the creation of content to the viewing of content is performed by the same information processing device (for example, a personal computer), the transmission unit 14 and the reception unit 15 need not be provided.

[0081] In some cases, the information processing system 10a can be implemented by the same implementer for all the functional blocks, or can be executed by different implementers for each functional block. For example, a company A generates 3D content through the data acquisition unit 11, the 3D model generation unit 12, and the encoding unit 13a. Further, the 3D content is distributed through the transmission unit 14 (platform) of a B company, and a display device of a C company performs reception, rendering, display control of the 3D content. In some cases, there can be a case where a plurality of implementers cooperate with each other.

[0082] Furthermore, each of the above-described functional blocks can be implemented on cloud computing. For example, the rendering unit 17a can be implemented by a display device or can be implemented by a server. In this configuration, information is exchanged between the display device and the server.

[0083] In Figure 1 In the information processing system 10a of the present specification, two or more functional blocks related to each other are referred to as an information processing system, and for example, excluding the display unit 18, the data acquisition unit 11, the 3D model generation unit 12, the encoding unit 13a, the transmission unit 14, the reception unit 15, the decoding unit 16a, and the rendering unit 17a can be collectively referred to as the information processing system 10a.

[0084] [1-2. Processing performed by the information processing system]

[0085] Next, the processing performed by the information processing system 10a will be described with reference to Figure 3 Figure 3 is a flowchart illustrating an example of the processing performed by the information processing system.

[0086] When the processing starts, in step S11, the data acquisition unit 11 acquires image data for generating a 3D model of a subject.

[0087] Next, in step S12, the 3D model generation unit 12 generates a 3D model having three-dimensional information about a subject based on the image data for generating a 3D model of the subject.

[0088] In step S13, the encoding unit 13a encodes the shape and texture data of the 3D model generated by the 3D model generation unit 12 into a format suitable for transmission / accumulation.

[0089] Then, in step S14, the transmission unit 14 transmits the encoded data, and in step S15, the reception unit 15 receives the transmitted data.

[0090] In step S16, the decoding unit 16a performs decoding processing to convert the 3D model into shape and texture data required for display. Then, in step S17, the rendering unit 17a performs rendering by using the shape and texture data. In step S18, the result of the rendering is displayed on the display unit. Then, when the processing of step S18 is completed, the processing of the information processing system 10a ends.

[0091] [1-3. Hardware configuration of the information processing system]

[0092] ​Next, the hardware configuration of the information processing system 10a will be described with reference to Figure 4 The hardware configuration of the information processing system 10a will be described. Figure 4 is a hardware block diagram illustrating an example of the hardware configuration of the information processing system.

[0093] In Figure 4 In the computer illustrated in FIG. 8, the CPU 21, the ROM 22, and the RAM 23 are connected to one another via the bus 24. The input / output interface 25 is also connected to the bus 24. The input unit 26, the output unit 27, the storage unit 28, the communication unit 29, and the drive 30 are connected to the input / output interface 25.

[0094] The input unit 26 includes, for example, a keyboard, a mouse, a microphone, a touch panel, an input terminal, and the like. The output unit 27 includes, for example, a display, a speaker, an output terminal, and the like. The storage unit 28 includes, for example, a hard disk, a RAM disk, a nonvolatile memory, and the like. The communication unit 29 includes, for example, a network interface and the like. The drive 30 drives a removable medium such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0095] In the computer configured as described above, the CPU 21 loads a program stored in the storage unit 28 into the RAM 23 via the input / output interface 25 and the bus 24, and executes a series of processing steps described above. The RAM 23 also appropriately stores data required by the CPU 21 to execute various processing.

[0096] For example, the program executed by the computer can be applied by being recorded on a removable medium such as a package medium or the like. In this configuration, when the removable medium is mounted to the drive, the program can be installed in the storage unit 28 via the input / output interface.

[0097] Further, the program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, and a digital satellite broadcast. In this configuration, the program is allowed to be received by the communication unit 29 and installed in the storage unit 28.

[0098] [1-4. Detailed Configuration of Encoding Unit]

[0099] Next, the detailed configuration of the encoding unit 13a will be described with reference to Figure 5 The detailed configuration of the encoding unit 13a will be described. Figure 5 is a functional block diagram illustrating an example of the functional configuration of the encoding unit.

[0100] The encoding unit 13a receives the three-dimensional mesh data of the subject 90 and the color information texture data C (see FIG. 6) from the 3D model generation unit 12. Then, the encoding unit 13a converts the three-dimensional mesh data and the color information texture data C into bitstream data suitable for transmission / accumulation, and outputs the bitstream data. As described above, the encoding unit 13a is configured to include a mesh data conversion unit 131, a color information conversion unit 132, and a bitstream generation unit 133. Figure 9 ) from the 3D model generation unit 12. Then, the encoding unit 13a converts the three-dimensional mesh data and the color information texture data C into bitstream data suitable for transmission / accumulation, and outputs the bitstream data. As described above, the encoding unit 13a is configured to include a mesh data conversion unit 131, a color information conversion unit 132, and a bitstream generation unit 133.Figure 5 As shown, the encoding unit 13a includes a first layer depth map generation unit 131, a second layer depth map generation unit 132, a color information packing unit 133, a tiling unit 134a, and an image compression processing unit 135.

[0101] Based on the 3D shape acquired by the 3D model generation unit 12, the first layer depth map generation unit 131 acquires a first distance dl between a first point Pl at which a straight line extending from the projection point Q as a starting point to the subject 90 contacts the subject 90 and the projection point Q. Then, the first layer depth map generation unit 131 generates an image visualizing distances to the surface of the subject 90 viewed from the projection point Q, i.e., a first layer depth map Dml. Note that the first layer depth map generation unit 131 is an example of the distance acquisition unit in the present disclosure.

[0102] Based on the 3D shape acquired by the 3D model generation unit 12, the second layer depth map generation unit 132 acquires a second distance d2 between the projection point Q as a starting point and a second point P2 from which a straight line extending from the projection point Q and from the first point Pl into the inside of the subject 90 penetrates to the outside of the subject 90. Then, the second layer depth map generation unit 132 generates an image visualizing distances to the back surface of the subject 90 viewed from the projection point Q, i.e., a second layer depth map Dm2. Note that the second layer depth map generation unit 132 is an example of the distance acquisition unit in the present disclosure. Note that in some cases, a straight line extending from the projection point Q as a starting point to the subject 90 can penetrate the subject 90 multiple times. In such a configuration, a plurality of pairs of the first point Pl and the second point P2 are generated. In other words, each time a straight line penetrates the subject 90, a pair of the first point Pl and the second point P2 is generated.

[0103] As described above, the three-dimensional shape of the subject 90 is restored in the 3D model generation unit 12. Therefore, when a perspective transformation is performed on the three-dimensional shape of the subject 90 from any viewpoint, distances to the front surface of the subject 90 can be expressed as an image. This image provides the above-described first layer depth map Dml and the second layer depth map Dm2.

[0104] Note that, for example, the perspective transformation is calculated by Formula (1).

[0105] [Mathematical Formula 1]

[0106]

[0107] Equation (1) indicates that a two-dimensional coordinate (u, v) and a depth s of a projection plane are obtained by multiplying a matrix of camera external parameters including rotational components r11 to r33 and translational components t1 to t3 and a matrix of camera internal parameters represented by fx, fy, and cx, cy with a point (X, Y, Z) on the subject 90.

[0108] The first layer depth map Dm1 has two-dimensional array data in which distances to the closest points of all projection planes of the three-dimensional mesh of the subject 90 visible from the projection point Q, that is, distances from the projection point Q to the nearest points are stored.

[0109] In the present embodiment, in addition to the first layer depth map Dm1, a second layer depth map Dm2 that records distances to the surface on the back side of the subject 90 is also used. Figure 6 is a diagram showing a method of generating a depth map.

[0110] The first layer depth map generation unit 131 quantizes the depth s calculated as the first distance d1 within a range of a minimum value and a maximum value, and stores the depth s at a point Ps on the projection plane R(u, v) as shown in Figure 6 For example, the minimum value is a distance near from the projection point Q to the near plane Sn in Figure 6 , and for example, the maximum value is a distance far from the projection point Q to the far plane Sf in Figure 6 .

[0111] In the present embodiment, for example, the depth is stored in an 8-bit luminance value to be compressed by Advanced Video Coding (AVC). In other words, the first distance d1 is converted into a quantized depth value D1 by a calculation formula shown in Equation (2). Note that D1 = 0 is stored in a region in which the subject 90 does not exist. Note that hereinafter, the quantized depth value D1 is simply referred to as a depth value D1.

[0112] [Equation 2]

[0113]

[0114] The second layer depth map generation unit 132 stores a depth s calculated as a second distance d2 that passes through the inside of the subject 90 to a point on the back side of the subject 90 using the same perspective projection parameters as those for the first layer, but Figure 6The point Ps on the projection plane R(u, v) is shown at a point Ps. In the region of the depth map in which the subject 90 is projected, the second distance d2 has a value greater than the value of the first distance d1. The quantized depth value D2 of the second layer is calculated by formula (3) in which the distance near and the distance far in formula (2) are interchanged so that the quantized depth value D2 has a value of the opposite magnitude to the depth value D1 of the first layer. Note that in the region in which the subject 90 is not present, D2 = 0 representing an invalid value is stored. This invalid value (D2 = 0) is the same value as that obtained when the second distance d2 is equal to the distance near. Note that round(x) in formula (2) is a function that rounds x to an integer. Hereinafter, the quantized depth value D2 will be simply referred to as the depth value D2.

[0115] [mathematical formula 3]

[0116]

[0117] Note that although Figure 6 An example in which a depth map is generated by perspective projection with the projection point Q as the projection center is shown, but a projection method other than perspective projection, for example, parallel projection can be used to generate a depth map.

[0118] Next, a specific example of the first layer depth map Dm1 and the second layer depth map Dm2 will be described with reference to Figure 7 and Figure 8 Figure 7 is a graph showing the quantized depth values of the first layer depth map and the second layer depth map. Figure 8 is a graph showing an example of the first layer depth map and the second layer depth map.

[0119] As shown in Figure 7 , the encoding unit 13a stores the depth value D1 in the region in which the subject 90 is not present (invalid region) to be stored in the first layer depth map Dm1 as the distance far in the near plane Sn. Furthermore, the depth value D2 in the region in which the subject 90 is not present (invalid region) to be stored in the second layer depth map Dm2 is stored as the distance near smaller than the distance far in the far plane Sf. Note that the distance far is an example of the first distance value in the present disclosure. The distance near is an example of the second distance value in the present disclosure. Thus, it can be determined that the region in which d1 < d2 is a region indicating the subject 90.

[0120] Furthermore, the encoding unit 13a reverses the quantization direction between the first layer and the second layer as expressed in formula (2) and formula (3). Thus, as shown in Figure 7 , the depth values D1 and D2 corresponding to the invalid region after quantization are set to equal values, that is, the depth value D1 and the depth value D2 are both set to 0.​

[0121] As Figure 8 shown in the upper diagram of FIG. 10, if the quantization direction between the first layer and the second layer is the same, the first layer depth map Dm1 and the second layer depth map Dm2 arranged in one image have a gap of depth values on the boundary. On the other hand, as shown in the lower diagram of FIG. 10, reversing the quantization direction between the first layer and the second layer does not generate a gap of depth values on the boundary between the first layer depth map Dm1 and the second layer depth map Dm2 arranged in one image. Thus, when encoding the first layer depth map Dm1 and the second layer depth map Dm2 arranged in one image, the compression efficiency of the image can be improved. Figure 8

[0122] Note that the quantization method described here is an example, and a quantization method other than the linear quantization given in Equations (2) and (3) can be applied. For example, the inverse quantization given in Equation (4) can be applied. According to Equation (4), as the distance increases, quantization is performed using a coarser step size.

[0123] [Equation 4]

[0124]

[0125] Returning to Figure 5 , the color information packing unit 133 performs a process of arranging (packing) the plurality of pieces of color information texture data C input from the 3D model generation unit 12 in one image. This process makes it possible to reduce the number of images to be transmitted to the decoding apparatus 50a. Note that the color information texture data C is an example of texture information in the present disclosure.

[0126] The tiling unit 134a performs a process of collecting the first layer depth map Dm1, the second layer depth map Dm2, and the color information texture data C packed by the color information packing unit 133 into one image. The values (depth values D1 and D2) stored in the first layer depth map Dm1 and the second layer depth map Dm2 are converted into a signal format of Y / Cb / Cr (luminance signal and color difference signal) to be arranged in the same image in which the color information is arranged. In other words, the depth values D1 and D2 are stored in the luminance values. Furthermore, when the image is 8 bits per pixel, a constant value such as a constant value of 128 indicating a gray scale is stored in the color difference. Furthermore, the tiling unit 134a outputs image arrangement information indicating at which position of one collected image the depth map and the color information texture data C are stored. For example, the image arrangement information includes an offset value indicating whether the original image is arranged in one image and how much it is moved in one image, the values of the distances near and far for quantization of the depth map, the position of the projection point Q, and the like.

[0127] Figure 9 ​is a diagram showing an example of an image generated by the tiling unit. The tiling unit 134a collects the first layer depth map Dm1, the second layer depth map Dm2, and the color information texture data C in one tiled image T1. As shown in Figure 9 , the color information texture data C is an actual image obtained by capturing the subject 90 by a plurality of imaging devices 70 (see Figure 2 ).

[0128] The image compression processing unit 135 compresses the images collected in one image in the tiling unit 134a using a general video codec. Specifically, a widely used codec such as AVC or HEVC is used.

[0129] Note that in the above equations (2) and (3), the depth values D1 and D2 are quantized with values of 0 to 255 in a case where it is assumed that the depth values D1 and D2 are 8 bits. However, values of 10 bits can be handled in HEVC or the like, and the depth values D1 and D2 can be quantized with 10 bits (0 to 1023).

[0130] The bitstream generated by the image compression processing unit 135 and the image arrangement information generated by the tiling unit 134a are transmitted to the decoding device 50a via the transmission unit 14. In other words, the transmission unit 14 transmits the first distance d1 and the second distance d2 to the decoding device 50a that reconstructs the three-dimensional shape of the subject 90 in association with each other.

[0131] [1-5. Detailed configuration of decoding unit]

[0132] Next, a detailed configuration of the decoding unit 16a will be described with reference to Figure 10 . Figure 10 is a functional block diagram showing an example of a functional configuration of the decoding unit.

[0133] As shown in Figure 10 , the decoding unit 16a includes an image decompression processing unit 161, a depth valid region determination unit 162, a triangle patch generation unit 163, and a color information texture clipping unit 164. The decoding device 50a (see Figure 1 ) including the decoding unit 16a acquires the first distance d1 between the projection point Q and the first point P1 and the second distance d2 between the projection point Q and the second point P2 based on the information transmitted from the encoding device 40a, that is, the 3D shape of the surface of the subject 90, and associates the position of the projection point Q, the first distance d1, and the second distance d2 to generate the 3D model 90M of the subject 90, a straight line extending from the projection point Q to the subject 90 contacts the subject 90 at the first point P1, and a straight line extending from the first point P1 into the inside of the subject 90 penetrates to the outside of the subject 90 from the second point P2.

[0134] The image decompression processing unit 161 decompresses a bitstream compressed by the image compression processing unit 135 into an image. Note that encoding and decoding are paired, and thus, for example, an AVC encoded bitstream is decoded by an AVC decoding process.

[0135] The depth effective region determination unit 162 identifies a region of the subject 90 based on a size relationship between the first distance d1 stored in the first layer depth map Dm1 acquired by the first layer depth map generation unit 131 and the second distance d2 stored in the second layer depth map Dm2 acquired by the second layer depth map generation unit 132. More specifically, the depth effective region determination unit 162 determines a region onto which the subject 90 is projected as an effective region, and determines a region onto which the subject 90 is not projected as an ineffective region. In the region determined as the effective region, the stored depth values D1 and D2 are used for reconstructing the shape of the subject 90. Meanwhile, data of the region determined as the ineffective region is discarded without being used for rendering. Note that the depth effective region determination unit 162 is an example of a region determination unit in the present disclosure. Further, details of the processing performed by the depth effective region determination unit 162 will be described later.

[0136] The triangle patch generation unit 163 generates shape data, so-called mesh data, which can be used for a rendering process, on a boundary between the inside of the subject 90 and the outside of the subject 90. Further, the triangle patch generation unit 163 forms a closed space by connecting a front surface side end portion of the subject 90 obtained based on the first distance d1 stored in the first layer depth map Dm1 with respect to the projection point Q and a back surface side end portion of the subject 90 obtained based on the second distance d2 stored in the second layer depth map Dm2 with respect to the projection point Q. Note that the triangle patch generation unit 163 is an example of a closed space generation unit in the present disclosure. Details of the processing performed by the triangle patch generation unit 163 will be described later.

[0137] Based on information about the image decoded by the image decompression processing unit 161 and the image arrangement information generated by the tiling unit 134a, the color information texture clipping unit 164 generates color information texture data C to be mapped to the three-dimensional mesh data in the subsequent rendering unit 17a.

[0138] [1-6. Depth effectiveness determination process]

[0139] Next, the depth effective region determination processing performed by the depth effective region determination unit 162 will be described with reference to Figure 11 and Figure 12 Figure 11 is a diagram showing a method of reading a depth value from each depth map by the depth effective region determination unit. Figure 12 ​is a diagram showing a method of determining the validity / invalidity of a depth value by the depth valid region determination unit.

[0140] Figure 11 The first layer depth map Dm1 and the second layer depth map Dm2 shown on the left side are obtained by enlarging the portions (the portions of the shoulder of the subject) of the first layer depth map Dm1 and the second layer depth map Dm2 shown on the right side. In Figure 11 , the white circles represent points inside the subject, i.e., points for which valid depth values D1 or D2 are stored. Further, the black circles represent points outside the subject, i.e., points for which invalid depth values D1 or D2 are stored. In other words, Figure 11 The first layer depth map Dm1 and the second layer depth map Dm2 shown on the left side should actually be represented by curved surfaces having points storing depth values, but are drawn on a plane for ease of understanding.

[0141] The reading points of the first layer depth map Dm1 and the second layer depth map Dm2 are arranged in a quadrangular grid as Figure 11 indicated. The grid interval of the grid can be freely set. Further, the position of each reading point of the depth map does not necessarily need to match each pixel position of each depth map. When the reading points and the pixel positions do not match, a depth value interpolated by an interpolation filter is used.

[0142] The triangle patch generation unit 163 described later performs a drawing process in units of a triangle connecting three depth reading points, and thus the smaller the grid interval of the reading points, the greater the amount of the drawing process. On the other hand, when the grid interval of the reading points is set to be large, the load of the drawing process is reduced, and this makes it possible to adjust the processing load of the decoding device 50a.

[0143] Among the depth values D1 and D2 read from the depth reading points, the depth value D1 of the front surface of the subject 90 is stored in the first layer in the valid region, i.e., the region in which the subject 90 is located. The depth value D2 of the back surface of the subject 90 is stored in the second layer. Meanwhile, in the invalid region, i.e., the region in which the subject 90 is not present, the far value is stored in the first layer, and the near value is stored in the second layer. Thus, as shown in the left drawing of Figure 12 , D1≤D2 is satisfied in the valid region. Note that in the case where the subject 90 is very thin in the depth direction, it is considered that the front surface and the back surface have quantized equal depth values, and thus D1=D2 is also included. Meanwhile, D1>D2 is satisfied in the invalid region.

[0144] The depth values Dl and D2 are compressed in transmission, and therefore, the depth values Dl and D2 are deteriorated due to compression errors. In particular, there is a possibility that the boundary of the subject 90 can have a large gap in the depth values. However, in the present embodiment, the effective area of the subject 90 is determined by the relative comparison between the first layer and the second layer, and the effective areas of the first layer and the second layer completely match each other, as shown in the right drawing of FIG. 6. Figure 12

[0145] Next, the depth effective area determination processing performed by the depth effective area determination unit 162 will be described with reference to Figure 13 Figure 13 is a flowchart showing an example of the processing performed by the depth effective area determination unit.

[0146] The depth effective area determination unit 162 sets a depth reading point (step S21).

[0147] The depth effective area determination unit 162 acquires the depth value Dl of the first layer from the first layer depth map Dml (step S22).

[0148] Subsequently, the depth effective area determination unit 162 acquires the depth value D2 of the second layer from the second layer depth map Dm2 (step S23).

[0149] The depth effective area determination unit 162 determines whether the depth value Dl of the first layer is equal to or smaller than the depth value D2 of the second layer (step S24). When it is determined that the depth value Dl of the first layer is equal to or smaller than the depth value D2 of the second layer (step S24: Yes), the processing proceeds to step S25. On the other hand, when it is determined that the depth value Dl of the first layer is not equal to or smaller than the depth value D2 of the second layer (step S24: No), the processing proceeds to step S26.

[0150] If Yes in step S24, the depth effective area determination unit 162 determines that the depth reading point is included in the area of the subject 90 and sets an effective flag to the depth reading point (step S25). Then, the processing proceeds to step S27.

[0151] On the other hand, if No in step S24, the depth effective area determination unit 162 determines that the depth reading point is not included in the area of the subject 90 and sets an ineffective flag to the depth reading point (step S26). Then, the processing proceeds to step S27.

[0152] The depth effective area determination unit 162 determines whether the depth plane has been completely processed (step S27). When it is determined that the depth plane has been completely processed (step S27: Yes), the depth effective area determination unit 162 completes the processing of the depth effective area determination processing. Figure 13 ​​On the other hand, if it is not determined that the depth plane has been fully processed (step S27: no), the processing returns to step S21, and the above series of processing steps are repeated for different depth reading points.

[0153] Note that the valid flag can be set to all depth reading points first, so that when it is determined to be no in step S24, the valid flag at the corresponding depth reading point can be removed.

[0154] [1-7. Grid Data Generation and Processing]

[0155] Next, we will refer to Figure 14 and Figure 15 Describe the grid data generation process performed by the triangle tile generation unit 163. Figure 14 This is a diagram showing the triangular patches set on each depth map. Figure 15 It is a graph showing the number of valid points for each triangular block.

[0156] The depth values ​​D1 and D2, along with their valid flags, read from the reading points on the square grid, are input from the depth valid region determination unit 162 to the triangle tile generation unit 163. Figure 14 The left figure shows an example of a state where a depth readout point and its valid flag have been detected.

[0157] like Figure 14 As shown in the right figure, the triangle tile generation unit 163 generates a graphic structure in which two right-angled triangles are assigned to a cell of a square grid. The graphic structure has a triangular shape in which three adjacent depth readout points are connected as vertices. Note that... Figure 14 The right figure shows a layer, and in fact, similar graphical structures are generated for the first layer depth map Dm1 and the second layer depth map Dm2.

[0158] The triangle tile generation unit 163 performs the following processing on units of triangle tiles generated in this manner. Then, the triangle tile generation unit 163 passes the coordinate-transformed triangle groups (mesh data) based on the triangle tiles to the subsequent rendering unit 17a.

[0159] like Figure 15 As shown, the triangle tile generation unit 163 counts the number N of valid points for each triangle tile. Specifically, the number N of valid points is a value indicating how many of the three vertices of the triangle tile of interest belong to the interior of the subject 90. The number N of valid points is an evaluation metric indicating whether the triangle tile of interest belongs to the interior of the subject 90.

[0160] In other words, the number N of valid points takes any value among 0, 1, 2, and 3. N = 0 indicates that the triangular patch is entirely outside the subject 90. On the other hand, N = 3 indicates that the triangular patch is entirely inside the subject 90. N = 1 and N = 2 indicate that the triangular patch extends inside and outside the subject 90.

[0161] As shown in the right drawing of FIG. 17, when N = 0 and N = 1, the triangular patch generation unit 163 deletes the triangular patch. Then, the graph structure connecting the three vertices of the corresponding triangle is deleted. Figure 15

[0162] On the other hand, when N = 2 and N = 3, the triangular patch generation unit 163 does not delete the triangular patch but leaves the triangular patch. Then, when N = 3, the inverse transformation (inverse perspective transformation) of Equation (1) is performed for each of the three vertices of the triangular patch to obtain (X, Y, Z) from the coordinate position on the projection plane R(u, v) and the depth s stored at the coordinate position. When N = 2, the above-described inverse perspective transformation is performed on two valid vertices of the triangular patch. Then, the vertex coordinate change processing described later is performed on one invalid vertex. Note that, except for N = 2, the first layer and the second layer can be processed independently.

[0163] Next, the vertex coordinate change processing performed by the triangular patch generation unit 163 when N = 2 will be described with reference to Figure 16 Figure 16 is a drawing illustrating the vertex coordinate change processing.

[0164] Figure 16 The upper left drawing of FIG. 18 illustrates a state in which the first layer depth map Dm1 and the second layer depth map Dm2 are arranged close to each other. Figure 16 The upper right drawing of FIG. 18 illustrates a state in which only a group of triangular patches satisfying N = 2 is extracted. As illustrated in the upper right drawing of FIG. 18, when the triangular patch satisfying N = 2 is generated in one depth map, the triangular patch satisfying N = 2 is also generated at the same position in the opposite depth map. Figure 16 As illustrated in the upper left drawing of FIG. 19, the vertices of the triangular patch of the first layer depth map Dm1 are denoted by Al, Bl, and Cl. Then, the vertices of the triangular patch of the second layer depth map Dm2 are denoted by A2, B2, and C2.

[0165] Figure 16 As illustrated in the upper left drawing of FIG. 19, the vertices of the triangular patch of the first layer depth map Dm1 are denoted by Al, Bl, and Cl. Then, the vertices of the triangular patch of the second layer depth map Dm2 are denoted by A2, B2, and C2.

[0166] The triangular patch generation unit 163 performs the vertex change processing on the invalid vertices Al and A2 belonging to the outside of the subject 90.

[0167] In other words, as illustrated in the upper left drawing of FIG. 19, the vertices Al and A2 of the triangular patch of the first layer depth map Dm1 are changed to Al' and A2' respectively. Figure 16 ​​​As shown in the lower left figure, the triangle tile generation unit 163 replaces the invalid vertex A1 with the value (depth value D2) of vertex B2 of the valid relative depth map. In other words, vertex A1 is considered to be located at vertex B2. Furthermore, the triangle tile generation unit 163 replaces the invalid vertex A2 with the value (depth value D1) of vertex C1 of the valid relative depth map. In other words, vertex A2 is considered to be located at vertex C1. In this way, alternating the replacement of invalid vertices with the values ​​(depth values) of valid vertices in the relative depth map makes the square B1-C1-C2-B2 connected by the two triangle tiles connected. Therefore, the previously separate edges B1-C1 and B2-C2 are connected to each other without gaps when facing each other. In other words, the boundary between the front and back surfaces of the subject 90 viewed from the projection point is seamlessly connected, forming a closed cube (closed space). Note that a similar effect can be obtained by replacing vertex A1 with the depth value D2 of vertex C2 and replacing vertex A2 with the depth value D1 of vertex B1.

[0168] The above processing is performed on all triangular blocks satisfying N=2 on the depth plane, and the boundary between the front and back surfaces of the subject 90 is seamlessly connected, such as... Figure 16 As shown in the lower right image.

[0169] Note that in Figure 16 In the description, for ease of understanding, the depth values ​​D1 of the first layer and D2 of the second layer are described as being essentially equal. However, in practice, the three-dimensional coordinates of each vertex are calculated by inverse transformation of the depth values ​​D1 of the first layer and D2 of the second layer using formula (1). The vertices are arranged in three-dimensional space, thereby reproducing the closed solid shape of the subject 90, which is held between the depth planes of the two layers.

[0170] Figure 17 This is a schematic diagram illustrating how to generate 3D mesh data about a subject. For example... Figure 17 As shown, three-dimensional mesh data is obtained by seamlessly connecting the region of the subject 90 facing the projection point Q and the region of the subject 90 on the back surface side relative to the projection point Q.

[0171] Next, we will refer to Figure 18 Describes the mesh data generation process performed by the triangle tile generation unit 163. Figure 18 This is a flowchart illustrating an example of mesh data generation processing performed by the triangle tile generation unit.

[0172] Triangle tile generation unit 163 is for example Figure 14 As shown in the right figure, each generated triangular patch counts the number of valid points N (step S31).

[0173] The triangle patch generation unit 163 determines whether the number N of valid points counted in step S31 is 2 or more (step S32). When it is determined that the number N of valid points is 2 or more, i.e., the number N of valid points is 2 or 3 (step S32: YES), the processing proceeds to step S33. On the other hand, when it is determined that the number N of valid points is not 2 or more (step S32: NO), i.e., when it is determined that the number N of valid points is 0 or 1, the processing proceeds to step S34.

[0174] If YES in step S32, the triangle patch generation unit 163 calculates the three-dimensional coordinates of the vertices determined to be valid (step S33). Then, the processing proceeds to step S35.

[0175] On the other hand, if NO in step S32, the triangle patch generation unit 163 does not generate a triangular mesh for the triangle patch of interest (step S34). Then, the triangle patch of interest is erased so that rendering is not performed thereon. Then, the processing proceeds to step S38.

[0176] After step S33, the triangle patch generation unit 163 determines whether the number N of valid points for the triangle patch of interest is 3 (step S35). When it is determined that the number N of valid points is 3 (step S35: YES), the processing proceeds to step S37. On the other hand, when it is determined that the number N of valid points is not 3 (step S35: NO), i.e., when it is determined that the number N of valid points is 2, the processing proceeds to step S36.

[0177] If NO in step S35, the triangle patch generation unit 163 performs the above-described vertex coordinate change processing (see Figure 16 ) (step S36).

[0178] After step S36 or if YES in step S35, the triangle patch generation unit 163 generates a triangular mesh for the triangle patch of interest (step S37).

[0179] The triangle patch generation unit 163 determines whether all the triangle patches have been processed (step S38). When it is determined that all the triangle patches have been processed (step S38: YES), the triangle patch generation unit 163 completes the processing. On the other hand, when it is determined that all the triangle patches have not been processed (step S38: NO), the processing returns to step S31, and the above-described series of processing steps are repeated for a different triangle patch. Figure 18

[0180] [1-8. Detailed Configuration of the Rendering Unit]

[0181] Next, the detailed configuration of the rendering unit 17a will be described with reference to Figure 19 .​Figure 19 is a functional block diagram showing an example of a functional configuration of the rendering unit.

[0182] As Figure 19 shown, the rendering unit 17a includes a depth map generation unit 171, a visibility confirmation unit 172, and a drawing unit 173.

[0183] The depth map generation unit 171 generates a depth map of the three-dimensional mesh data of the subject 90 generated by the triangle patch generation unit 163, which is viewed from the same projection point Q as the projection point Q at which the color information texture data C is viewed. In other words, the depth map generation unit 171 generates a plurality of depth maps each corresponding to the plurality of color information texture data C.

[0184] The visibility confirmation unit 172 compares each of the depth maps generated by the depth map generation unit 171 with the three-dimensional coordinates of the three-dimensional mesh data to perform a visibility confirmation that determines whether the three-dimensional mesh data of interest is visible from the projection point Q. This determination determines mesh cells that are not visible from the projection point Q due to overlapping of a plurality of objects. Note that the visibility confirmation will be described in detail later (see Figure 22 ).

[0185] The drawing unit 173 draws the three-dimensional shape of the mesh determined to be visible (have visibility) by the visibility confirmation unit 172. Furthermore, the drawing unit 173 maps the corresponding color information texture data C to the drawn mesh. This configuration makes it possible to generate an image of the subject 90 observed from any projection point Q.

[0186] [1-9. Modification of the First Embodiment]

[0187] Note that the first embodiment can be implemented in various modified forms. For example, in the tiled image T1 (see Figure 9 ), the first layer depth map Dm1, the second layer depth map Dm2, and the color information texture data C do not necessarily need to be in one image. In other words, the first layer depth map Dm1 and the second layer depth map Dm2 can be collected in one image, and the color information texture data C can be in another image. Furthermore, the first layer depth map Dm1, the second layer depth map Dm2, and the color information texture data C can each be in one image (in this configuration, the tiling does not need to be performed). Furthermore, all of the color information texture data C can be in separate images (in this configuration, neither the packing process nor the tiling process needs to be performed). Note that the plurality of images to be transmitted by the transmission unit 14 needs to increase the number of image channels of the bitstream, but synchronization between the channels can be achieved using a mechanism of general multi-channel streaming.

[0188] Furthermore, the transmission unit 14 can simultaneously transmit the tiled image T1 and the voice information.

[0189] Further, the tiled image T1 can be still image data or moving image data. In a case where the tiled image T1 is still image data, the image compression processing unit 135 performs still image compression processing such as JPEG. Further, in a case where the tiled image T1 is moving image data, the image compression processing unit 135 performs moving image compression processing such as MPEG2, AVC, HEVC, VVC, or AV1.

[0190] Note that the tiled image T1 to be transmitted does not necessarily need to be compressed or decompressed. In this configuration, the image compression processing unit 135 (see Figure 5 ) and the image decompression processing unit 161 (see Figure 10 ) are unnecessary.

[0191] Further, the depth values D1 stored in the first layer depth map Dm1 and the depth values D2 stored in the second layer depth map Dm2 can have an arbitrary bit length. In other words, in addition to 8 bits described in the embodiments, 10 bits, 12 bits, or the like can be employed. Note that the longer the bit length of the depth values D1 and D2, the more finely the depth of the subject 90 can be expressed.

[0192] Further, the tiled image T1 can be accumulated as accumulated data in a storage unit instead of being transmitted via the transmission unit 14 and the reception unit 15, so that the accumulated data can be read from the storage unit and reproduced.

[0193] Further, the data acquisition unit 11 to the transmission unit 14 do not necessarily need to perform real-time processing operation. In other words, the tiled image T1 to be transmitted can be temporarily accumulated in a storage unit or the like, and the accumulated data can be read and transmitted to the decoding apparatus 50a.

[0194] Note that the real-time processing operation of the data acquisition unit 11 to the transmission unit 14 can be used for real-time communication, for example, live streaming.

[0195] [1-10. Effects of the First Embodiment]

[0196] As described above, in the encoding apparatus 40 (information processing apparatus) of the first embodiment, the first-layer depth map generation unit 131 (distance acquisition unit) acquires the first distance dl between the projection point Q and the first point Pl based on the 3D shape of the surface of the subject 90, and a straight line extending from the projection point Q contacts the subject 90 at the first point Pl. Further, the second-layer depth map generation unit 132 (distance acquisition unit) acquires the second distance d2 between the projection point Q and the second point P2 based on the 3D shape of the surface of the subject 90, and a straight line starting from the projection point Q and extending into the inside of the subject 90 from the first point Pl penetrates to the outside of the subject 90 at the second point P2.

[0197] This configuration makes it possible to acquire the shapes of the front surface and the back surface of the subject 90 at the same time when the subject 90 is observed from the same position.

[0198] Further, according to the encoding apparatus 40 (information processing apparatus) of the first embodiment, when the first point Pl does not exist, the first-layer depth map generation unit 131 (distance acquisition unit) acquires the distance far (first distance value) as the first distance dl. Further, when the second point P2 does not exist, the second-layer depth map generation unit 132 (distance acquisition unit) acquires the distance near (second distance value) smaller than the distance far as the second distance d2.

[0199] This configuration makes it possible to easily identify the region of the subject 90 and the region in which the subject 90 does not exist.

[0200] Further, according to the encoding apparatus 40 (information processing apparatus) of the first embodiment, the first-layer depth map generation unit 131 (distance acquisition unit) and the second-layer depth map generation unit 132 (distance acquisition unit) set the distance far (first distance value) and the distance near (second distance value) to equal values.

[0201] Therefore, when the first-layer depth map Dml generated by the first-layer depth map generation unit 131 and the second-layer depth map Dm2 generated by the second-layer depth map generation unit 132 are arranged side by side, there is no gap in the depth values at the boundary between the depth maps. Therefore, when the first-layer depth map Dml and the second-layer depth map Dm2 are arranged in one image and encoded, it is possible to improve the compression efficiency of the image.

[0202] Further, in the encoding apparatus 40 (information processing apparatus) of the first embodiment, the transmission unit 14 transmits the first distance dl acquired by the first-layer depth map generation unit 131 and the second distance d2 acquired by the second-layer depth map generation unit 131 to the decoding apparatus 50a that reconstructs the three-dimensional shape of the subject 90 in association with each other.

[0203] This configuration makes it possible to transmit the coordinates of the front surface side and the back surface side of the subject in association with each other. Therefore, it is possible to improve the transmission efficiency of information.

[0204] Further, the decoding device 50a (information processing device) of the first embodiment acquires the first distance d1 acquired by the first layer depth map generation unit 131 and the second distance d2 acquired by the second layer depth map generation unit 132, and identifies the region of the subject 90 based on the size relationship between the first distance d1 and the second distance d2.

[0205] This configuration makes it possible to identify the position range of the subject 90 by only a simple comparison calculation.

[0206] Further, in the decoding device 50a (information processing device) of the first embodiment, the triangular patch generation unit 163 (closed space generation unit) connects the front surface side end of the subject 90 viewed from the projection point Q and the back surface side end of the subject 90 viewed from the projection point Q. The front surface side end of the subject 90 is obtained based on the plurality of first distances d1 relating to the projection point Q, and the back surface side end of the subject 90 is obtained based on the plurality of second distances d2 relating to the projection point Q.

[0207] Therefore, the front surface and the back surface of the subject 90 are connected, thereby generating the 3D model 90M of the subject 90 without a gap.

[0208] Further, in the decoding device 50a (information processing device) of the first embodiment, in a case where two of the first distances d1 of the adjacent three points (the vertices A1, B1, and C1) of the triangle, the vertices B1 and C1, represent the front surface of the subject 90 and the first distance d1 of the remaining one point, the vertex A1, does not represent the front surface of the subject 90, the triangular patch generation unit 163 (closed space generation unit) replaces the remaining one point, the vertex A1, with any one of the two points, the vertices B2 and C2, of the second distances d2 corresponding to the three points, the vertices A1, B1, and C1, which represent the back surface of the subject 90, and replaces one point, the vertex A2, of the second distances d2 corresponding to the three points, the vertices A1, B1, and C1, which does not represent the back surface of the subject 90, with one point, either the vertex B1 or the vertex C1, of the two points representing the front surface of the subject 90 opposite to the point replacing the remaining one point, the vertex A1.

[0209] Therefore, it is possible to connect the front surface and the back surface of the subject 90 without a gap by a simple arithmetic process.

[0210] (2. Second Embodiment)

[0211] [2-1. Configuration of Tiled Image]

[0212] The information processing system 10b (not illustrated) of the second embodiment includes an encoding device 40b (not illustrated) in place of the encoding device 40a of the information processing system 10b (see Figure 1 ) already described in the first embodiment. Further, a decoding device 50b (not illustrated) is provided in place of the decoding device 50a.

[0213] The encoding device 40b includes an encoding unit 13b (not illustrated) and a transmission unit 14. Further, the decoding device 50b includes a reception unit 15 and a decoding unit 16b (see Figure 20 ). Further, the decoding device 50b includes a rendering unit 17b (see Figure 20 ) in place of the rendering unit 17a.

[0214] Next, the function of the encoding unit 13b of the information processing system 10b will be described. The encoding unit 13b generates, for each piece of color information texture data C, a plurality of first layer depth maps D acquired from the same projection point Q as the projection point from which the respective piece of color information texture data C is acquired. Note that the encoding unit 13b includes a tiling unit 134b (not illustrated) in place of the tiling unit 134a in the encoding unit 13a illustrated in Figure 5 .

[0215] Figure 20 is a diagram illustrating an example of an image to be encoded generated by the tiling unit in the second embodiment.

[0216] Figure 20 The tiled image T2 illustrated in Figure 9 differs from the tiled image T1 (see Figure 20 ) described in the first embodiment in that the tiled image T2 includes a plurality of first layer depth maps D corresponding one-to-one to the plurality of pieces of color information texture data C. In other words, the tiling unit 134b generates a tiled image T2 in which the plurality of pieces of color information texture data C regarding the subject 90 viewed from the plurality of projection points Q and the first layer depth maps D viewed from the same plurality of projection points as the projection points from which the plurality of pieces of color information texture data C are acquired are associated with each other. Note that the first layer depth map Dm1 described in the first embodiment is included in the plurality of first layer depth maps D arranged in the tiled image T2 illustrated in Figure 20 . The tiled image T2 also includes the second layer depth map Dm2 described in the first embodiment. The tiling unit 134b is an example of a color / distance acquisition unit in the present disclosure.

[0217] When the decoding unit 16b reconstructs the three-dimensional shape of the subject 90, the plurality of first layer depth maps D included in the tiled image T2 are used for visibility confirmation (see Figure 22 ) to be described later.

[0218] Next, the functions of the decoding unit 16b and the rendering unit 17b of the information processing system 10b will be described with reference to Figure 21 Figure 21 is a functional block diagram showing an example of the functional configuration of the decoding unit and the rendering unit of the second embodiment.

[0219] The decoding unit 16b has a configuration including a depth map cropping unit 165 in addition to the decoding unit 16a (see Figure 10 ) described in the first embodiment. The depth map cropping unit 165 crops the region of the subject 90 from the plurality of pieces of color information texture data C on the basis of the first layer depth maps D corresponding to the respective pieces of color information texture data C. More specifically, when a valid depth value is stored in any of the first layer depth maps D, the color information texture data C corresponding to the coordinates at which the depth value is stored is determined as the region of the subject 90 and is cropped out.

[0220] The rendering unit 17b has a configuration not including the depth map generation unit 171 in comparison with the configuration of the rendering unit 17a (see Figure 19 ) described in the first embodiment.

[0221] Although in the information processing system 10a, the generation of the depth map is performed by the rendering unit 17a in the information processing system 10b for visibility confirmation, the generation of the depth map is performed by the decoding unit 16b. Further, the decoding unit 16b does not actually create the depth map, but only performs acquisition of the plurality of first layer depth maps D generated by the encoding unit 13b. Then, the depth map cropping unit 165 of the decoding unit 16b crops the color information texture data C corresponding to the region of the subject 90 using the first layer depth maps D.

[0222] Accordingly, it is possible to reduce the processing having a high computational load, that is, the generation of the depth map.

[0223] [2-2. Visibility confirmation]

[0224] Next, the visibility confirmation will be described with reference to Figure 22 Figure 22 is a diagram showing the visibility confirmation.

[0225] ​​In the case where an image of the subject 90 viewed from a plurality of different projection points Q is generated, it is necessary to determine whether or not a point on the front surface of the subject 90 is visible from each projection point Q. In other words, the color information texture data C recorded on the projection plane R(u, v) is the surface closest to the projection point Q, and the surface on the back side that is occluded by this surface is not visible from the projection point Q.

[0226] Specifically, in the case where the color information texture data C is recorded on the projection plane R(u, v) from the projection point Qa, the point Pa (an example of a first point) on the subject 90 is visualized from the projection point Qa, and the depth value (a first distance) of the point Pa is recorded on the projection plane Ra. At the same time, when the point Pa is observed from a different projection point Qb, the point Pa is occluded by the point Pb on the front side and cannot be visualized. The depth value (a distance different from the first distance) of the point Pb is stored in the projection plane Rb. Figure 22

[0227] Therefore, in the case where the three-dimensional coordinates of the point Pa calculated from the depth value stored in the projection plane Ra do not match the three-dimensional coordinates of the point Pb calculated from the depth value stored in the projection plane Rb, the visibility confirmation unit 172 determines that the point Pa viewed from the projection point Qa is not visible from the projection point Qb different from the projection point Qa. Figure 22

[0228] Then, when an image of the subject 90 viewed from the projection point Qb is drawn, the drawing unit 173 does not use the color information texture data C at the point Pb, but uses the color information texture data C projected to a projection point closer to the projection point Qb and determined to visualize the point Pa. Furthermore, the color information texture data C projected to be closer to the projection point Qb and to visualize the point Pa can be mixed and used.

[0229] [2-3. Effects of the second embodiment]

[0230] As described above, in the encoding device 40b (information processing device) of the second embodiment, the tiling unit 134b (color / distance acquisition unit) acquires the color information texture data C (texture information) about the subject 90 viewed from a plurality of projection points Q and the first distance dl viewed from the same plurality of projection points Q as the projection points from which the color information texture data C is acquired in association with each other.

[0231] This configuration makes it possible to provide one of the first layer depth map Dm1 paired with the second layer depth map Dm2. Therefore, it is possible to reduce the amount of data transmitted by the transmission unit 14. ​​

[0232] Further, in the decoding device 50b (information processing device) of the second embodiment, when the three-dimensional position of the point Pa according to the first distance d1 viewed from the projection point Qa does not match the three-dimensional position of the point Pa according to the first distance d1 corresponding to the direction of the point Pa viewed from the projection point Qb different from the projection point Qa, the visibility confirmation unit 172 determines that the point Pa viewed from the projection point Qa is not visible from the projection point Qb based on the information acquired by the tiling unit 134b (color / distance acquisition unit).

[0233] Therefore, the visibility confirmation can be performed by a simple process using the tiled image T2.

[0234] (3. Third Embodiment)

[0235] [3-1. Function of Third Embodiment]

[0236] In the first and second embodiments, in the case where the subject 90 has a complex shape, it is difficult to continuously convert the 3D shape of the surface of the subject 90 into data due to the occurrence of occlusion. To solve such a problem, the information processing system 10b according to the third embodiment decomposes the subject 90 into a plurality of objects, generates a plurality of first layer depth maps Dm1 and second layer depth maps Dm2 from the same projection point Q with respect to the objects after the decomposition, and thereby continuously converts the surface of the subject 90 into data.

[0237] Note that the configuration (hardware configuration and functional configuration) of the information processing system 10b according to the third embodiment is the same as that of the information processing system 10b (the encoding device 40b and the decoding device 50b) described in the second embodiment, and thus the description thereof is omitted.

[0238] Next, the function of the information processing system according to the third embodiment will be described with reference to Figure 23 to FIG. 13. Figure 23 is a diagram illustrating the function of the information processing system according to the third embodiment.

[0239] In the encoding device 40b included in the information processing system 10b, the 3D model generation unit 12 decomposes the subject 90 into a plurality of objects. Then, the first layer depth map generation unit 131 and the second layer depth map generation unit 132 generate a plurality of first layer depth maps Dm1 and a plurality of second layer depth maps Dm2 viewed from the same projection point Q with respect to the objects after the decomposition. In other words, the first layer depth map generation unit 131 and the second layer depth map generation unit 132 acquire the first distance d1 and the second distance d2 viewed from the same projection point Q with respect to the objects into which the subject 90 is decomposed. As described in the first embodiment, when a straight line extending from the projection point Q as a starting point to the subject 90 passes through the subject 90 a plurality of times, a pair of the first point P1 and the second point P2 is generated each time the straight line passes through the subject 90.

[0240] For example, as shown in FIG. 13, the 3D model generation unit 12 decomposes the subject 90 to be modeled into a first object of an arrow shape and a second object of a rectangular shape. Then, the first layer depth map generation unit 131 and the second layer depth map generation unit 132 generate the first layer depth map Dm1 and the second layer depth map Dm2 of the first object viewed from the projection point Q. Figure 23

[0241] However, since the information processing system 10b needs to reconstruct a video of the subject 90 viewed from the projection point Q (view-dependent rendering), the first layer depth map Dm1 is a depth map of the entire subject 90 viewed from the projection point Q.

[0242] In other words, the first layer depth map generation unit 131 generates the first layer depth map Dm1 storing the depth values indicated by the dotted lines shown in FIG. 14 with respect to the subject 90. Then, the second layer depth map generation unit 132 generates the second layer depth map Dm2 storing the depth values indicated by the thick lines representing the depth values of the back surface side of the first object with respect to the first object of the arrow shape. The first layer depth map Dm1 and the second layer depth map Dm2 are paired and used for the reconstruction of the subject 90. Figure 23

[0243] Next, the first layer depth map generation unit 131 generates the first layer depth map Dm1 of the second object of the rectangular shape viewed from the projection point Q. Note that the first layer depth map Dm1 generated here is referred to as a third layer depth map in order to be distinguished from the first layer depth map Dm1 described above.

[0244] Then, the second layer depth map generation unit 132 generates the second layer depth map Dm2 of the second object viewed from the projection point Q. Note that the second layer depth map Dm2 generated here is referred to as a fourth layer depth map in order to be distinguished from the second layer depth map Dm2 described above. The third layer depth map and the fourth layer depth map generated in this way are paired and used for the reconstruction of the subject 90. ​​

[0245] The rendering unit 17b performs rendering of the object by the above-described method using the pair of depth maps of the object into which the subject 90 is decomposed. Then, the object is combined with the nearest point when the rendered object is observed from the projection point Q, thereby continuously reconstructing the surface of the subject 90.

[0246] Note that, in a case where the subject 90 has a more complex shape, the subject 90 can be decomposed into more objects, so that the same processing as the above-described processing can be repeated to generate more pairs of depth maps.

[0247] [3-2. Effects of the third embodiment]

[0248] As described above, in the encoding apparatus 40b (information processing apparatus) of the third embodiment, the first layer depth map generation unit 131 and the second layer depth map generation unit 132 (distance acquisition unit) acquire the first distance dl and the second distance d2 from the same projection point Q with respect to the object into which the subject 90 is decomposed. Then, the respective objects reconstructed on the basis of the first distance dl and the second distance d2 are combined to reconstruct the image of the subject 90.

[0249] This configuration makes it possible to continuously reconstruct the image of the subject 90 even when the subject 90 has a complex shape.

[0250] (4. Fourth embodiment)

[0251] [4-1. Functions of the fourth embodiment]

[0252] In the first embodiment and the second embodiment, in a case where the subject 90 has a complex shape, it is difficult to continuously convert the 3D shape of the surface of the subject 90 into data due to occurrence of occlusion. To solve such a problem, the information processing system 10b according to the fourth embodiment decomposes the subject 90 into a plurality of objects, generates the first layer depth map Dm1 and the second layer depth map Dm2 from a plurality of projection points with respect to the objects after the decomposition, thereby continuously converting the surface of the subject 90 into data.

[0253] Note that the configuration (hardware configuration and functional configuration) of the information processing system 10b according to the fourth embodiment is the same as that of the information processing system 10b (the encoding apparatus 40b and the decoding apparatus 50b) described in the second embodiment, and thus the description thereof is omitted.

[0254] Next, the functions of the information processing system according to the fourth embodiment will be described with reference to Figure 24 The functions of the information processing system according to the fourth embodiment will be described. Figure 24 is a diagram illustrating the functions of the information processing system according to the fourth embodiment.

[0255] In the encoding device 40b included in the information processing system 10b, the 3D model generation unit 12 decomposes the subject 90 into a plurality of objects. Then, the first layer depth map generation unit 131 and the second layer depth map generation unit 132 generate the first layer depth map Dm1 and the second layer depth map Dm2 viewed from the different projection points Q1, Q2,... for the objects after the decomposition. In other words, for the objects into which the subject 90 is decomposed, the first distance d1 and the second distance d2 viewed from the different projection points Q1, Q2,... are acquired. As described in the first embodiment, when a straight line extending from the projection point Q as a starting point to the subject 90 passes through the subject 90 a plurality of times, a pair of the first point P1 and the second point P2 is generated each time the straight line passes through the subject 90.

[0256] For example, as shown in FIG. 13A, the 3D model generation unit 12 decomposes the subject 90 to be modeled into a first object of an arrow shape and a second object of a rectangular shape. Then, the first layer depth map generation unit 131 and the second layer depth map generation unit 132 generate the first layer depth map Dm1 and the second layer depth map Dm2 of the first object viewed from the projection point Q1. Further, the first layer depth map generation unit 131 and the second layer depth map generation unit 132 generate the first layer depth map Dm1 and the second layer depth map Dm2 of the first object viewed from the projection point Q2. Figure 24

[0257] However, since the information processing system 10b needs to reconstruct a video (view-dependent rendering) of the subject 90 viewed from the projection points Q1 and Q2, the first layer depth map Dm1 is a depth map of the entire subject 90 viewed from the projection points Q1 and Q2.

[0258] In other words, the first layer depth map generation unit 131 generates the first layer depth map Dm1 of the entire subject 90 viewed from the projection point Q1. Thus, the first layer depth map Dm1 storing the depth values indicated by the dotted lines shown in FIG. 13B is generated. Then, the second layer depth map generation unit 132 generates the second layer depth map Dm2 storing the depth values indicated by the thick lines indicating the depth values of the back surface side of the first object of the arrow shape. The first layer depth map Dm1 and the second layer depth map Dm2 are paired and used for the reconstruction of the subject 90. Figure 24

[0259] Next, the first layer depth map generation unit 131 generates the first layer depth map Dm1 of the entire subject 90 viewed from the projection point Q2. Thus, the first layer depth map Dm1 storing the depth values indicated by the dotted lines shown in FIG. 14B is generated. Then, the second layer depth map generation unit 132 generates the second layer depth map Dm2 storing the depth values indicated by the thick lines indicating the depth values of the back surface side of the first object of the arrow shape. The first layer depth map Dm1 and the second layer depth map Dm2 are paired and used for the reconstruction of the subject 90. Figure 24 ​​The first layer depth map Dm1 stores depth values indicated by thin lines representing depth values of the front surface side of the first object in a rectangular shape. The second layer depth map Dm2 stores depth values indicated by thick lines representing depth values of the back surface side of the second object in a rectangular shape. The first layer depth map Dm1 and the second layer depth map Dm2 are paired and used for reconstruction of the subject 90.

[0260] [4-2. Effects of the fourth embodiment]

[0261] As described above, in the encoding device 40b (information processing device) of the fourth embodiment, the first layer depth map generation unit 131 and the second layer depth map generation unit 132 (distance acquisition unit) acquire the first distance d1 and the second distance d2 from different projection points Q1 and Q2 with respect to the objects into which the subject 90 is decomposed. Then, the respective objects reconstructed based on the first distance d1 and the second distance d2 are combined to reconstruct the image of the subject 90.

[0262] This configuration makes it possible to continuously reconstruct the image of the subject 90 even when the subject 90 has a complex shape.

[0263] (5. Application examples of the present disclosure)

[0264] [5-1. Production of content]

[0265] For example, the 3D model 90M of the subject 90 generated by the 3D model generation unit 12 and a 3D model managed by another server can be combined to produce video content. Further, for example, when background data is stored in an imaging device such as Lidar, combining the 3D model 90M of the subject 90 generated by the 3D model generation unit 12 with the background data makes it possible to create content in which the subject 90 seems to be located at a place indicated by the background data.

[0266] [5-2. Experience in virtual space]

[0267] For example, the subject 90 generated by the 3D model generation unit 12 can be arranged in a virtual space that is a space in which a user performs communication using an avatar. In this configuration, the user can use the avatar to live view the subject 90.

[0268] [5-3. Application for communication with remote location]

[0269] For example, the 3D model 90M of the subject 90 generated by the 3D model generation unit 12 is transmitted from the transmission unit 14 to a remote location so that a user at the remote location can view the 3D model 90M of the subject 90 through a reproduction device at the remote location. For example, the real-time transmission of the 3D model 90M of the subject 90 enables the subject 90 and the user at the remote location to communicate with each other in real time. For example, it can be assumed that the subject 90 is a teacher and the user is a student, or the subject 90 is a doctor and the user is a patient, and the like.

[0270] [5-4. Other aspects]

[0271] For example, a free viewpoint video of sports and the like can also be generated based on the 3D models 90M of a plurality of subjects 90 generated by the 3D model generation unit 12. Furthermore, an individual can distribute his / her own self generated by the 3D model generation unit 12 to a distribution platform. As described above, the contents in the embodiments described in this specification are applicable to various technologies and services.

[0272] Although the present disclosure has been described in accordance with several embodiments, these embodiments can be implemented by any device. The device preferably has the necessary functional blocks to obtain the necessary information.

[0273] Furthermore, for example, the individual steps of one flowchart can be executed by one device, or can be shared among and executed by a plurality of devices. Furthermore, when a plurality of processes are included in one step, the plurality of processes can be executed by one device, or can be shared among and executed by a plurality of devices. In other words, the plurality of processes included in one step can also be executed as processes including a plurality of steps. Conversely, processes described as a plurality of steps can be collectively executed as one step.

[0274] Furthermore, for example, the processes of a program executed by a computer can be executed in a time series in accordance with the description in this specification, or the processes can be executed in parallel or individually at necessary timings such as calls. In other words, the processes of the individual steps can be executed in a different order from the above-described order within a consistent range. Furthermore, the processes of the program can be executed in parallel or in combination with the processes of another program.

[0275] Furthermore, for example, a plurality of technologies related to the present technology can be independently implemented within a consistent range. Of course, a plurality of arbitrary present technologies can be implemented in combination with each other. For example, the present technology described in any embodiment can be implemented in combination with the present technology described in another embodiment. Furthermore, any of the above-described present technologies can be implemented in combination with another technology not described above.

[0276] It should be noted that the effects described herein are merely examples and the present application is not limited to these effects and can have other effects. Embodiments of the present disclosure are not limited to the above-described embodiments and various modifications and changes can be made thereto without departing from the spirit and scope of the present disclosure.

[0277] Note that the present disclosure can also have the following configurations.

[0278] (1) An information processing apparatus comprising:

[0279] a distance acquisition unit that acquires a first distance between a projection point and a first point and a second distance between the projection point and a second point based on a 3D shape of a surface of a subject, a straight line extending from the projection point to the subject being in contact with the subject at the first point, a straight line extending from the first point into an interior of the subject penetrating through an exterior of the subject from the second point.

[0280] (2) The information processing apparatus according to (1), wherein

[0281] the distance acquisition unit:

[0282] acquires a predetermined first distance value as the first distance when the first point is not present, and

[0283] acquires a predetermined second distance value smaller than the first distance value as the second distance when the second point is not present.

[0284] (3) The information processing apparatus according to (2),

[0285] the distance acquisition unit sets the first distance value and the second distance value to equal values.

[0286] (4) The information processing apparatus according to any one of (1) to (3), further comprising:

[0287] a transmission unit that transmits the first distance and the second distance acquired by the distance acquisition unit to a decoding apparatus for reconstructing a three-dimensional shape of a subject in association with each other.

[0288] (5) The information processing apparatus according to any one of (1) to (4), further comprising:

[0289] a color / distance acquisition unit that acquires texture information of a subject viewed from a plurality of projection points and a first distance viewed from the same plurality of projection points as the projection points from which the texture information is acquired in association with each other.

[0290] (6) The information processing device according to any one of (1) to (5), in which

[0291] The distance acquisition unit acquires the first distance and the second distance from the same projection point for each object into which the subject is divided.

[0292] (7) The information processing device according to any one of (1) to (5), in which

[0293] The distance acquisition unit acquires the first distance and the second distance from different projection points for each object into which the subject is divided.

[0294] (8) An information processing device comprising:

[0295] a region determination unit that:

[0296] acquires a first distance between a projection point and a first point from which a straight line extending to the subject is in contact with the subject at the first point, or acquires a predetermined first distance value as the first distance when the first point does not exist, based on a 3D shape of a surface of the subject;

[0297] acquires a second distance between the projection point and a second point from which a straight line extending into the inside of the subject penetrates to the outside of the subject from the second point, or acquires a predetermined second distance value smaller than the first distance value as the second distance when the second point does not exist; and

[0298] identifies a region of the subject based on a size relationship between the first distance and the second distance.

[0299] (9) The information processing device according to (8), further comprising:

[0300] a closed space generation unit that connects a front surface side end portion of the subject viewed from the projection point with a back surface side end portion of the subject viewed from the projection point, the front surface side end portion of the subject being based on a plurality of the first distances related to the projection point, the back surface side end portion of the subject being based on a plurality of the second distances related to the projection point.

[0301] (10) The information processing device according to (9), in which

[0302] the closed space generation unit:

[0303] when two of the first distances of the three points of the triangular shape represent the front surface of the subject and the first distance of the remaining one point does not represent the front surface of the subject,

[0304] replacing any one of the two points of the second distances corresponding to the three points, which do not represent the back surface of the subject, with the remaining one point, and

[0305] replacing one of the two points representing the front surface of the subject at the opposite side of the point replaced with the remaining one point, with one of the second distances corresponding to the three points, which does not represent the back surface of the subject.

[0306] (11) The information processing device according to (8), further comprising:

[0307] a color / distance acquisition unit that acquires texture information of the subject viewed from a plurality of projection points and first distances viewed from the same plurality of projection points as the projection points from which the texture information is acquired, in association with each other; and

[0308] a visibility confirmation unit that determines, based on the information acquired by the color / distance acquisition unit, that a first point viewed from a projection point is not visible from a projection point different from the projection point when a three-dimensional position of the first point according to the first distance viewed from the projection point does not match a three-dimensional position according to a first distance corresponding to a direction toward the first point viewed from a projection point different from the projection point.

[0309] (12) A 3D model generation method comprising:

[0310] acquiring a first distance between a projection point and a first point and a second distance between the projection point and a second point based on a 3D shape of a surface of a subject, a straight line extending from the projection point to the subject contacts the subject at the first point, and a straight line extending from the first point into the inside of the subject penetrates to the outside of the subject at the second point, and generating a 3D model of the subject by associating the position of the projection point, the first distance, and the second distance with each other.

[0311] (13) A program for causing a computer to function as:

[0312] a distance acquisition unit acquires a first distance between a projection point and a first point and a second distance between the projection point and a second point, based on a 3D shape of a surface of a subject, a straight line extending from the projection point to the subject contacts the subject at the first point, and a straight line extending from the first point into the inside of the subject penetrates through the outside of the subject at the second point.

[0313] List of reference signs

[0314] 10a, 10b information processing system

[0315] 11 data acquisition unit

[0316] 12 3D model generation unit

[0317] 13a, 13b encoding unit

[0318] 14 transmission unit

[0319] 15 reception unit

[0320] 16a, 16b decoding unit

[0321] 17a, 17b rendering unit

[0322] 18 display unit

[0323] 40a, 40b encoding device (information processing device)

[0324] 50a, 50b decoding device (information processing device)

[0325] 70 (70a, 70b, 70c, 70d, 70e) imaging device

[0326] 90 subject

[0327] 90M 3D model

[0328] 131 first layer depth map generation unit (distance acquisition unit)

[0329] 132 second layer depth map generation unit (distance acquisition unit)

[0330] 133 color information packing unit

[0331] 134a, 134b tiling unit (color / distance acquisition unit)

[0332] 135 image compression processing unit

[0333] 161 image decompression processing unit

[0334] 162 Depth valid region determination unit (region determination unit)

[0335] 163 Triangle patch generation unit (enclosed space generation unit)

[0336] 164 Color information texture clipping unit

[0337] 165 Depth map clipping unit

[0338] 171 Depth map generation unit

[0339] 172 Visibility confirmation unit

[0340] 173 Drawing unit

[0341] C Color information texture data (texture information)

[0342] D1, D2 Quantized depth value (depth value)

[0343] Dm1, D First layer depth map

[0344] Dm2 Second layer depth map

[0345] d1 First distance

[0346] d2 Second distance

[0347] N Number of valid points

[0348] P1 First point

[0349] P2 Second point

[0350] Q, Q1, Q2, Qa, Qb Projected point

[0351] R, Ra, Rb Projected plane

[0352] T1, T2 Tiled image

[0353] far Distance (first distance value)

[0354] near Distance (second distance value)

Claims

1. An information processing device, comprising: The distance acquisition unit acquires distance based on the 3D shape of the subject's surface: The first distance between the projection point and the first point, the straight line extending from the projection point to the subject, contacts the front surface of the subject at the first point, and The second distance between the projection point and the second point is a straight line extending from the first point through the interior of the subject and passing through the second point via the rear surface of the subject to the exterior of the subject. The first distance and the second distance are used to connect the front surface side and the rear surface side of the subject in the generated 3D model of the subject.

2. The information processing device according to claim 1, wherein, The distance acquisition unit: When the first point does not exist, a predetermined first distance value is obtained as the first distance, and When the second point does not exist, a predetermined second distance value less than the first distance value is obtained as the second distance.

3. The information processing device according to claim 1, wherein, The distance acquisition unit: If the first point does not exist, a predetermined first distance value is obtained as the first distance. When the second point does not exist, a predetermined second distance value is obtained as the second distance, and Set the first distance value and the second distance value to be equal.

4. The information processing device according to claim 1, further comprising: A sending unit that transmits the first distance and the second distance acquired by the distance acquisition unit in association with each other to a decoding device for reconstructing the three-dimensional shape of the subject.

5. The information processing device according to claim 1, further comprising: Color / distance acquisition units acquire texture information of a subject viewed from multiple projection points, and a first distance viewed from the same multiple projection points from which the texture information is acquired.

6. The information processing device according to claim 1, wherein, The distance acquisition unit acquires the first distance and the second distance from the same projection point for each object into which the subject is decomposed.

7. The information processing device according to claim 1, wherein, The distance acquisition unit acquires the first distance and the second distance from each projection point that is different from the other for each object into which the subject is decomposed.

8. An information processing device, comprising: Region determination unit, the region determination unit: Based on the 3D shape of the surface of the subject, a first distance between the projection point and the first point is obtained. A straight line extending from the projection point to the subject contacts the front surface of the subject at the first point. Alternatively, if the first point does not exist, a predetermined first distance value is obtained as the first distance. Obtain the second distance between the projection point and the second point. A straight line extending from the first point through the interior of the subject passes through the second point via the rear surface of the subject to the exterior of the subject. Alternatively, if the second point does not exist, obtain a predetermined second distance value that is less than the first distance value as the second distance. as well as The region of the subject is identified based on the magnitude relationship between the first distance and the second distance. The first distance and the second distance are used to connect the front surface side and the rear surface side of the subject in the generated 3D model of the subject.

9. The information processing device according to claim 8, further comprising: An enclosed space generation unit connects the front surface side of the subject viewed from the projection point to the back surface side of the subject viewed from the projection point, wherein the front surface side of the subject is based on a plurality of first distances related to the projection point, and the back surface side of the subject is based on a plurality of second distances related to the projection point.

10. The information processing device according to claim 9, wherein, The enclosed space generation unit: When the first distance of two of the three adjacent points in a triangle shape represents the front surface of the subject, and the first distance of the remaining point does not represent the front surface of the subject... Replace the remaining point with any one of the two points representing the back surface of the subject from the second distance corresponding to the three points, and Replace a point in the second distance corresponding to the three points that does not represent the back surface of the subject with a point on the opposite side of the point that replaces the remaining point.

11. The information processing device according to claim 8, further comprising: Color / distance acquisition units, which acquire texture information of the subject viewed from multiple projection points and a first distance viewed from the same multiple projection points from which the texture information is acquired; as well as A visibility confirmation unit, based on information obtained by the color / distance acquisition unit, determines that the first point viewed from the projection point is invisible from a projection point different from the projection point when the three-dimensional position of the first point according to the first distance viewed from the projection point does not match the three-dimensional position of the first point according to the first distance corresponding to the direction towards the first point viewed from a projection point different from the projection point.

12. A method for generating a 3D model, comprising: Based on the 3D shape of the surface of the subject, a first distance between the projection point and a first point and a second distance between the projection point and a second point are obtained. A straight line extending from the projection point to the subject contacts the front surface of the subject at the first point. A straight line extending from the first point through the interior of the subject passes through the second point via the rear surface of the subject to the exterior of the subject. as well as A 3D model of the subject is generated by correlating the position of the projection point, the first distance, and the second distance with each other. The first distance and the second distance are used to connect the front surface side and the rear surface side of the subject in the generated 3D model of the subject.

13. A computer-readable medium having stored thereon a program that causes a computer to function as: The distance acquisition unit acquires a first distance between a projection point and a first point, and a second distance between the projection point and a second point, based on the 3D shape of the subject's surface. A straight line extending from the projection point to the subject contacts the front surface of the subject at the first point. A straight line extending from the first point through the interior of the subject passes through the second point, via the rear surface of the subject, and extends to the exterior of the subject. in, The first and second distances obtained are used to connect the front and rear surfaces of the subject in the generated 3D model of the subject.

Images