Information processing device, 3D model generation method and program

Abstract

A first layer depth map generation unit (131) (distance acquisition unit) of an encoding device (40a) (information processing device) acquires a first distance (d1) between a projection point (Q) and a first point (P1) where a straight line extending from the projection point (Q) to a subject (90) makes contact with the subject (90), on the basis of the 3D shape of a surface of the subject (90). A second layer depth map generation unit (132) (distance acquisition unit) acquires a second distance (d2) between the projection point (Q) and a second point (P2) where a straight line extending from the first point (P1) and entering the inside of the subject (90) comes out of the subject (90), on the basis of the 3D shape of the surface of the subject (90).

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, a 3D model generating method, and a program. [Background technology]

[0002] Previously, methods have been proposed that use sensed information from real 3D space, such as multi-viewpoint video of a subject captured from different viewpoints, to generate 3D objects within a viewing space, and generate images (volumetric images) that make the objects appear as if they are actually present within the viewing space.

[0003] To generate a volumetric image, coordinate information in a three-dimensional space is necessary. For example, in Patent Document 1, the 3D shape of the subject is found based on a depth map that indicates the distance from the camera to the surface of the subject. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2018 / 074252 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in Patent Document 1, it was only possible to obtain the 3D shape of the surface of the subject facing the camera, i.e., the front side of the subject. Therefore, the 3D shape of the back side of the subject had to be obtained based on a depth map of the subject viewed from the back side.

[0006] This makes it difficult to generate an accurate shape of the subject.

[0007] The present disclosure proposes an information processing device, a 3D model generation method, and a program capable of generating an accurate shape of a subject. [Means for solving the problem]

[0008] In order to solve the above problems, one form of information processing device according to the present disclosure is an information processing device that includes a distance acquisition unit that acquires, based on a 3D shape of the surface of a subject, a first distance between a first point where a straight line from a projection point to the subject contacts the subject and the projection point, and a second distance between a second point where the straight line that enters the subject from the first point penetrates from the subject to the outside and the projection point. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram showing a schematic configuration of an information processing system according to a first embodiment. [Diagram 2] FIG. 1 illustrates an example of an imaging device. [Diagram 3] 11 is a flowchart illustrating an example of a flow of processing performed by the information processing system. [Figure 4] FIG. 2 is a hardware block diagram showing an example of a hardware configuration of an information processing system. [Diagram 5] 4 is a functional block diagram showing an example of a functional configuration of an encoding unit. FIG. [Figure 6] FIG. 11 is a diagram illustrating a method for generating a depth map. [Figure 7] 11 is a diagram illustrating quantized depth values ​​of a first layer depth map and a second layer depth map. FIG. [Figure 8] FIG. 2 is a diagram showing an example of a first layer depth map and a second layer depth map. [Figure 9] FIG. 13 is a diagram illustrating an example of an image generated by a tiling processing unit. [Figure 10] FIG. 2 is a functional block diagram showing an example of a functional configuration of a decoding unit. [Figure 11] 11 is a diagram illustrating a method in which a depth validity area determining unit reads out depth values ​​from a depth map. FIG. [Figure 12] 11 is a diagram illustrating a method in which a depth validity area determining unit determines whether a depth value is valid or invalid. FIG. [Figure 13] 13 is a flowchart showing an example of the flow of processing performed by a depth effective area determining unit. [Figure 14] FIG. 13 is a diagram illustrating triangular patches set on a depth map. [Figure 15] FIG. 13 is a diagram illustrating the number of effective points of a triangular patch. [Figure 16] 11A and 11B are diagrams illustrating a vertex coordinate changing process. [Figure 17] FIG. 2 is a schematic diagram showing how three-dimensional mesh data of a subject is generated. [Figure 18] 10 is a flowchart showing an example of the flow of a mesh data generation process performed by a triangular patch generating unit. [Figure 19] FIG. 2 is a functional block diagram illustrating an example of a functional configuration of a rendering unit. [Figure 20] FIG. 11 is a diagram showing an example of an image to be encoded, which is generated by a tiling processing unit in the second embodiment. [Figure 21] FIG. 11 is a functional block diagram illustrating an example of a functional configuration of a decoding unit and a rendering unit according to the second embodiment. [Figure 22] FIG. 13 is a diagram illustrating a process for checking visibility. [Diagram 23] FIG. 13 is a diagram illustrating the operation of the information processing system according to the third embodiment. [Figure 24] FIG. 13 is a diagram illustrating the operation of the information processing system according to the fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and duplicated descriptions will be omitted.

[0011] The present disclosure will be described in the following order. 1. First embodiment 1-1. Functional configuration of information processing system 1-2. Processing flow of information processing system 1-3.Hardware configuration of information processing system 1-4.Detailed configuration of the encoding unit 1-5.Detailed configuration of the decoding section 1-6. Depth validity determination process 1-7.Mesh data generation process 1-8.Detailed configuration of the rendering section 1-9. Modification of the first embodiment 1-10. Effects of the First Embodiment 2. Second embodiment 2-1. Composition of tiling images 2-2. Visibility check process 2-3. Advantages of the Second Embodiment 3. Third embodiment 3-1. Operation of the Third Embodiment 3-2. Effects of the Third Embodiment 4. Fourth embodiment 4-1. Operation of the Fourth Embodiment 4-2. Effects of the Fourth Embodiment 5. Application Examples of the Disclosure 5-1. Content Creation 5-2. Virtual space experience 5-3. Application to communication with remote locations 5-4.Other

[0012] 1. First Embodiment [1-1. Information processing system configuration] First, an overview of an information processing system 10a to which the present disclosure is applied will be described with reference to Fig. 1. Fig. 1 is a block diagram showing a schematic configuration of an information processing system according to a first embodiment.

[0013] As shown in FIG. 1, the information processing system 10a includes a data acquisition unit 11, a 3D model generation unit 12, an encoding unit 13a, a transmission unit 14, a reception unit 15, a decoding unit 16a, a rendering unit 17a, and a display unit 18.

[0014] The data acquisition unit 11 acquires image data for generating a 3D model 90M of a subject 90, which is an object to be imaged. For example, as shown in FIG. 2, a plurality of viewpoint images captured by a plurality of imaging devices 70 (70a, 70b, ...) arranged to surround the subject 90 are acquired as image data. In this case, it is preferable that the plurality of viewpoint images are images captured synchronously by a plurality of cameras. The data acquisition unit 11 may acquire the image data, for example, by moving one camera and capturing the subject from a plurality of viewpoints. The data acquisition unit 11 may perform calibration based on the image data to acquire internal parameters and external parameters of each imaging device. The data acquisition unit 11 may acquire a plurality of pieces of depth information indicating the distance from a plurality of viewpoints to the subject 90, for example.

[0015] The 3D model generating unit 12 generates a model having three-dimensional information of the subject based on image data for generating a 3D model 90M of the subject 90. The 3D model generating unit 12 generates a 3D model of the subject by, for example, using a so-called Visual Hull to carve out a three-dimensional shape of the subject using images from multiple viewpoints (for example, silhouette images from multiple viewpoints). In this case, the 3D model generating unit 12 can further highly accurately deform the 3D model 90M generated using the Visual Hull using multiple pieces of depth information indicating the distances from multiple viewpoints to the subject. The 3D model generating unit 12 may also generate the 3D model 90M of the subject 90 from one captured image of the subject 90.

[0016] The 3D model 90M generated by the 3D model generating unit 12 can be said to be a video of a 3D model by generating it in units of frames in a time series. In addition, the 3D model 90M can be said to be a live-action 3D model because it is generated using images captured by an imaging device. The 3D model can be expressed in the form of three-dimensional shape mesh data that expresses shape information representing the surface shape of the subject 90, for example, called a polygon mesh, and expresses the connections between vertices. The three-dimensional shape mesh data has, for example, three-dimensional coordinates of the vertices of the mesh and index information indicating which vertices are combined to form a triangular mesh. Note that the method of expressing the 3D model is not limited to these, and it may be described by a so-called point cloud expression method that expresses the position information of points. Color information data is also generated as a texture in a form linked to these 3D shape data. There are a case of a View Independent texture that has a constant color from any direction, and a case of a View Dependent texture that changes color depending on the viewing direction.

[0017] The encoding unit 13a converts the data of the 3D model 90M generated by the 3D model generating unit 12 into a format suitable for transmission and storage. In this embodiment, the 3D shape data input in a format such as 3D shape mesh data is converted into a depth information image projected from one or more viewpoints, a so-called depth map. The depth information and color information in this 2D image state are compressed and output to the transmitting unit 14. The depth information and color information may be transmitted side by side as one image, or may be transmitted as two separate images. Since both are in the form of 2D image data, they can also be compressed using a 2D compression technology such as AVC (Advanced Video Coding).

[0018] The transmitting unit 14 transmits the transmission data formed by the encoding unit 13a to the receiving unit 15. The transmitting unit 14 performs a series of processes by the data acquiring unit 11, the 3D model generating unit 12, and the encoding unit 13a offline, and then transmits the transmission data to the receiving unit 15. The transmitting unit 14 may also transmit the transmission data generated from the series of processes described above to the receiving unit in real time.

[0019] The 3D model generating unit 12, the encoding unit 13a, and the transmitting unit 14 configure an encoding device 40a. The encoding device 40a is an example of an information processing device in the present disclosure.

[0020] The receiving unit 15 receives the transmission data transmitted from the transmitting unit 14 .

[0021] The decoding unit 16a restores the bit stream received by the receiving unit 15 to a two-dimensional image, and restores mesh and texture information that can be rendered by the rendering unit 17a from the restored two-dimensional image.

[0022] The receiving unit 15, the decoding unit 16a, and the rendering unit 17a configure a decoding device 50a. The decoding device 50a is an example of an information processing device in the present disclosure.

[0023] The rendering unit 17a performs texture mapping by projecting the mesh of the 3D model 90M from the viewpoint of the rendering camera and pasting a texture representing a color or pattern. The feature of this embodiment is that the rendering at this time can be viewed from any freely set viewpoint regardless of the camera position at the time of shooting. Texture mapping includes a method called View Dependent (VD method) that takes into account the user's viewing viewpoint, and a method called View Independent (VI method) that does not take into account the user's viewing viewpoint. The VD method has the advantage of being able to realize higher quality rendering than the VI method because the texture pasted on the 3D model changes depending on the position of the viewing viewpoint. On the other hand, the VI method has the advantage of requiring less processing than the VD method because it does not take into account the position of the viewing viewpoint. The data on the viewing viewpoint is input from the display device to the rendering unit 17a after the display device detects the user's viewing location (Region of Interest).

[0024] The display unit 18 displays the result of rendering by the rendering unit 17a on the display unit of the display device. The display device may be, for example, a head mounted display, a spatial display, a mobile phone, a television, a PC, or the like, and may be a 2D monitor or a 3D monitor.

[0025] The information processing system 10a in FIG. 1 shows a series of flows from the data acquisition unit 11 that acquires captured images, which are materials for generating content, to the display control unit that controls the display device on which the user views. However, this does not mean that all functional blocks are necessary to implement this embodiment, and this embodiment can be implemented for each functional block or a combination of multiple functional blocks. For example, in FIG. 1, the transmission unit 14 and the reception unit 15 are provided to show a series of flows from the side that creates the content to the side that views the content through the distribution of content data, but if the content is created and viewed using the same information processing device (for example, a personal computer), it is not necessary to provide the transmission unit 14 and the reception unit 15.

[0026] When implementing the information processing system 10a, the same implementer may implement everything, or different implementers may implement each functional block. For example, the operator A generates 3D content through the data acquisition unit 11, the 3D model generation unit 12, and the encoding unit 13a. In addition, the 3D content may be distributed through the transmission unit 14 (platform) of the operator B, and the display device of the operator C may receive, render, and control the display of the 3D content. In some cases, the system may be implemented jointly by multiple implementers.

[0027] In addition, each of the above-mentioned functional blocks can be implemented on the cloud. For example, the rendering unit 17a may be implemented in the display device or in a server. In that case, information is exchanged between the display device and the server.

[0028] 1, the data acquisition unit 11, the 3D model generation unit 12, the encoding unit 13a, the transmission unit 14, the reception unit 15, the rendering unit 17a, and the display unit 18 are collectively described as the information processing system 10a. However, the information processing system 10a in this specification is referred to as an information processing system if two or more functional blocks are related, and for example, the data acquisition unit 11, the 3D model generation unit 12, the encoding unit 13a, the transmission unit 14, the reception unit 15, the decoding unit 16a, and the rendering unit 17a can be collectively referred to as the information processing system 10a without including the display unit 18.

[0029] [1-2. Processing flow of information processing system] Next, the flow of processing performed by the information processing system 10a will be described with reference to Fig. 3. Fig. 3 is a flowchart showing an example of the flow of processing performed by the information processing system.

[0030] When the process starts, in step S11, the data acquisition unit 11 acquires image data for generating a 3D model of the subject.

[0031] Next, in step S12, the 3D model generating unit 12 generates a 3D model having three-dimensional information of the subject, based on the image data for generating a 3D model of the subject.

[0032] In step S13, the encoding unit 13a encodes the shape and texture data of the 3D model generated by the 3D model generating unit 12 into a format suitable for transmission and storage.

[0033] Then, in step S14, the transmitting unit 14 transmits the encoded data, and in step S15, the receiving unit 15 receives this transmitted data.

[0034] In step S16, the decoding unit 16a performs a decoding process and converts the 3D model into shape and texture data required for display. Then, in step S17, the rendering unit 17a performs rendering using the shape and texture data. The rendering result is displayed on the display unit in step S18. Then, when the process of step S18 ends, the process of the information processing system 10a ends.

[0035] [1-3. Hardware configuration of information processing system] Next, the hardware configuration of the information processing system 10a will be described with reference to Fig. 4. Fig. 4 is a hardware block diagram showing an example of the hardware configuration of the information processing system.

[0036] 4, a CPU 21, a ROM 22, and a RAM 23 are interconnected via a bus 24. An input / output interface 25 is also connected to the bus 24. An input unit 26, an output unit 27, a storage unit 28, a communication unit 29, and a drive 30 are connected to the input / output interface 25.

[0037] The input unit 26 includes, for example, a keyboard, a mouse, a microphone, a touch panel, an input terminal, etc. The output unit 27 includes, for example, a display, a speaker, an output terminal, etc. The storage unit 28 includes, for example, a hard disk, a RAM disk, a non-volatile memory, etc. The communication unit 29 includes, for example, a network interface, etc. The drive 30 drives a removable medium such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0038] In the computer configured as above, the CPU 21 performs the above-mentioned series of processes by, for example, loading a program stored in the storage unit 28 into the RAM 23 via the input / output interface 25 and the bus 24 and executing the program. The RAM 23 also stores data and the like necessary for the CPU 21 to execute various processes as appropriate.

[0039] The program executed by the computer can be applied by recording it on a removable medium such as a package medium, for example. In this case, the program can be installed in the storage unit 28 via the input / output interface by inserting the removable medium into a drive.

[0040] This program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, digital satellite broadcasting, etc. In this case, the program can be received by the communication unit 29 and installed in the storage unit 28.

[0041] [1-4. Detailed configuration of the encoding section] Next, a detailed configuration of the encoding unit 13a will be described with reference to Fig. 5. Fig. 5 is a functional block diagram showing an example of the functional configuration of the encoding unit.

[0042] The encoding unit 13a receives the three-dimensional shape mesh data of the subject 90 and color information texture data C (see FIG. 9) from the 3D model generation unit 12. Then, the encoding unit 13a converts the data into bit stream data suitable for transmission and storage, and outputs the data. As shown in FIG. 5, the encoding unit 13a includes a first layer depth map generation unit 131, a second layer depth map generation unit 132, a color information packing unit 133, a tiling processing unit 134a, and an image compression processing unit 135.

[0043] The first layer depth map generation unit 131 acquires a first distance d1 between a first point P1 at which a straight line starting from the projection point Q to the subject 90 contacts the subject 90, and the projection point Q, based on the 3D shape acquired by the 3D model generation unit 12. Then, the first layer depth map generation unit 131 generates an image that visualizes the distance to the surface of the subject 90 as viewed from the projection point Q, that is, a first layer depth map Dm1. Note that the first layer depth map generation unit 131 is an example of a distance acquisition unit in the present disclosure.

[0044] The second layer depth map generating unit 132 acquires a second distance d2 between the projection point Q and a second point P2 at which a straight line that enters the subject 90 from the first point P1 and penetrates the subject 90 to the outside, with the projection point Q as the starting point, based on the 3D shape acquired by the 3D model generating unit 12. Then, the second layer depth map generating unit 132 generates an image that visualizes the distance to the back surface of the subject 90 as seen from the projection point Q, that is, a second layer depth map Dm2. Note that the second layer depth map generating unit 132 is an example of a distance acquiring unit in the present disclosure. Note that a straight line that starts from the projection point Q and reaches the subject 90 may penetrate the subject 90 multiple times. In such a case, multiple pairs of the first point P1 and the second point P2 are generated. That is, each time the straight line penetrates the subject 90, a pair of the first point P1 and the second point P2 is generated.

[0045] In this way, since the three-dimensional shape of the subject 90 is restored in the 3D model generation unit 12, when a perspective transformation is performed from an arbitrary viewpoint on the three-dimensional shape of the subject 90, the distance to the surface of the subject 90 can be expressed as an image. This image is the above-mentioned first layer depth map Dm1 and second layer depth map Dm2.

[0046] The perspective transformation is calculated, for example, by equation (1).

[0047]

number

[0048] Equation (1) shows that the two-dimensional coordinates (u, v) of the projection plane and the depth s can be obtained by multiplying a point (X, Y, Z) on the subject 90 by a matrix formed by the camera's external parameters consisting of rotational components r11 to r33 and translational components t1 to t3, and a matrix formed by the camera's internal parameters represented by fx, fy and cx, cy.

[0049] All faces of the three-dimensional mesh of the subject 90 visible from the projection point Q are projected, and the distance from the nearest point, that is, the projection point Q to the closest point, is stored in a two-dimensional array of data to create the first layer depth map Dm1.

[0050] In this embodiment, in addition to the first layer depth map Dm1, a second layer depth map Dm2 is also used, which records the distance to the rear surface of the subject 90. Fig. 6 is a diagram for explaining a method of generating a depth map.

[0051] The first layer depth map generator 131 quantizes the depth s calculated as the first distance d1 within a range between a minimum value and a maximum value, and stores the quantized depth s at a point Ps on the projection plane R(u,v) shown in Fig. 6. The minimum value is, for example, the distance near from the projection point Q to the near plane Sn in Fig. 6, and the maximum value is, for example, the distance far from the projection point Q to the far plane Sf in Fig. 6.

[0052] In this embodiment, the depth is stored in an 8-bit luminance value so that it can be compressed by, for example, AVC (Advanced Video Coding). That is, the first distance d1 is converted into a quantized depth value D1 by the calculation formula shown in Equation (2). Note that D1=0 is stored in an area where the subject 90 does not exist. Note that hereinafter, the quantized depth value D1 is simply referred to as the depth value D1.

[0053]

number

[0054] The second layer depth map generator 132 uses the same perspective projection parameters as the first layer, but stores the depth s calculated as the second distance d2 from the inside of the subject 90 to the point where the subject 90 exits from the back side of the subject 90 at a point Ps on the projection plane R(u,v) shown in FIG. 6. In the area where the subject 90 is projected in the depth map, the second distance d2 is greater than the first distance d1. The quantized depth value D2 of the second layer is calculated by equation (3) in which the distances near and far in equation (2) are swapped so that the quantized depth value D2 of the second layer is a value that is the inverse of the depth value D1 of the first layer. Note that D2=0, which represents an invalid value, is stored in an area where the subject 90 does not exist. This invalid value (D2=0) is the same value as when the second distance d2 is equal to the distance near. Note that round(x) in equation (2) is a function that rounds x to an integer. Moreover, hereinafter, the quantized depth value D2 will be simply referred to as the depth value D2.

[0055]

number

[0056] Although FIG. 6 shows an example in which the depth map is generated by perspective projection with the projection point Q as the projection center, the depth map may be generated by using a projection method other than perspective projection, for example, parallel projection.

[0057] Next, with reference to FIGS. 7 and 8, specific examples of the first-layer depth map Dm1 and the second-layer depth map Dm2 will be described. FIG. 7 is a diagram for explaining the quantized depth values of the first-layer depth map and the second-layer depth map. FIG. 8 is a diagram showing an example of the first-layer depth map and the second-layer depth map.

[0058] As shown in FIG. 7, the encoding unit 13a stores the depth value D1 in the region (invalid region) where the subject 90 to be stored in the first-layer depth map Dm1 does not exist, as the distance far, in the near plane Sn. Also, the encoding unit 13a stores the depth value D2 in the region (invalid region) where the subject 90 to be stored in the second-layer depth map Dm2 does not exist, as the distance near which is smaller than the distance far, in the far plane Sf. Note that the distance far is an example of the first distance value in the present disclosure. Also, the distance near is an example of the second distance value in the present disclosure. Thereby, it becomes possible to determine that the region where d1 < d2 is the region indicating the subject 90.

[0059] Further, as shown in Expressions (2) and (3), the encoding unit 13a reverses the quantization direction between the first layer and the second layer. Thereby, as shown in FIG. 7, the depth values D1 and D2 corresponding to the invalid region after quantization are made equal values, that is, both the depth value D1 and the depth value D2 are set to 0.

[0060] If the quantization directions are the same between the first layer and the second layer, as shown in the upper diagram of FIG. 8, when the first-layer depth map Dm1 and the second-layer depth map Dm2 are arranged side by side in one image, a gap in the depth value occurs at the boundary line. On the other hand, by reversing the quantization directions between the first layer and the second layer, as shown in the lower diagram of FIG. 8, when the first-layer depth map Dm1 and the second-layer depth map Dm2 are arranged side by side in one image, no gap in the depth value occurs at the boundary line. Therefore, when encoding by arranging the first-layer depth map Dm1 and the second-layer depth map Dm2 side by side in one image, the compression efficiency of the image can be increased.

[0061] The quantization method shown here is merely an example, and a quantization method other than the linear quantization shown in formulas (2) and (3) may be applied. For example, the inverse quantization shown in formula (4) may be applied. According to formula (4), the further away the pixel is, the coarser the quantization step is.

[0062]

number

[0063] Returning to Fig. 5, the color information packing unit 133 performs a process of arranging (packing) multiple pieces of color information texture data C input from the 3D model generation unit 12 into one image. This process makes it possible to reduce the number of images to be transmitted to the decoding device 50a. Note that the color information texture data C is an example of texture information in the present disclosure.

[0064] The tiling processing unit 134a performs a process of combining the first layer depth map Dm1, the second layer depth map Dm2, and the color information texture data C packed by the color information packing unit 133 into one image. The values ​​(depth values ​​D1, D2) stored in the first layer depth map Dm1 and the second layer depth map Dm2 are converted into a Y / Cb / Cr signal format (luminance signal and color difference signal) in order to arrange them in the same image as the color information. That is, the depth values ​​D1, D2 are stored in the luminance value. In addition, a constant value is stored in the color difference, for example, a constant value such as 128 representing gray in the case of an 8-bit image per pixel. In addition, the tiling processing unit 134a outputs image arrangement information indicating at which position in the combined image the depth map and the color information texture data C are stored. The image arrangement information includes, for example, an offset value indicating how far the original image has been moved to be arranged in one image, the values ​​of distances near and far used in quantizing the depth map, the position of the projection point Q, and the like.

[0065] Fig. 9 is a diagram showing an example of an image generated by the tiling processing unit. The tiling processing unit 134a combines a first layer depth map Dm1, a second layer depth map Dm2, and color information texture data C into one tiling image T1. As shown in Fig. 9, the color information texture data C is an actual image obtained by capturing an object 90 using a plurality of imaging devices 70 (see Fig. 2).

[0066] The image compression processing unit 135 compresses the image that has been combined into one frame by the tiling processing unit 134a using a general video codec. Specifically, a widely used codec such as AVC or HEVC is used.

[0067] In the above equations (2) and (3), the depth values ​​D1 and D2 are assumed to be 8 bits and quantized to values ​​from 0 to 255. However, since HEVC and the like can also handle 10-bit values, the depth values ​​D1 and D2 may also be quantized to 10 bits (0 to 1023).

[0068] The bit stream generated by the image compression processing unit 135 and the image arrangement information generated by the tiling processing unit 134a are transmitted to the decoding device 50a via the transmission unit 14. That is, the transmission unit 14 transmits the first distance d1 and the second distance d2 in association with each other to the decoding device 50a, which reconstructs the three-dimensional shape of the subject 90.

[0069] [1-5. Detailed configuration of the decoding section] Next, a detailed configuration of the decoding unit 16a will be described with reference to Fig. 10. Fig. 10 is a functional block diagram showing an example of the functional configuration of the decoding unit.

[0070] As shown in Fig. 10, the decoding unit 16a includes an image decompression processing unit 161, a depth effective area determination unit 162, a triangular patch generation unit 163, and a color information texture cutout unit 164. A decoding device 50a (see Fig. 1) including the decoding unit 16a acquires a first distance d1 between a first point P1 where a straight line from the projection point Q to the subject 90 contacts the subject 90 and the projection point Q, and a second distance d2 between a second point P2 where a straight line entering the subject 90 from the first point P1 penetrates the subject 90 to the outside, and the projection point Q, based on information transmitted from the encoding device 40a, i.e., the 3D shape of the surface of the subject 90, and generates a 3D model 90M of the subject 90 by associating the position of the projection point Q with the first distance d1 and the second distance d2.

[0071] The image decompression processing unit 161 decompresses the bit stream compressed by the image compression processing unit 135 into an image. Note that since encoding and decoding are paired, for example, an AVC-encoded bit stream is decoded by AVC decoding processing.

[0072] The depth effective area determination unit 162 specifies the area of ​​the subject 90 based on the magnitude relationship between the first distance d1 stored in the first layer depth map Dm1 acquired by the first layer depth map generation unit 131 and the second distance d2 stored in the second layer depth map Dm2 acquired by the second layer depth map generation unit 132. More specifically, the depth effective area determination unit 162 determines the area where the subject 90 is projected as a valid area, and determines the area where the subject 90 is not projected as an invalid area. For the area determined as a valid area, the stored depth values ​​D1 and D2 are used to restore the shape of the subject 90. On the other hand, data for the area determined as an invalid area is discarded without being used for rendering. Note that the depth effective area determination unit 162 is an example of an area determination unit in the present disclosure. Details of the process performed by the depth effective area determination unit 162 will be described later.

[0073] The triangular patch generating unit 163 generates shape data, so-called mesh data, that can be used for rendering processing, inside the subject 90 and at the boundary between the subject 90 and the outside. The triangular patch generating unit 163 also forms a closed space by connecting an end portion on the front side of the subject 90 based on the first distance d1 stored in the first layer depth map Dm1 related to the projection point Q and an end portion on the rear side of the subject 90 based on the second distance d2 stored in the second layer depth map Dm2 related to the projection point Q. The triangular patch generating unit 163 is an example of a closed space generating unit in the present disclosure. The process performed by the triangular patch generating unit 163 will be described in detail later.

[0074] Based on the image information decoded by the image decompression processing unit 161 and the image arrangement information generated by the tiling processing unit 134a, the color information texture cut-out unit 164 generates color information texture data C to be pasted onto the three-dimensional shape mesh data in the downstream rendering unit 17a.

[0075] [1-6. Depth validity determination process] Next, the depth valid area determination process performed by the depth valid area determination unit 162 will be described with reference to Fig. 11 and Fig. 12. Fig. 11 is a diagram for explaining a method in which the depth valid area determination unit reads out depth values ​​from a depth map. Fig. 12 is a diagram for explaining a method in which the depth valid area determination unit determines whether a depth value is valid or invalid.

[0076] The first layer depth map Dm1 and the second layer depth map Dm2 shown on the left side of Fig. 11 are enlarged views of a portion (the shoulder portion of the subject) of the first layer depth map Dm1 and the second layer depth map Dm2 shown on the right side. In Fig. 11, white circles represent points inside the subject, i.e., points where valid depth values ​​D1 or D2 are stored. Black circles represent points outside the subject, i.e., points where invalid depth values ​​D1 or D2 are stored. That is, the first layer depth map Dm1 and the second layer depth map Dm2 shown on the left side of Fig. 11 should actually be represented as a curved surface with a depth value stored at each point, but are drawn as a flat surface for ease of explanation.

[0077] The readout points of the first layer depth map Dm1 and the second layer depth map Dm2 are arranged in a square lattice pattern as shown in Fig. 11. The grid interval of the lattice may be set freely. The positions of the readout points of the depth map do not necessarily have to coincide with the pixel positions of the depth map. If the readout points and the pixel positions differ, a depth value interpolated by an interpolation filter is used.

[0078] The triangle patch generator 163 (described later) performs rendering processing for each triangle connecting three depth readout points, so the smaller the grid interval of the readout points, the greater the rendering processing load. On the other hand, if the grid interval of the readout points is set to a larger value, the rendering processing load becomes lighter. By utilizing this, it is possible to adjust the processing load of the decoding device 50a.

[0079] Of the depth values ​​D1 and D2 read from the depth readout point, in the valid area, i.e., the area where the subject 90 is present, the depth value D1 of the front surface of the subject 90 is stored in the first layer. And the depth value D2 of the back surface of the subject 90 is stored in the second layer. On the other hand, in the invalid area, i.e., the area where the subject 90 is not present, the far value is stored in the first layer, and the near value is stored in the second layer. Therefore, as shown in the left diagram of FIG. 12, in the valid area, D1≦D2. Note that if the subject 90 is very thin in the depth direction, it is possible that the front surface and the back surface will have the same depth value when quantized, and therefore the case of D1=D2 is also included. On the other hand, in the invalid area, D1>D2.

[0080] The depth values ​​D1 and D2 are compressed during transmission, and therefore may be degraded due to compression errors. In particular, large gaps in depth values ​​may occur at the boundaries of the subject 90. However, in this embodiment, the effective area of ​​the subject 90 is determined by a relative comparison between the first and second layers, so that the effective areas of the first and second layers completely match, as shown in the right diagram of FIG.

[0081] Next, the flow of the depth effective area determination process performed by the depth effective area determination section 162 will be described with reference to Fig. 13. Fig. 13 is a flowchart showing an example of the flow of the process performed by the depth effective area determination section.

[0082] The depth valid area determining unit 162 sets a depth readout point (step S21).

[0083] The depth valid area determining unit 162 acquires the first layer depth value D1 from the first layer depth map Dm1 (step S22).

[0084] Next, the depth valid area determining unit 162 obtains the second layer depth value D2 from the second layer depth map Dm2 (step S23).

[0085] The depth validity area determination unit 162 determines whether the depth value D1 of the first layer is equal to or less than the depth value D2 of the second layer (step S24). If it is determined that the depth value D1 of the first layer is equal to or less than the depth value D2 of the second layer (step S24: Yes), the process proceeds to step S25. On the other hand, if it is not determined that the depth value D1 of the first layer is equal to or less than the depth value D2 of the second layer (step S24: No), the process proceeds to step S26.

[0086] If the result of the determination in step S24 is Yes, the depth valid area determination unit 162 determines that the depth readout point is included in the area of ​​the subject 90, and sets a valid flag for the depth readout point (step S25). Then, the process proceeds to step S27.

[0087] On the other hand, if the result of the determination in step S24 is No, the depth valid area determination unit 162 determines that the depth readout point is not included in the area of ​​the subject 90, and sets an invalid flag for the depth readout point (step S26). Then, the process proceeds to step S27.

[0088] The depth valid area determination unit 162 determines whether the entire depth surface has been processed (step S27). If it is determined that the entire depth surface has been processed (step S27: Yes), the depth valid area determination unit 162 ends the processing in Fig. 13. On the other hand, if it is not determined that the entire depth surface has been processed (step S27: No), the process returns to step S21 and the above-described series of processes are repeated for a different depth readout point.

[0089] Alternatively, valid flags may be set for all depth readout points initially, and if the result of the determination in step S24 is No, the valid flags for the depth readout points may be lowered.

[0090] [1-7. Mesh data generation process] Next, the mesh data generation process performed by the triangular patch generator 163 will be described with reference to Fig. 14 and Fig. 15. Fig. 14 is a diagram for explaining triangular patches set on a depth map. Fig. 15 is a diagram for explaining the number of effective points of a triangular patch.

[0091] The depth values ​​D1 and D2 read from the read points on the square lattice and their validity flags are input to the triangular patch generation unit 163 from the depth valid area determination unit 162. The left diagram in Fig. 14 shows an example of a state in which the depth read points and their validity flags are detected.

[0092] The triangular patch generating unit 163 generates a graph structure in which two right-angled triangles are assigned to one square lattice, as shown in the right diagram of Fig. 14. The graph structure has depth readout points as vertices, and forms a triangle shape in which three adjacent points are connected. Note that the right diagram of Fig. 14 is for one layer, and in reality, similar graph structures are generated for the first layer depth map Dm1 and the second layer depth map Dm2.

[0093] The triangular patch generating unit 163 performs the following processes for each triangular patch thus generated. The triangular patch generating unit 163 then passes the triangles (mesh data) that have been subjected to coordinate conversion processing based on the triangular patches to the downstream rendering unit 17a.

[0094] 15, the triangular patch generating unit 163 counts the number of valid points N for each triangular patch. More specifically, the number of valid points N is a value indicating how many of the three vertices of the triangular patch in question belong to the inside of the subject 90. This number of valid points N is an evaluation measure indicating whether the triangular patch in question belongs to the inside of the subject 90.

[0095] That is, the number of valid points N can take any one of the values ​​0, 1, 2, or 3. N=0 indicates that the triangular patch is completely outside the subject 90. On the other hand, N=3 indicates that the triangular patch is completely inside the subject 90. And N=1 or 2 indicates that the triangular patch straddles both the inside and outside of the subject 90.

[0096] When N=0, 1, the triangular patch generating unit 163 deletes the triangular patch as shown in the right diagram of Fig. 15. Then, it deletes the graph structure connecting the three vertices of the corresponding triangle.

[0097] On the other hand, when N=2 or 3, the triangular patch generating unit 163 does not delete the triangular patch but leaves it. When N=3, the inverse transformation (inverse perspective transformation) of equation (1) is performed to obtain (X, Y, Z) for each of the three vertices of the triangular patch from the coordinate positions on the projection surface R (u, v) and the depth s stored in the corresponding coordinate positions. When N=2, the inverse perspective transformation described above is performed on two valid vertices of the triangular patch. Then, the vertex coordinate change process described later is performed on the one invalid vertex. Note that when N=2 or less, the first and second layers can be processed independently.

[0098] Next, the vertex coordinate changing process performed by the triangular patch generating unit 163 when N = 2 will be described with reference to Fig. 16. Fig. 16 is a diagram for explaining the vertex coordinate changing process.

[0099] The upper left of Fig. 16 shows a state where a first layer depth map Dm1 and a second layer depth map Dm2 are arranged in close proximity to each other. The upper right of Fig. 16 shows a state where only one pair of N=2 triangular patches are extracted. As shown in the upper right of Fig. 16, when a N=2 triangular patch is generated in one depth map, a N=2 triangular patch is also generated in the same position in the opposite depth map.

[0100] 16, the vertices of the triangular patch of the first-layer depth map Dm1 are designated as A1, B1, and C1, and the vertices of the triangular patch of the second-layer depth map Dm2 are designated as A2, B2, and C2.

[0101] The triangular patch generating unit 163 performs a vertex changing process on the invalid vertices A1 and A2 that belong to the outside of the subject 90.

[0102] That is, as shown in the lower left of FIG. 16, the triangular patch generating unit 163 substitutes the value (depth value D2) of the valid vertex B2 of the opposite depth map into the invalid vertex A1. That is, the vertex A1 is considered to be located at the position of the vertex B2. The triangular patch generating unit 163 also substitutes the value (depth value D1) of the valid vertex C1 of the opposite depth map into the invalid vertex A2. That is, the vertex A2 is considered to be located at the position of the vertex C1. In this way, the quadrangle B1-C1-C2-B2 is connected by two triangular patches by alternately substituting the values ​​(depth values) of the valid vertices of the opposite depth map into the invalid vertices. As a result, the side B1-C1 and the side B2-C2, which were separated, are connected by a surface without any gaps. That is, the boundary between the front and back of the subject 90 as seen from the projection point is seamlessly connected, forming a closed solid (closed space). The same effect can be obtained by substituting the depth value D2 of vertex C2 for vertex A1 and the depth value D1 of vertex B1 for vertex A2.

[0103] By performing the above-mentioned process on all N=2 triangular patches on the depth plane, the boundary between the front and back of the subject 90 is seamlessly connected, as shown in the lower right of FIG.

[0104] In Figure 16, for ease of explanation, the depth values ​​D1 of the first layer and the depth values ​​D2 of the second layer are approximately equal. However, in reality, the three-dimensional coordinates of each vertex are calculated from the depth value D1 of the first layer and the depth value D2 of the second layer by inverse transformation of equation (1) and arranged in three-dimensional space, reproducing a closed three-dimensional shape representing subject 90 sandwiched between the depth surfaces of the two layers.

[0105] Fig. 17 is a schematic diagram showing how 3D shape mesh data of a subject is generated. As shown in Fig. 17, 3D shape mesh data is obtained in which the area facing the projection point Q of the subject 90 and the area behind the projection point Q of the subject 90 are seamlessly connected.

[0106] Next, the flow of the mesh data generation process performed by the triangular patch generating unit 163 will be described with reference to Fig. 18. Fig. 18 is a flowchart showing an example of the flow of the mesh data generation process performed by the triangular patch generating unit.

[0107] The triangular patch generating unit 163 counts the number of valid points N for the triangular patch generated as shown in the right diagram of FIG. 14 (step S31).

[0108] The triangular patch generating unit 163 determines whether the number of valid points N counted in step S31 is 2 or more (step S32). If it is determined that the number of valid points N is 2 or more, i.e., that the number of valid points N is 2 or 3 (step S32: Yes), the process proceeds to step S33. On the other hand, if it is determined that the number of valid points N is not 2 or more (step S32: No), i.e., that the number of valid points N is 0 or 1, the process proceeds to step S34.

[0109] If the result of the determination in step S32 is Yes, the triangular patch generating unit 163 calculates the three-dimensional coordinates of the vertices determined to be valid (step S33), and then proceeds to step S35.

[0110] On the other hand, if the result of step S32 is No, the triangular patch generating unit 163 does not generate a triangular mesh for the triangular patch of interest (step S34), and erases the triangular patch of interest so that rendering is not performed. Then, the process proceeds to step S38.

[0111] Following step S33, the triangular patch generation unit 163 determines whether the number of valid points N for the triangular patch of interest is 3 (step S35). If it is determined that the number of valid points N is 3 (step S35: Yes), the process proceeds to step S37. On the other hand, if it is determined that the number of valid points N is not 3 (step S35: No), that is, if it is determined that the number of valid points N is 2, the process proceeds to step S36.

[0112] If the result of the determination in step S35 is No, the triangular patch generating unit 163 performs the above-mentioned vertex coordinate changing process (see FIG. 16) (step S36).

[0113] Following step S36, or if the determination in step S35 is Yes, the triangular patch generating unit 163 generates a triangular mesh for the triangular patch of interest (step S37).

[0114] The triangular patch generating unit 163 determines whether all triangular patches have been processed (step S38). If it is determined that all triangular patches have been processed (step S38: Yes), the triangular patch generating unit 163 ends the process in Fig. 18. On the other hand, if it is determined that all triangular patches have not been processed (step S38: No), the process returns to step S31 and the above series of processes are repeated for a different triangular patch.

[0115] [1-8. Detailed configuration of the rendering section] Next, a detailed configuration of the rendering unit 17a will be described with reference to Fig. 19. Fig. 19 is a functional block diagram showing an example of the functional configuration of the rendering unit.

[0116] As shown in FIG. 19, the rendering unit 17a includes a depth map generation unit 171, a visibility confirmation unit 172, and a drawing unit 173.

[0117] The depth map generation unit 171 generates a depth map in which the three-dimensional shape mesh data of the subject 90 generated by the triangular patch generation unit 163 is viewed from the same projection point Q as the color information texture data C. In other words, the depth map generation unit 171 generates a plurality of depth maps corresponding to the plurality of color information texture data C, respectively.

[0118] The visibility checking unit 172 performs a visibility check to determine whether or not the three-dimensional shape mesh data of interest is visible from the projection point Q by comparing the depth map generated by the depth map generating unit 171 with the three-dimensional coordinates of the three-dimensional shape mesh data. This determination determines that the mesh is not visible from the projection point Q because multiple objects overlap. The visibility check will be described in detail later (see FIG. 22).

[0119] The drawing unit 173 draws the three-dimensional shape of the mesh that the visibility checking unit 172 has determined to be visible (has visibility). The drawing unit 173 also attaches the corresponding color information texture data C to the drawn mesh. This makes it possible to generate an image of the subject 90 observed from an arbitrary projection point Q.

[0120] [1-9. Modification of the first embodiment] The first embodiment can be modified and executed in various forms. For example, in the tiling image T1 (see FIG. 9), the first layer depth map Dm1, the second layer depth map Dm2, and the color information texture data C may not be one image. That is, the first layer depth map Dm1 and the second layer depth map Dm2 may be one image, and the color information texture data C may be another image. Also, the first layer depth map Dm1, the second layer depth map Dm2, and the color information texture data C may each be one image (in this case, there is no need to perform the tiling process). Furthermore, the color information texture data C may all be another image (in this case, there is no need to perform the packing process or the tiling process). When the transmission unit 14 transmits a plurality of images, the number of image channels of the bit stream increases accordingly, but the channels can be synchronized by using a general multi-channel stream distribution mechanism.

[0121] Furthermore, the transmission unit 14 may transmit the tiling image T1 and the audio information simultaneously.

[0122] The tiling image T1 may be still image data or moving image data. If the tiling image T1 is still image data, the image compression processing unit 135 performs compression processing for still images such as JPEG. If the tiling image T1 is moving image data, the image compression processing unit 135 performs compression processing for moving images such as MPEG2, AVC, HEVC, VVC, AV1, etc.

[0123] It is not necessary to compress or expand the tiling image T1 to be transmitted. In this case, the image compression processing unit 135 (see FIG. 5) and the image expansion processing unit 161 (see FIG. 10) are not necessary.

[0124] Furthermore, the bit length of the depth values ​​D1, D2 stored in the first layer depth map Dm1 and the second layer depth map Dm2 does not matter. That is, in addition to the 8 bits described in the embodiment, 10 bits, 12 bits, etc. may be used. Note that the larger the bit length of the depth values ​​D1, D2, the finer the depth of the subject 90 can be expressed.

[0125] Alternatively, instead of transmitting the tiling image T1 via the transmitting unit 14 and the receiving unit 15, the tiling image T1 may be stored in a storage unit as stored data, and the stored data may be read out from the storage unit and played back.

[0126] Furthermore, the data acquisition unit 11 to the transmission unit 14 do not have to operate in real time. That is, the tiling image T1 to be transmitted may be temporarily stored in a storage unit or the like, and the stored data may be read out and transmitted to the decoding device 50a.

[0127] If the data acquisition unit 11 through the transmission unit 14 are operated in real time, the data can be used for real time communication such as live streaming.

[0128] [1-10. Effects of the first embodiment] As described above, in the encoding device 40 (information processing device) of the first embodiment, the first-layer depth map generation unit 131 (distance acquisition unit) acquires, based on the 3D shape of the surface of the subject 90, a first distance d1 between the projection point Q and a first point P1 at which a straight line extending from the projection point Q to the subject 90 comes into contact with the subject 90. In addition, the second-layer depth map generation unit 132 (distance acquisition unit) acquires, based on the 3D shape of the surface of the subject 90, a second distance d2 between the projection point Q and a second point P2 at which a straight line extending from the first point P1 into the subject 90, starting from the projection point Q, penetrates from the subject 90 to the outside.

[0129] This makes it possible to simultaneously obtain the shapes of the front and back of subject 90 when the subject 90 is observed from the same position.

[0130] Furthermore, in the encoding device 40 (information processing device) of the first embodiment, the first-layer depth map generating unit 131 (distance acquiring unit) acquires the distance far (first distance value) as the first distance d1 when the first point P1 does not exist. Furthermore, the second-layer depth map generating unit 132 (distance acquiring unit) acquires the distance near (second distance value) smaller than the distance far as the second distance d2 when the second point P2 does not exist.

[0131] This makes it possible to easily distinguish between the area of ​​the subject 90 and the area where the subject 90 does not exist.

[0132] Furthermore, according to the encoding device 40 (information processing device) of the first embodiment, the first layer depth map generation unit 131 (distance acquisition unit) and the second layer depth map generation unit 132 (distance acquisition unit) set the distance far (first distance value) and the distance near (second distance value) to equal values.

[0133] As a result, when the first layer depth map Dm1 generated by the first layer depth map generation unit 131 and the second layer depth map Dm2 generated by the second layer depth map generation unit 132 are arranged side by side, no gap in depth values ​​occurs at the boundary line. Therefore, when the first layer depth map Dm1 and the second layer depth map Dm2 are arranged side by side in a single image and encoded, it is possible to improve the compression efficiency of the image.

[0134] Furthermore, in the encoding device 40 (information processing device) of the first embodiment, the transmitting unit 14 associates the first distance d1 acquired by the first layer depth map generation unit 131 and the second distance d2 acquired by the second layer depth map generation unit 132 and transmits them to a decoding device 50a, which reconstructs the three-dimensional shape of the subject 90.

[0135] This makes it possible to transmit the coordinates of the front side and the rear side of the subject in association with each other, thereby improving the efficiency of information transmission.

[0136] In addition, the decoding device 50a (information processing device) of the first embodiment acquires a first distance d1 acquired by the first layer depth map generation unit 131 and a second distance d2 acquired by the second layer depth map generation unit 132, and identifies the area of ​​the subject 90 based on the magnitude relationship between the first distance d1 and the second distance d2.

[0137] This makes it possible to identify the range in which subject 90 exists by simply carrying out a comparison operation.

[0138] In addition, in the decoding device 50a (information processing device) of the first embodiment, the triangular patch generation unit 163 (closed space generation unit) connects the front end as seen from the projection point Q of the subject 90 based on a plurality of first distances d1 related to the projection point Q, and the back end as seen from the projection point Q of the subject 90 based on a plurality of second distances d2 related to the projection point Q.

[0139] In this way, by connecting the front and back faces of the subject 90, a 3D model 90M of the subject 90 without any gaps can be generated.

[0140] Furthermore, in the decoding device 50a (information processing device) of the first embodiment, when the first distance d1 between two points (vertices B1 and C1) of the first distances d1 between three points (vertices A1, B1, and C1) adjacent to each other in a triangular shape represents the front surface of the subject 90 and the first distance d1 between the remaining point (vertex A1) does not represent the front surface of the subject 90, the triangular patch generating unit 163 (closed space generating unit) determines the remaining point (vertex A1) as the second distance d2 between the three points (vertices A1, B1, and C1), One of the two points representing the back surface of subject 90 (either vertex B2 or C2) is substituted, and of the second distance d2 corresponding to the three points (vertices A1, B1, C1), one point (vertex A2) that does not represent the back surface of subject 90 is substituted for one of the two points (vertices B1, C1) representing the front surface of subject 90 on the opposite side to the point to which the remaining point (vertex A1) was substituted (vertex C2 if the remaining point is substituted for vertex B2, or vertex B2 if the remaining point is substituted for vertex C2).

[0141] This makes it possible to connect the front and back of subject 90 without any gaps by simple calculation processing.

[0142] 2. Second embodiment [2-1. Composition of tiling images] The information processing system 10b (not shown) of the second embodiment includes an encoding device 40b (not shown) instead of the encoding device 40a of the information processing system 10b (see FIG. 1) described in the first embodiment, and also includes a decoding device 50b (not shown) instead of the decoding device 50a.

[0143] The encoding device 40b includes an encoding unit 13b and a transmitting unit 14, both of which are not shown. The decoding device 50b includes a receiving unit 15 and a decoding unit 16b (see FIG. 20). Furthermore, the decoding device 50b includes a rendering unit 17b (see FIG. 20) instead of the rendering unit 17a.

[0144] Next, a function of the encoding unit 13b of the information processing system 10b will be described. The encoding unit 13b generates a plurality of first layer depth maps D for a plurality of color information texture data C, which are acquired from the same projection point Q as that at which each of the color information texture data C was acquired. Note that the encoding unit 13b includes a tiling processing unit 134b (not shown) instead of the tiling processing unit 134a in the encoding unit 13a shown in FIG. 5.

[0145] FIG. 20 is a diagram showing an example of an image to be encoded, which is generated by the tiling processor in the second embodiment.

[0146] The tiling image T2 shown in FIG. 20 differs from the tiling image T1 (see FIG. 9) described in the first embodiment in that it includes a plurality of first layer depth maps D each corresponding one-to-one to a plurality of color information texture data C. That is, the tiling processing unit 134b generates a tiling image T2 in which the color information texture data C obtained by viewing the subject 90 from a plurality of projection points Q is associated with the first layer depth map D obtained by viewing the subject 90 from the same plurality of projection points Q as those from which the color information texture data C was obtained. Note that the first layer depth map Dm1 described in the first embodiment is included in the plurality of first layer depth maps D arranged in the tiling image T2 as shown in FIG. 20. The tiling image T2 also includes the second layer depth map Dm2 described in the first embodiment. Note that the tiling processing unit 134b is an example of a color-distance acquisition unit in the present disclosure.

[0147] The multiple first layer depth maps D included in the tiling image T2 are used in a visibility confirmation process (see FIG. 22) described later when the decoding unit 16b reconstructs the three-dimensional shape of the subject 90.

[0148] Next, functions of the decoding unit 16b and the rendering unit 17b of the information processing system 10b will be described with reference to Fig. 21. Fig. 21 is a functional block diagram showing an example of the functional configuration of the decoding unit and the rendering unit of the second embodiment.

[0149] The decoding unit 16b has a configuration in which a depth map cutout unit 165 is added to the decoding unit 16a (see FIG. 10) described in the first embodiment. The depth map cutout unit 165 cuts out a region of the subject 90 from among a plurality of pieces of color information texture data C, based on the first layer depth map D corresponding to each piece of color information texture data C. More specifically, when a valid depth value is stored in the first layer depth map D, the color information texture data C corresponding to the coordinates at which the depth value is stored is determined to be the region of the subject 90 and cuts it out.

[0150] The rendering unit 17b has a configuration in which the depth map generating unit 171 is removed from the rendering unit 17a (see FIG. 19) described in the first embodiment.

[0151] In the information processing system 10b, the rendering unit 17a generates depth maps for the visibility check process in the information processing system 10a, but the decoding unit 16b does not actually create depth maps, and only acquires the first-layer depth maps D generated by the encoding unit 13b. Then, the depth map cutout unit 165 of the decoding unit 16b cuts out color information texture data C corresponding to the area of ​​the subject 90 using the first-layer depth maps D.

[0152] Therefore, the computationally intensive process of generating a depth map can be reduced.

[0153] [2-2. Visibility Check Processing] Next, the visibility checking process will be described with reference to Fig. 22. Fig. 22 is a diagram for explaining the process of checking visibility.

[0154] When generating images of subject 90 viewed from multiple different projection points Q, it is necessary to determine whether or not points on the surface of subject 90 are visible from projection points Q. In other words, the color information texture data C recorded on projection surface R(u,v) is the surface closest to projection point Q, and the surface on the far side that is blocked by this surface is not visible from projection point Q.

[0155] 22, point Pa (an example of a first point) of subject 90 is visualized from projection point Qa, and the depth value (first distance) of point Pa is recorded on projection plane Ra. On the other hand, when point Pa is observed from a different projection point Qb, point Pa cannot be visualized because it is occluded by point Pb on the near side. Then, the depth value (a distance different from the first distance) of point Pb is stored on projection plane Rb.

[0156] 22, the three-dimensional coordinates of point Pa calculated from the depth value stored in projection plane Ra do not match the three-dimensional coordinates of point Pb calculated from the depth value stored in projection plane Rb. That is, when the three-dimensional position of point Pa according to the depth value seen from projection point Qa does not match the three-dimensional position according to the depth value corresponding to the direction of point Pa seen from projection point Qb different from projection point Qa, the visibility confirmation unit 172 determines that point Pa seen from projection point Qa is not visible from projection point Qb different from projection point Qa.

[0157] Then, when drawing an image of the subject 90 seen from the projection point Qb, the drawing unit 173 does not use the color information texture data C of the point Pb, but uses the color information texture data C projected onto a projection point close to the projection point Qb that has been determined to visualize the point Pa. Also, a mixture of color information texture data C projected onto a plurality of projection points close to the projection point Qb that visualize the point Pa may be used.

[0158] [2-3. Effects of the second embodiment] As described above, in the encoding device 40b (information processing device) of the second embodiment, the tiling processing unit 134b (color / distance acquisition unit) acquires color information texture data C (texture information) obtained by viewing the subject 90 from a plurality of projection points Q and a first distance d1 obtained by viewing the subject 90 from the same plurality of projection points Q as used to acquire the color information texture data C, in association with each other.

[0159] This allows one of the first layer depth maps D to be the first layer depth map Dm1 paired with the second layer depth map Dm2, thereby reducing the amount of data transmitted by the transmitter 14.

[0160] Furthermore, in the decoding device 50b (information processing device) of the second embodiment, the visibility confirmation unit 172 determines, based on information acquired by the tiling processing unit 134b (color / distance acquisition unit), that point Pa seen from projection point Qa is not visible from projection point Qb when the three-dimensional position of point Pa according to the first distance d1 seen from projection point Qa does not match the three-dimensional position according to the first distance d1 corresponding to the direction of point Pa seen from projection point Qb different from projection point Qa.

[0161] This makes it possible to perform a visibility check using the tiling image T2 with simple processing.

[0162] 3. Third embodiment [3-1. Operation of the Third Embodiment] In the first and second embodiments, when the shape of the subject 90 is complex, it is difficult to seamlessly digitize the 3D shape of the surface of the subject 90 due to the occurrence of occlusion. In order to solve this problem, the information processing system 10b of the third embodiment decomposes the subject 90 into multiple objects, and generates multiple first layer depth maps Dm1 and second layer depth maps Dm2 for each of the decomposed objects from the same projection point Q, thereby seamlessly digitizing the surface of the subject 90.

[0163] Note that the configuration (hardware configuration, functional configuration) of the information processing system 10b of the third embodiment is the same as that of the information processing system 10b (encoding device 40b, decoding device 50b) described in the second embodiment, and therefore a description thereof will be omitted.

[0164] Next, the operation of the information processing system of the third embodiment will be described with reference to Fig. 23. Fig. 23 is a diagram for explaining the operation of the information processing system of the third embodiment.

[0165] In the encoding device 40b included in the information processing system 10b, the 3D model generating unit 12 decomposes the subject 90 into a plurality of objects. Then, the first layer depth map generating unit 131 and the second layer depth map generating unit 132 generate a first layer depth map Dm1 and a second layer depth map Dm2, which are viewed from the same projection point Q, for each of the decomposed objects. That is, the first layer depth map generating unit 131 and the second layer depth map generating unit 132 acquire a first distance d1 and a second distance d2, which are viewed from the same projection point Q, for each of the objects into which the subject 90 is decomposed. Note that, as described in the first embodiment, when a straight line that starts from the projection point Q and reaches the subject 90 passes through the subject 90 a plurality of times, a pair of a first point P1 and a second point P2 is generated each time the straight line passes through the subject 90.

[0166] 23, the 3D model generation unit 12 decomposes a subject 90 to be modeled into an arrow-shaped first object and a rectangular second object. Then, the first layer depth map generation unit 131 and the second layer depth map generation unit 132 generate a first layer depth map Dm1 and a second layer depth map Dm2 for the first object as viewed from a projection point Q.

[0167] However, since the information processing system 10b needs to reconstruct an image of the subject 90 viewed from the projection point Q (view dependent rendering), the first layer depth map Dm1 is a depth map of the subject 90 viewed from the projection point Q as a whole.

[0168] That is, the first layer depth map generation unit 131 generates a first layer depth map Dm1 in which the depth values ​​of the dotted lines shown in Fig. 23 are stored for the subject 90. The second layer depth map generation unit 132 generates a second layer depth map Dm2 in which the depth values ​​of the thick lines representing the depth values ​​of the rear side of the arrow-shaped first object are stored for the first object. The first layer depth map Dm1 and the second layer depth map Dm2 are used as a pair to reconstruct the subject 90.

[0169] Next, the first layer depth map generation unit 131 generates a first layer depth map Dm1 of the rectangular second object viewed from the projection point Q. Note that the first layer depth map Dm1 generated here is called a third layer depth map to distinguish it from the above-mentioned first layer depth map Dm1.

[0170] Then, the second layer depth map generating unit 132 generates a second layer depth map Dm2 of the second object viewed from the projection point Q. Note that the second layer depth map Dm2 generated here is called a fourth layer depth map to distinguish it from the above-mentioned second layer depth map Dm2. The third layer depth map and the fourth layer depth map generated in this manner are used as a pair to reconstruct the subject 90.

[0171] The rendering unit 17b performs rendering of each object by the above-mentioned method using a pair of depth maps for each object obtained by decomposing the subject 90. Then, when each rendered object is observed from a projection point, the rendered objects are synthesized so that the closest point becomes the surface of the subject 90, thereby seamlessly reconstructing the surface of the subject 90.

[0172] If the shape of the subject 90 is complex, the subject 90 may be decomposed into more objects and the same process as described above may be repeated to generate more pairs of depth maps.

[0173] [3-2. Effects of the third embodiment] As described above, in the encoding device 40b (information processing device) of the third embodiment, the first layer depth map generation unit 131 and the second layer depth map generation unit 132 (distance acquisition unit) acquire the first distance d1 and the second distance d2 for each object obtained by decomposing the subject 90 from the same projection point Q. Then, the image of the subject 90 is reconstructed by combining each object reconstructed based on the first distance d1 and the second distance d2.

[0174] This makes it possible to seamlessly reconstruct an image of the subject 90 even if the subject 90 has a complex shape.

[0175] 4. Fourth Embodiment [4-1. Operation of the Fourth Embodiment] In the first and second embodiments, when the shape of the subject 90 is complex, it is difficult to seamlessly digitize the 3D shape of the surface of the subject 90 due to the occurrence of occlusion. In order to solve this problem, the information processing system 10b of the fourth embodiment decomposes the subject 90 into multiple objects, and generates a first layer depth map Dm1 and a second layer depth map Dm2 for each of the decomposed objects from multiple projection points, thereby seamlessly digitizing the surface of the subject 90.

[0176] The configuration (hardware configuration, functional configuration) of the information processing system 10b of the fourth embodiment is the same as that of the information processing system 10b (encoding device 40b, decoding device 50b) described in the second embodiment, so a description thereof will be omitted.

[0177] Next, the operation of the information processing system of the fourth embodiment will be described with reference to Fig. 24. Fig. 24 is a diagram for explaining the operation of the information processing system of the fourth embodiment.

[0178] In the encoding device 40b included in the information processing system 10b, the 3D model generating unit 12 decomposes the subject 90 into a plurality of objects. Then, the first layer depth map generating unit 131 and the second layer depth map generating unit 132 generate a first layer depth map Dm1 and a second layer depth map Dm2 for each of the decomposed objects, respectively, as viewed from different projection points Q1, Q2, .... That is, for each of the objects into which the subject 90 is decomposed, a first distance d1 and a second distance d2 as viewed from different projection points Q1, Q2, ... are obtained. Note that, as described in the first embodiment, when a straight line starting from the projection point Q to the subject 90 passes through the subject 90 a plurality of times, a pair of a first point P1 and a second point P2 is generated each time the straight line passes through the subject 90.

[0179] For example, as shown in Fig. 24, the 3D model generation unit 12 decomposes a subject 90 to be modeled into an arrow-shaped first object and a rectangular second object. Then, the first layer depth map generation unit 131 and the second layer depth map generation unit 132 generate a first layer depth map Dm1 and a second layer depth map Dm2 for the first object as viewed from a projection point Q1. In addition, the first layer depth map generation unit 131 and the second layer depth map generation unit 132 generate a first layer depth map Dm1 and a second layer depth map Dm2 for the first object as viewed from a projection point Q2.

[0180] However, since the information processing system 10b needs to reconstruct the image of the subject 90 seen from each projection point Q1, Q2 (View Dependent rendering), the first layer depth map Dm1 is a depth map showing the entire subject 90 from each projection point Q1, Q2.

[0181] That is, the first layer depth map generating unit 131 generates a first layer depth map Dm1 in which the entire subject 90 is viewed from the projection point Q1. As a result, the first layer depth map Dm1 in which the depth values ​​of the dotted lines shown in Fig. 24 are stored is generated. Then, the second layer depth map generating unit 132 generates a second layer depth map Dm2 in which the depth values ​​of the thick lines representing the depth values ​​of the rear side of the arrow-shaped first object are stored. The first layer depth map Dm1 and the second layer depth map Dm2 are used as a pair to reconstruct the subject 90.

[0182] Next, the first layer depth map generating unit 131 generates a first layer depth map Dm1 in which the entire subject 90 is viewed from the projection point Q2. As a result, the first layer depth map Dm1 in which the depth values ​​of the dotted lines shown in Fig. 24 are stored is generated. Then, the second layer depth map generating unit 132 generates a second layer depth map Dm2 in which the depth values ​​of the thick lines representing the depth values ​​of the rear surface side of the rectangular second object are stored. The first layer depth map Dm1 and the second layer depth map Dm2 are used as a pair to reconstruct the subject 90.

[0183] [4-2. Effects of the Fourth Embodiment] As described above, in the encoding device 40b (information processing device) of the fourth embodiment, the first layer depth map generation unit 131 and the second layer depth map generation unit 132 (distance acquisition unit) acquire the first distance d1 and the second distance d2 for each object obtained by decomposing the subject 90 from different projection points Q1 and Q2. Then, the image of the subject 90 is reconstructed by combining each object reconstructed based on the first distance d1 and the second distance d2.

[0184] This makes it possible to seamlessly reconstruct an image of the subject 90 even if the subject 90 has a complex shape.

[0185] (5. Application Examples of the Present Disclosure) [5-1. Content Creation] For example, video content may be produced by combining 3D model 90M of subject 90 generated by 3D model generation unit 12 with a 3D model managed by another server. In addition, for example, when background data exists from an imaging device such as Lidar, content can be produced in which subject 90 appears as if it were in a location indicated by the background data by combining 3D model 90M of subject 90 generated by 3D model generation unit 12 with the background data.

[0186] [5-2. Virtual space experience] For example, a subject 90 generated by the 3D model generating unit 12 can be placed in a virtual space where a user becomes an avatar and communicates. In this case, the user becomes an avatar and can view a live-action subject 90 in the virtual space.

[0187] [5-3. Application to remote communication] For example, by transmitting the 3D model 90M of the subject 90 generated by the 3D model generating unit 12 to a remote location from the transmitting unit 14, a user at a remote location can view the 3D model 90M of the subject 90 through a playback device at the remote location. For example, by transmitting the 3D model 90M of the subject 90 in real time, the subject 90 and a user at a remote location can communicate in real time. For example, a case can be assumed in which the subject 90 is a teacher and the user is a student, or the subject 90 is a doctor and the user is a patient.

[0188] [5-4.Other] For example, a free viewpoint video of sports or the like can be generated based on the 3D models 90M of multiple subjects 90 generated by the 3D model generation unit 12. Also, an individual can distribute the image of himself / herself generated by the 3D model generation unit 12 to a distribution platform. In this way, the contents of the embodiments described in this specification can be applied to various technologies and services.

[0189] Although the present disclosure has been described using several embodiments, these embodiments may be executed in any device as long as the device has the necessary functional blocks and is capable of obtaining the necessary information.

[0190] Also, for example, each step of one flowchart may be executed by one device, or may be shared and executed by multiple devices. Furthermore, when one step includes multiple processes, the multiple processes may be executed by one device, or may be shared and executed by multiple devices. In other words, multiple processes included in one step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as one step.

[0191] Also, for example, the processing of the steps of a program executed by a computer may be executed in chronological order according to the order described in this specification, or may be executed in parallel, or individually at a required timing such as when a call is made. In other words, the processing of each step may be executed in an order different from the order described above, so long as no contradiction occurs. Furthermore, the processing of the steps of a program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.

[0192] Also, for example, a plurality of techniques related to the present technology can be implemented independently and independently, as long as no contradiction occurs. Of course, any of a plurality of the present technologies can be applied and implemented. For example, a part or all of the present technology described in any of the embodiments can be implemented in combination with a part or all of the present technology described in other embodiments. Also, a part or all of any of the above-mentioned present technologies can be implemented in combination with other technologies not described above.

[0193] In addition, the effects described in this specification are merely examples and are not intended to be limiting, and other effects may be achieved. In addition, the embodiments of the present disclosure are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit and scope of the present disclosure.

[0194] For example, the present disclosure can also be configured as follows.

[0195] (1) a distance acquisition unit that acquires, based on a 3D shape of a surface of a subject, a first distance between a first point where a straight line from a projection point to the subject contacts the subject and the projection point, and a second distance between a second point where a straight line that enters the subject from the first point penetrates the subject to the outside and the projection point, Information processing device. (2) The distance acquisition unit is If the first point does not exist, a predetermined first distance value is obtained as the first distance; If the second point does not exist, a predetermined second distance value smaller than the first distance value is obtained as the second distance. The information processing device according to (1). (3) The distance acquisition unit is setting the first distance value and the second distance value to equal values; The information processing device according to (2). (4) a transmitting unit that transmits the first distance and the second distance acquired by the distance acquisition unit to a decoding device that reconstructs a three-dimensional shape of the object by associating the first distance and the second distance acquired by the distance acquisition unit. The information processing device according to any one of (1) to (3). (5) a color / distance acquisition unit that acquires texture information of the object viewed from a plurality of projection points and the first distance viewed from the same plurality of projection points as those used to acquire the texture information in association with each other; The information processing device according to any one of (1) to (4). (6) the distance acquisition unit acquires the first distance and the second distance from the same projection point to each of the objects obtained by decomposing the subject. The information processing device according to any one of (1) to (5). (7) the distance acquisition unit acquires the first distance and the second distance from different projection points to each object obtained by decomposing the subject. The information processing device according to any one of (1) to (5). (8) obtain a first distance between a first point at which a straight line from a projection point to the object contacts the object and the projection point based on a 3D shape of the surface of the object, or obtain a predetermined first distance value as the first distance when the first point does not exist; obtain a second distance between the projection point and a second point at which the straight line, which entered the subject from the first point, penetrates from the subject to the outside, or, when the second point does not exist, obtain a second distance value that is smaller than the first distance value as the second distance; a region determining unit that identifies a region of the subject based on a magnitude relationship between the first distance and the second distance; Information processing device. (9) Further comprising a closed space generating unit that connects an end portion of the subject on a front side as viewed from the projection point based on the plurality of first distances related to the projection point and an end portion of the subject on a rear side as viewed from the projection point based on the plurality of second distances related to the projection point, The information processing device according to (8). (10) The closed space generating unit includes: When the first distances of two of the three points adjacent to each other in a triangular shape represent the front surface of the subject, and the first distance of the remaining point does not represent the front surface of the subject, The remaining one point is substituted for one of two points representing the rear surface of the subject among the second distances corresponding to the three points, Substituting one point that does not represent the rear surface of the subject among the second distances corresponding to the three points into one point that is opposite to the point into which the remaining point is substituted among the two points that represent the front surface of the subject. The information processing device according to (9). (11) a color / distance acquisition unit that acquires texture information of the object viewed from a plurality of projection points and the first distance viewed from the same plurality of projection points as used to acquire the texture information in association with each other; and a visibility confirmation unit that, when a three-dimensional position of a first point according to the first distance seen from the projection point does not match a three-dimensional position according to the first distance corresponding to a direction of the first point seen from a projection point other than the projection point, determines that the first point seen from the projection point is not visible from the projection point other than the projection point, based on the information acquired by the color / distance acquisition unit. The information processing device according to (8). (12) Based on a 3D shape of the surface of the subject, a first distance between a first point where a straight line from a projection point to the subject contacts the subject and the projection point, and a second distance between a second point where the straight line, which entered the subject from the first point, penetrates the subject to the outside and the projection point are obtained, and a 3D model of the subject is generated by associating the position of the projection point, the first distance, and the second distance. How to generate 3D models. (13) Computer, a distance acquisition unit that acquires, based on a 3D shape of a surface of a subject, a first distance between a first point at which a straight line extending from a projection point to the subject contacts the subject and the projection point, and a second distance between a second point at which the straight line extending from the first point into the subject penetrates from the subject to the outside and the projection point; A program that functions as a [Explanation of symbols]

[0196] 10a, 10b...information processing system, 11...data acquisition unit, 12...3D model generation unit, 13a, 13b...encoding unit, 14...transmission unit, 15...reception unit, 16a, 16b...decoding unit, 17a, 17b...rendering unit, 18...display unit, 40a, 40b...encoding device (information processing device), 50a, 50b...decoding device (information processing device), 70 (70a, 70b, 70c, 70d, 70e)...imaging device, 90...subject, 90M...3D model, 131...first layer depth map generation unit (distance acquisition unit), 132...second layer depth map generation unit (distance acquisition unit), 133...color information packing unit, 134a, 134b...tiling processing unit (color / distance acquisition unit), 135...image compression processing unit, 161...image decompression processing unit processing unit, 162... depth effective area determination unit (area determination unit), 163... triangular patch generation unit (closed space generation unit), 164... color information texture extraction unit, 165... depth map extraction unit, 171... depth map generation unit, 172... visibility confirmation unit, 173... drawing unit, C... color information texture data (texture information), D1, D2... quantized depth value (depth value), Dm1, D... first layer depth map, Dm2... second layer depth map, d1... first distance, d2... second distance, N... number of effective points, P1... first point, P2... second point, Q, Q1, Q2, Qa, Qb... projection point, R, Ra, Rb... projection surface, T1, T2... tiling image, far... distance (first distance value), near... distance (second distance value)

Claims

1. a distance acquisition unit that acquires, based on a 3D shape of a surface of a subject, a first distance between a first point at which a straight line extending from a projection point to the subject contacts the subject and the projection point, and a second distance between a second point at which the straight line extending from the first point into the subject penetrates from the subject to the outside and the projection point. Information processing device.

2. The distance acquisition unit is If the first point does not exist, a predetermined first distance value is obtained as the first distance; If the second point does not exist, a predetermined second distance value smaller than the first distance value is obtained as the second distance. The information processing device according to claim 1 .

3. The distance acquisition unit is setting the first distance value and the second distance value to equal values; The information processing device according to claim 2 .

4. a transmitting unit that transmits the first distance and the second distance acquired by the distance acquisition unit to a decoding device that reconstructs a three-dimensional shape of the object by associating the first distance and the second distance acquired by the distance acquisition unit. The information processing device according to claim 1 .

5. a color / distance acquisition unit that acquires texture information of the object viewed from a plurality of projection points and the first distance viewed from the same plurality of projection points as those used to acquire the texture information in association with each other, The information processing device according to claim 1 .

6. the distance acquisition unit acquires the first distance and the second distance from the same projection point to each of the objects obtained by decomposing the subject. The information processing device according to claim 1 .

7. the distance acquisition unit acquires the first distance and the second distance from different projection points to each of the objects obtained by decomposing the subject. The information processing device according to claim 1 .

8. Based on a 3D shape of the surface of the subject, a first distance between a first point at which a straight line from a projection point to the subject contacts the subject and the projection point, or, when the first point does not exist, a predetermined first distance value is obtained as the first distance; a second distance between the projection point and a second point at which the straight line, which entered the subject from the first point, penetrates from the subject to the outside, or a predetermined second distance value smaller than the first distance value is obtained as the second distance when the second point does not exist; a region determining unit that identifies a region of the subject based on a magnitude relationship between the first distance and the second distance; Information processing device.

9. A closed space generating unit that connects a front end of the subject as viewed from the projection point based on the plurality of first distances related to the projection point and a rear end of the subject as viewed from the projection point based on the plurality of second distances related to the projection point, The information processing device according to claim 8.

10. The closed space generating unit includes: When the first distances of two of the three points adjacent to each other in a triangular shape represent the front surface of the subject, and the first distance of the remaining point does not represent the front surface of the subject, The remaining one point is substituted for one of two points representing the rear surface of the subject among the second distances corresponding to the three points, Substituting one point that does not represent the rear surface of the subject, among the second distances corresponding to the three points, for one point that is on the opposite side to the point into which the remaining point is substituted, among the two points that represent the front surface of the subject. The information processing device according to claim 9.

11. a color / distance acquisition unit that acquires texture information of the subject viewed from a plurality of projection points and the first distance viewed from the same plurality of projection points as used to acquire the texture information in association with each other; a visibility confirmation unit that, when a three-dimensional position of a first point according to the first distance as seen from the projection point does not match a three-dimensional position according to the first distance corresponding to a direction of the first point as seen from a projection point other than the projection point, determines that the first point as seen from the projection point is not visible from the projection point other than the projection point, based on the information acquired by the color / distance acquisition unit. The information processing device according to claim 8.

12. based on a 3D shape of the surface of the subject, a first distance between a first point where a straight line from a projection point to the subject contacts the subject and the projection point, and a second distance between a second point where the straight line, which entered the subject from the first point, penetrates the subject to the outside and the projection point, are obtained, and a 3D model of the subject is generated by associating the position of the projection point, the first distance, and the second distance; How to generate a 3D model.

13. Computer, a distance acquisition unit that acquires, based on a 3D shape of a surface of a subject, a first distance between a first point at which a straight line extending from a projection point to the subject contacts the subject and the projection point, and a second distance between a second point at which the straight line extending from the first point into the subject penetrates from the subject to the outside and the projection point; A program that functions as a

Images