Information processing device and information processing method

Abstract

The present invention improves the convenience in generating a three-dimensional model using a two-dimensional image.　A depth data acquisition unit acquires, for each of N (N being an integer of two or higher) viewpoints, depth data indicating the depth from a predetermined viewpoint to a feature point of an object. A surface labeling unit 53 generates labeling data indicating a result of labeling the surface of the object on the basis of the N items of depth data. An image data acquisition unit 54 acquires M (M being an integer of N or lower) items of image data obtained as a result of capturing the object from a viewpoint synchronized with each of M viewpoints among the N viewpoints. A three-dimensional model generation unit 55 generates a three-dimensional model for the object on the basis of the M items of image data and the labeling data.

Description

[Technical Field]

[0001] The present invention relates to an information processing device and an information processing method. [Background technology]

[0002] BACKGROUND ART Conventionally, there have been techniques for generating a three-dimensional model from two-dimensional images (a large number of two-dimensional images) containing a subject (see, for example, Patent Document 1 and Non-Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2010-145186 [Non-patent literature]

[0004] [Non-Patent Document 1] Thomas M. et Al., “Instant Neural Graphics Primitives with a Multiresolution Hash Encoding”, ACM Trans. Graph., Vol.4, Num.4, pp.102:1--102:15, July 2022, https: / / doi.org / 10.1145 / 3528223.3530127 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the prior art including the above-mentioned Patent Document 1 and Non-Patent Document 1 has not been able to fully meet the demands for accuracy and speed in generating three-dimensional models.

[0006] The present invention has been made in view of the above circumstances, and aims to improve the convenience of generating a three-dimensional model using two-dimensional images. [Means for solving the problem]

[0007] In order to achieve the above object, an information processing device according to one aspect of the present invention comprises: a depth data acquisition means for acquiring depth data indicating the depth from a predetermined viewpoint to a feature point of an object for each of N viewpoints (N is an integer value of 2 or more); an object surface labeling means for generating labeling data indicating a result of labeling the surface of the object based on the N depth data; an image data acquisition means for acquiring M image data obtained by capturing images of the object from viewpoints synchronized with M (M is an integer value equal to or less than N) viewpoints among the N viewpoints; a model generation means for generating a three-dimensional model of the object based on the image data of M and the labeling data; Equipped with.

[0008] An information processing method according to one aspect of the present invention is an information processing method corresponding to the information processing device according to the above-described one aspect of the present invention. [Effects of the Invention]

[0009] According to the present invention, it is possible to improve the convenience in generating a three-dimensional model using two-dimensional images. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram showing an overview of a service that can be realized by a model generation device according to an embodiment of the information processing device of the present invention. [Figure 2] FIG. 2 is a diagram showing an example of how to arrange cameras and lidars to acquire depth data and image data in the present service shown in FIG. 1. [Figure 3] FIG. 2 is a block diagram showing an example of the hardware configuration of a model generation device applied to the present service described with reference to FIG. 1, that is, a model generation device according to an embodiment of the information processing device of the present invention. [Figure 4] FIG. 4 is a functional block diagram showing an example of a functional configuration of the model generating device of FIG. 3. [Figure 5] 5 is a diagram showing an example of a block for generating a three-dimensional model in a model generating device having the functional configuration of FIG. 4. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0012] The information processing device according to the embodiment of the present invention is premised on the use of an algorithm for generating a three-dimensional model based on a two-dimensional image. In other words, a service to which an embodiment of the information processing device of the present invention is applied (hereinafter referred to as "this service") acquires a two-dimensional image of a specified object that exists in the real world and generates a three-dimensional model from that two-dimensional image.

[0013] First, the basic prior art technology described in the above-mentioned Patent Document 1 will be described. Conventional photogrammetry technology employs an algorithm that extracts feature points from multiple images of an object taken from multiple viewpoints, generates a point cloud in three-dimensional space by associating the feature points across the images, and then generates a three-dimensional image by adding additional points other than the feature points. This algorithm performs linear interpolation, reconstructing the distances from several viewpoints to the feature points as a point cloud in three-dimensional space based on triangulation techniques. Therefore, the conventional technology described in Patent Document 1 and elsewhere suffers from the problem of extremely poor reproducibility of angles between images.

[0014] In response to this, with the recent advances in machine learning technology, NeRF (Neural Radiance Fields) using neural networks and its advanced methods (algorithms) have been proposed. NeRF is an algorithm that can perform nonlinear interpolation between multiple viewpoints using a neural network. More specifically, in NeRF, a coarse grid is first generated and a learning process is performed, and then the results are used to learn to generate a finer grid, allowing the final 3D model to be output as the learning result.

[0015] Furthermore, in a technique (algorithm) called instant-ngp described in Non-Patent Document 1, the training data is encoded using a hash function during the training process, so that the training process, which required about three days in conventional NeRF, can be completed in just a few seconds.

[0016] The present invention is based on the prior art and aims to speed up the learning process for generating a three-dimensional model.

[0017] Using the example in Figure 1, we will explain the outline of the process for generating a 3D model using this service. In FIG. 1, an object T for which a 3D model is to be generated by this service is shown in a 3D space.

[0018] In this service, to generate a 3D model of object T, depth data obtained as a result of measurements by a lidar from N viewpoints (N is an integer value of 2 or greater) and image data of images captured by a camera from M viewpoints (M is an integer value of N or less) are used to generate a 3D model of object T.

[0019] In the following explanation of Figure 1, for simplicity's sake, the depth data and image data may be acquired at different times, but the explanation will be given assuming that a set of depth data and image data acquired simultaneously from a certain viewpoint is used. 1, it is assumed that N=M=2, and two viewpoints P1 and P2 are used for the explanation. When it is not necessary to distinguish between the points, the multiple points will be collectively referred to as point P.

[0020] When acquiring depth data and image data from the same viewpoint, the following methods can be used. For example, as a first method, a camera C1 is installed at viewpoint P1 in Fig. 1, image data is acquired from the camera C1, and then a lidar D1 is installed at viewpoint P1 and depth data is acquired from the lidar D1. In this way, a set of depth data and image data from two identical viewpoints is acquired.

[0021] FIG. 2 is a diagram showing an example of how to arrange cameras and lidars to acquire depth data and image data in the present service shown in FIG. 1. As another example, as a second method, as shown in Fig. 2, the camera C1 and the lidar D1 are fixed to each other in advance using a predetermined jig, and the camera C1 is placed at the viewpoint P1. Next, image data is acquired from the camera C1, and depth data is acquired from the lidar D1 at the same time. Then, the relative positions of the camera C1 and the lidar D1 and the measurement direction (line of sight) are calibrated using the predetermined jig, so that depth data and image data from the same viewpoint are acquired. Here, as a result of the calibration, both the image data and the depth data are linked to the same viewpoint, and therefore it can be said that the image data and the depth data are synchronized with each other. In the following, the service will be described assuming that the second method is adopted.

[0022] First, a captured image G1 is obtained by capturing an image of the object T using a camera C1 from a viewpoint P1 in the positive direction of the axis X of the object T. At the same time, depth data from a viewpoint synchronized with the viewpoint P1 is acquired using a lidar D1. As described above, the depth data is appropriately calibrated. The captured image G1 contains information about the shape and color of the object T as seen from the viewpoint P1.

[0023] Next, a captured image G2 is obtained by capturing an image of the object T using the camera C2 from a viewpoint P2 in the positive direction of the axis Y of the object T. At the same time, depth data from a viewpoint synchronized with the viewpoint P2 is acquired using the lidar D2. As described above, the depth data is appropriately calibrated. The captured image G2 contains information about the shape and color of the object T as seen from the viewpoint P2.

[0024] In the explanation of Figure 1, cameras C1 and C2 and lidars D1 and D2 are used at viewpoints P1 and P2, respectively. However, image data and depth data may be acquired sequentially by moving camera C1 and lidar D1 from viewpoint P1 to viewpoint P2 and using them as camera C2 and lidar D2. In this way, when there is no need to distinguish between cameras and LIDARs at multiple locations, they will be collectively referred to as "camera C" and "LIDAR D," respectively. In addition, in such a case, the image captured by camera C will be referred to as "captured image G."

[0025] For example, in the conventional method described in Patent Document 1, etc., a 3D model is generated using only image data of multiple captured images such as captured images G1 and G2, making it difficult to complement shadowed portions of the image, etc. Furthermore, even in the method described in Non-Patent Document 1, etc., generating a higher-resolution 3D model requires either a certain amount of calculation time or a large amount of calculation resources.

[0026] As will be described in more detail later, this service generates a 3D model of the object T using image data of captured images G acquired by cameras C from multiple viewpoints P as described above, and depth data acquired by lidar D. This enables this service to generate 3D models more quickly.

[0027] 1, an arrow passing from viewpoint P1 through a predetermined pixel PX1 in captured image G1 is shown as a two-dot chain line, and a white circle and a black circle are shown at points on the arrow indicated by the two-dot chain line.

[0028] The points on the arrow indicated by the open circles indicate that the points have not yet come into contact with the object T when viewed from the viewpoint P1. The points on the arrow indicated by black circles indicate that the points have come into contact with the object T when viewed from viewpoint P1.

[0029] That is, for example, if we consider the progression from viewpoint P1 along the arrow indicated by the two-dot chain line, there is no object T while the vehicle passes through the point on the arrow indicated by the open circle from viewpoint P1, and therefore no collision occurs. Then, the vehicle collides with object T between the point on the arrow indicated by the open circle and the point on the arrow indicated by the filled-in circle. The color of the point where the vehicle collides with object T is recorded as the color of a predetermined pixel PX1 in the captured image G1. Furthermore, if object T is opaque, points after the first filled-in circle will not be captured in the captured image G1. In this way, the color of the predetermined pixel PX1 of the captured image G1 is the color of the object T at the point where the object T collides on the arrow passing from the viewpoint P1 through the predetermined pixel PX1.

[0030] As mentioned above, this service simultaneously measures depth data from viewpoint P1. As a result, the distance between the above-mentioned open circle and black circle is obtained as depth data from the depth data. This service improves the speed of 3D model generation by limiting the area taken into consideration when generating a 3D model.

[0031] The outline of this service has been explained above using Figures 1 and 2. Below, the model generation device to which this service is applied will be explained using Figures 3 to 5.

[0032] FIG. 3 is a block diagram showing an example of the hardware configuration of a model generation device applied to the present service described with reference to FIG. 1, that is, a model generation device according to an embodiment of the information processing device of the present invention. The model generation device 1 includes a CPU 11, a GPU 12, a ROM 13, a RAM 14, a bus 15, an input / output interface 16, an output unit 17, an input unit 18, a memory unit 19, a communication unit 20, and a drive 21.

[0033] The CPU 11 and the GPU 12 execute various processes according to a program recorded in the ROM 13 or a program loaded from the storage unit 19 into the RAM 14 . The GPU 12 has a compute unit that executes software processing and an RT core that executes hardware processing. The RT core executes ray tracing in hardware for a predetermined three-dimensional space that includes an object. The RAM 14 also stores data and the like required for the CPU 11 and GPU 12 to execute various processes.

[0034] The CPU 11, GPU 12, ROM 13, and RAM 14 are connected to one another via a bus 15. An input / output interface 16 is also connected to this bus 15. An output unit 17, an input unit 18, a storage unit 19, a communication unit 20, and a drive 21 are connected to the input / output interface 16.

[0035] The output unit 17 is composed of a display, a speaker, etc., and outputs various information as images and sounds. The input unit 18 is composed of a keyboard, a mouse, etc., and is used to input various information.

[0036] The storage unit 19 is configured with a hard disk, a DRAM (Dynamic Random Access Memory), etc., and stores various data. The communication unit 20 communicates with other devices via a network including the Internet.

[0037] Removable media 31, such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, is appropriately loaded into the drive 21. A program read from the removable media 31 by the drive 21 is installed in the storage unit 19 as necessary. Furthermore, the removable medium 31 can also store various data stored in the storage unit 19 in the same manner as the storage unit 19.

[0038] Next, with reference to FIG. 4, the functional configuration of the model generation device 1 having the hardware configuration shown in FIG. 3 will be described. FIG. 4 is a functional block diagram illustrating an example of the functional configuration of the model generating device of FIG.

[0039] As shown in Figure 4, the CPU 11 of the model generating device 1 functions as an actual depth data acquisition unit 51, a depth data estimation unit 52, a surface labeling unit 53, an image data acquisition unit 54, a three-dimensional model generation unit 55, and a display control unit 56. In one area of ​​the storage unit 19, a depth model 80, labeling data 81, and a three-dimensional model 82 are stored.

[0040] The depth data acquisition unit acquires depth data from a predetermined viewpoint of N to the object T. The depth data includes information on the depth from the predetermined viewpoint to the feature points of the object T.

[0041] An example of the functional configuration of the depth data acquisition unit will be described below with reference to FIG. The depth data acquisition unit includes an actual depth data acquisition unit 51 and a depth data estimation unit 52. The actual depth data acquisition unit 51 acquires actual depth data of M obtained as a result of measurements from each of M viewpoints in the real world. That is, the actual depth data acquisition unit 51 acquires actual depth data of M obtained as a result of measurements from each of M viewpoints in the real world using a lidar D. The depth data estimation unit 52 estimates depth data of N based on the actual depth data of M acquired by the actual depth data acquisition unit 51, and acquires the estimated depth data of N. Specifically, for example, the depth data estimation unit 52 generates or updates a three-dimensional depth model 80 of the object T by performing a learning process based on the actual depth data of M acquired by the actual depth data acquisition unit 51 using an algorithm that uses a neural network. The three-dimensional depth model 80 of the object T is a model that allows for inferring depth data from a predetermined viewpoint. The depth model 80 is stored and managed in one area of ​​the storage unit 19.

[0042] The surface labeling unit 53 generates labeling data 81 indicating the result of labeling the surface of the object T based on the depth data of N acquired by the depth data acquisition unit. Labeling refers to recording the position of the surface of an object T in three-dimensional space at a position in three-dimensional virtual space where a three-dimensional model is generated. An example of labeling will be described later using Figure ★. The labeling data 81 is stored and managed in one area of ​​the storage unit 19.

[0043] The image data acquisition unit 54 acquires image data of M captured images G obtained by capturing images of an object T from viewpoints P synchronized with M (M is an integer value less than or equal to N) of the N viewpoints.

[0044] The three-dimensional model generation unit 55 generates a three-dimensional model 82 for the object T based on the image data of M and labeling data 81 acquired by the image data acquisition unit 54. The three-dimensional model 82 is stored and managed in one area of ​​the storage unit 19. The three-dimensional model generation unit 55 includes a block skip determination unit 551 and a color learning unit 552 .

[0045] When generating learning data for generating a three-dimensional model, the block skip determination unit 551 determines, based on the labeling data 81, whether or not the surface of the object T exists in a block through which the line of sight (the arrow indicated by the two-dot chain line in Figure 1) corresponding to a predetermined pixel in the captured image G from the viewpoint P passes. If the block skip determining unit 551 determines that the surface of the object T does not exist in the block, the block is determined not to contribute to the color of the predetermined pixel in the captured image G. Conversely, if the block skip determination unit 551 determines that the surface of the object T exists in the block, it is assumed that the block may contribute to the color of the specified pixel in the captured image G. Then, learning data is generated for the block that is determined by the block skip determining unit 551 to contribute to the color of the predetermined pixel. An example of blocks skipped by the block skip determining unit 551 will be described later with reference to FIG.

[0046] The color learning unit 552 uses learning data generated based on the determination result of the block skip determination unit 551 to perform learning to add color to the three-dimensional model 82, thereby generating or updating the three-dimensional model 82. Specifically, as described above, the learning data used in the color learning unit 552 is data on which learning processing is (substantially) not performed for blocks that do not contribute to the color of a predetermined pixel in the captured image G. This reduces the time required for learning processing in the color learning unit 552.

[0047] In this way, by using depth data in 3D modeling, learning in spaces where the object T does not exist (block-based spaces) can be omitted, enabling high-speed modeling. Furthermore, when generating a grid (voxels) with higher resolution compared to blocks in modeling, the surface of the object is labeled, i.e., the voxels in the space (space in block units) where the target object T exists are made dense. This allows for high-precision modeling as well as high-speed modeling.

[0048] The display control unit 56 executes control to display a three-dimensional model 82 of the object T on the user terminal 2 by executing a rendering process on the vicinity of the object T based on the depth model 80. This allows for faster image generation and display by omitting the rendering of areas that do not affect the color of the three-dimensional model 82 when the three-dimensional model 82 is viewed from each direction.

[0049] Furthermore, the display control unit 56 can execute control to display an image that is drawn as is using the network representation generated in the three-dimensional model 82 for the object T. Here, the 3D model 82 of the object T in a network representation refers to a representation form of a function created by a neural network. The representation form of a function created by a neural network is also called, for example, an implicit function representation. When the 3D model 82 is converted into a format using voxels, meshes, polygons, or the like, the data size becomes enormous. However, the data size of the 3D model 82 in the representation form of an implicit function representation is small. For this reason, employing a network representation (implicit function representation) has the advantage of high transfer speed when transmitting and receiving data of the 3D model 82 (for example, when downloading via the Internet). That is, the display control unit 56 can execute control to display the three-dimensional model 82 of the object T as is in the network representation without re-rendering it.

[0050] The model generation device to which this service is applied has been described above using FIGS. Next, we will explain in more detail the process for speeding up the generation of 3D models in this service. FIG. 5 is a diagram showing an example of blocks for generating a three-dimensional model in a model generating device having the functional configuration of FIG.

[0051] First, the concepts of blocks and voxels will be explained using FIG. The coarse grid shown in Fig. 5 indicates the boundaries of blocks obtained by dividing the virtual three-dimensional space in which the object T in Fig. 1 is placed into a grid-like pattern. The fine grid shown in Fig. 5 indicates boundaries divided into grids that are finer than the blocks.

[0052] 5 is an array of blocks LB and voxels VC at the coordinates of a certain axis Z. In other words, each region obtained by dividing the slice SLk into a predetermined first unit is a voxel VC. For example, if the voxels VC correspond to the resolution of the three-dimensional model 82 that will ultimately be generated, the process of generating the three-dimensional model will be inefficient if the voxels VC are used as units. Therefore, each region obtained by dividing the slice SLk into second units larger than the first units, in other words, a region consisting of a group of n voxels, is introduced as blocks BL1 to BL7 and BLK. In the example of Figure 5, n is 8 in total, 4 in the direction of axis X, 4 in the direction of axis Y, and 1 in the direction of axis Z. Note that, hereinafter, the direction of axis Z will be expressed as x x y, since there is only one. That is, each of blocks BL1 to BL7 is composed of n = 4 x 4 voxels VC. Hereinafter, when there is no need to distinguish between multiple voxels, they will be referred to as "voxels VC." Similarly, when there is no need to distinguish between individual blocks BL1 to BL7, etc., they will be referred to as "blocks BL."

[0053] The area of ​​the block BL indicated by the thick line in FIG. 5 is an area that may include objects of the two parts T1 and T2 of the target object T. That is, in the slice SLk, blocks BL that may include the surface of the object T and blocks BLK that are empty spaces are distinguished. The former blocks BL are reflected in the pixel values ​​(colors) of predetermined pixels in the captured image G, while the latter blocks BLK are not. Therefore, hereinafter, the former blocks BL are referred to as "processed blocks BL" and the latter blocks BLK are referred to as "unprocessed blocks BLK."

[0054] In Fig. 5, to facilitate understanding of the present invention, "processing blocks BL" are shown in bold lines, and "non-processing blocks BLK" are shown in dashed lines. Note that only "processing blocks BL" are shown in Figs. 3 to 5. 2, slice SLk has regions that may include two portions T1 and T2 of object T. Four processing blocks BL1 to BL4 are shown as regions that may include portion T1 of object T. Furthermore, three processing blocks BL5 to BL7 are shown as regions that may include portion T2 of object T.

[0055] 4 performs labeling by determining blocks BL that may include the surfaces of portions T1 and T2 of object T. Blocks surrounded by thick frames indicate blocks that have been labeled as containing the surface of object T.

[0056] The three-dimensional model generation unit 55 then learns (models) color information about the processing blocks BL, and generates color information about the non-processing blocks BLK as learning (modeling) data so that learning processing is not performed on the color information, and generates or updates the three-dimensional model 82. This speeds up the processing for generating or updating the three-dimensional model 82. That is, for example, in a captured image G (for example, captured image G2 in FIG. 1) captured from the positive direction of the axis Y of the portion T1 of the object T, learning data is generated without learning the unprocessed blocks BLK1 to BLK3.

[0057] Although one embodiment of the present invention has been described above, the present invention is not limited to the above-described embodiment, and modifications, improvements, etc. within the scope of achieving the object of the present invention are considered to be included in the present invention.

[0058] For example, in the above embodiment, it has been described that the number N of viewpoints P from which depth data is acquired is the same as the number M of viewpoints P from which image data is acquired, but this is not particularly limited. That is, the number N of viewpoints P from which depth data is acquired may be different from the number M of viewpoints P from which image data is acquired. In this case, for example, a depth model 80 is first generated from the depth data of N, and then the depth model 80 is used to calculate the depth data at a viewpoint P of M, which is different from the viewpoint P of N, from the depth model 80.

[0059] Furthermore, for example, the depth data may be data directly observed by a sensor such as a lidar D, or may be data estimated from other data. That is, for example, the depth data acquisition unit may acquire depth data at viewpoint P of M corresponding to image data from a pre-prepared depth model 80, rather than acquiring actual depth data or generating or updating a depth model 80 and using it as depth data at viewpoint P of M corresponding to image data.

[0060] Furthermore, for example, in the above-described embodiment, the camera C1 and the LIDAR D1 are fixed to each other in advance using a predetermined jig, and the relative positions and measurement direction (line of sight) of the camera C1 and the LIDAR D1 are calibrated using the predetermined jig, thereby acquiring depth data and image data from the same viewpoint. However, this is not particularly limited. That is, various calibration methods may be adopted. Specifically, for example, position information of the viewpoint P photographed by the camera C may be acquired and recorded using technology such as a GPS (Global Positioning System), and then the LIDAR D may acquire depth data from the same viewpoint using that position information. Needless to say, the order of data acquisition by the camera C and the LIDAR D does not matter.

[0061] Further, for example, the depth model 80 may be generated using a method for estimating depth from image data, such as structure from motion.

[0062] 4 is merely an example and is not particularly limited. That is, it is sufficient that the information processing system is provided with functions and databases that can execute the above-described series of processes as a whole, and the type of functional block used to realize this function is not particularly limited to the example in FIG. Furthermore, the locations of the functional blocks and databases are not limited to those shown in FIG. 4 and may be arbitrary.

[0063] The above-described series of processes can be executed by hardware or software. Furthermore, one functional block may be configured as a single piece of hardware, a single piece of software, or a combination thereof.

[0064] When a series of processes is executed by software, the programs that make up the software are installed into a computer or the like from a network or a recording medium. The computer may be a computer built on dedicated hardware. The computer may also be a computer capable of executing various functions by installing various programs, such as a server, a general-purpose smartphone, or a personal computer.

[0065] The recording medium containing such a program may be composed not only of a removable medium (not shown) that is distributed separately from the device main body in order to provide the program to the user, but also of a recording medium that is provided to the user in a state that is pre-installed in the device main body.

[0066] In this specification, the steps describing the program to be recorded on the recording medium include not only processes that are performed in chronological order, but also processes that are not necessarily performed in chronological order but are performed in parallel or individually. In addition, in this specification, the term "system" refers to an overall device that is made up of a plurality of devices, a plurality of means, etc.

[0067] In summary, the information processing system to which the present invention is applied is sufficient if it has the following configuration, and can take on a variety of different embodiments. That is, an information processing device to which the present invention is applied (for example, the model generation device 1 in FIG. 4) a depth data acquisition unit (e.g., a depth data acquisition unit shown in FIG. 4) that acquires depth data (e.g., depth data measured by the lidars D1 and D2 shown in FIG. 4) indicating the depth from a predetermined viewpoint (e.g., viewpoints P1 and P2 shown in FIG. 1) to a feature point of an object for each of N viewpoints (N is an integer value of 2 or more); an object surface labeling means (e.g., the surface labeling unit 53 in FIG. 4) that generates labeling data (e.g., the labeled block BL in FIG. 5) indicating the result of labeling the surface of the object based on the N depth data; an image data acquisition means (e.g., image data acquisition unit 54 in FIG. 4) for acquiring M image data (e.g., image data G1 and G2 in FIG. 1) obtained as a result of capturing images of the object from viewpoints (e.g., viewpoints P1 and P2 in FIG. 1) synchronized with M (M is an integer value equal to or less than N) viewpoints out of the N viewpoints; a model generation unit (for example, the three-dimensional model generation unit 55 in FIG. 4) that generates a three-dimensional model of the object based on the image data of M and the labeling data; It is enough to have this. This omits modeling (learning processing for colors, etc.) in a space where the object T does not exist, and achieves high-speed modeling. Furthermore, when generating a grid with higher resolution compared to blocks in modeling (for example, the voxels in Figure 5), it becomes possible to employ a method for densely packing the voxels in the space where the surface of the object is labeled (i.e., the block-unit space where the target object T exists). This makes it possible to achieve high-precision modeling as well as high-speed modeling.

[0068] The depth data acquisition means further comprises: a real depth data acquisition unit (e.g., the real depth data acquisition unit 51 in FIG. 4) that acquires M real depth data obtained as a result of measurements from M viewpoints in the real world; The depth data estimation unit may further include a depth data estimation unit (for example, the depth data estimation unit 52 in FIG. 4) that estimates the depth data of N based on the actual depth data of M and acquires the estimated depth data of N.

[0069] The depth data acquisition means further comprises: The depth data synchronized with each of the M viewpoints among the N viewpoints can be obtained as depth data calibrated to have the same viewpoint as the M image data.

[0070] a display control unit (for example, the display control unit 56 in FIG. 4) that executes control to display the three-dimensional model of the object by drawing only the vicinity of the object based on the depth data; The sensor may further include: This reduces the computational load during rendering processing by omitting rendering of areas where no object exists, enabling the 3D model to be drawn quickly.

[0071] Further, the display control means A control can be executed to display an image that is rendered as is of the network representation generated in the three-dimensional model of the object. This allows the 3D model to be rendered as a network representation. [Explanation of symbols]

[0072] 1···Model generation device, 2···User terminal, 11···CPU, 19···Storage unit, 21···Drive, 31···Removable media, 51···Actual depth data acquisition unit, 52···Depth data estimation unit, 53···Surface labeling unit, 54···Image data acquisition unit, 55···3D model generation unit, 56···Display control unit, 551···Block skip determination unit, 552···Color learning unit, 80···Depth model, 81···Labeling data, 82···3D model

Claims

1. a depth data acquisition means for acquiring depth data indicating the depth from a predetermined viewpoint to a feature point of an object for each of N viewpoints (N is an integer value of 2 or more); an object surface labeling means for generating labeling data indicating a result of labeling the surface of the object based on the N depth data; an image data acquisition means for acquiring M image data obtained by capturing images of the object from M (M is an integer value equal to or less than N) viewpoints synchronized with each other among the N viewpoints; a model generation means for generating a three-dimensional model of the object based on the image data of M and the labeling data; An information processing device comprising:

2. The depth data acquisition means a real depth data acquisition means for acquiring M real depth data obtained as a result of measurements from each of M viewpoints in the real world; a depth data estimation means for estimating N depth data based on the M actual depth data and acquiring the estimated N depth data; The information processing device according to claim 1 ,

3. The depth data acquisition means The depth data synchronized with each of the M viewpoints among the N viewpoints is acquired as depth data calibrated to have the same viewpoint as the M image data. The information processing device according to claim 1 .

4. a display control means for executing control to display the three-dimensional model of the object by rendering only the vicinity of the object based on the depth data; The information processing device according to claim 1 , further comprising:

5. The display control means and executing control to display an image rendered as a network representation generated in the three-dimensional model of the object. The information processing device according to claim 4 .

6. An information processing method executed by an information processing device, a depth data acquisition step of acquiring depth data indicating depths from a predetermined viewpoint to feature points of an object for each of N viewpoints (N is an integer value of 2 or more); an object surface labeling step of generating labeling data indicating a result of labeling the surface of the object based on the N depth data; an image data acquisition step of acquiring M image data obtained as a result of capturing images of the object from M (M is an integer value equal to or less than N) viewpoints synchronized with each other among the N viewpoints; a model generation step of generating a three-dimensional model of the object based on the image data of M and the labeling data; An information processing method including:

Images