3D reconstruction device, 3D reconstruction method, and program

Abstract

This three-dimensional reconstruction program (1) is provided with: an input unit (11) which inputs a panorama image (21) of a target indoor space, and three-dimensional coordinates (22) interpolated so as to connect corners to be imaged, shown on a drawing of the indoor space; an image encoding unit (12) which extracts image feature values from the panorama image (21); a shape encoding unit (13) which extracts three-dimensional shape feature values from the interpolated three-dimensional coordinates (22); a link calculation unit (14) which generates feature values that link image feature values and three-dimensional shape feature values; an image coordinate decoding unit (15) which generates boundary image coordinates by decoding the linked feature values; and a three-dimensional reconstruction unit (16) which performs three-dimensional reconstruction of the indoor space on the basis of the boundary image coordinates.

Description

[Technical field]

[0001] The present disclosure relates to a 3D reconstruction device, a 3D reconstruction method, and a program for performing 3D reconstruction of an indoor space. [Background technology]

[0002] Conventionally, HorizonNet, which uses a deep neural network, is one technique for reconstructing 3D structures from panoramic images of indoor spaces such as rooms. 3D reconstruction refers to reconstructing the original 3D structure from a projected image of a solid object or a planar image. A neural network is a mathematical model that reproduces the nerve cells (neurons) in the human brain and their connections, that is, a neural circuit network, on a computer and applies it to machine learning and other operations. A neural network is a multi-layer network consisting of an input layer, one or more hidden layers (intermediate layers), and an output layer. A deep neural network refers to a neural network that is made compatible with deep learning and has a network hierarchy of four or more layers. Non-Patent Document 1 describes a method for estimating a 3D room layout from a single panoramic image using HorizonNet. FIG. 5 is a reference to a photograph disclosed in Non-Patent Document 1. As shown in Fig. 5, existing technologies such as HorizonNet use only a panoramic image 21 as input information, and learn the corners and boundaries of wall surfaces only from the image. Then, a boundary B is detected, and 3D coordinates are estimated to perform 3D reconstruction 23. Non-Patent Document 2 describes an overview of PointNet, which is a deep neural network that can directly input a measured point cloud. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Cheng Sun, et al., "HorizonNet: Learning Room Layout with 1D Representation and Pano Stretch Data Augmentation," [online], [Retrieved October 6, 2021], Internet <URL:https: / / openaccess.thecvf.com / content_CVPR_2019 / papers / Sun_HorizonNet_Learning_Room_Layout_With_1D_Representation_and_Pano_Stretch_CVPR_2019_paper.pdf> [Non-Patent Document 2] Charles R. Qi, et al., "PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation," [online], [Retrieved October 6, 2021], Internet <URL:https: / / openaccess.thecvf.com / content_cvpr_2017 / papers / Qi_PointNet_Deep_Learning_CVPR_2017_paper.pdf> Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional boundary detection using only panoramic images is based on image features (edges, corners, etc.), and there are many indoor spaces where boundary detection from image features is difficult when multiple objects in the indoor space obscure the boundary. When conventional technology is applied to such indoor spaces, there is an issue that the accuracy of boundary detection decreases, resulting in a decrease in the accuracy of 3D image reconstruction.

[0005] The object of the present invention, made in consideration of the above circumstances, is to enable highly accurate 3D reconstruction of indoor spaces even in cases where boundary detection from image features is difficult, such as when a large number of objects are present in the indoor space. [Means for solving the problem]

[0006] In order to solve the above problem, one embodiment of a 3D reconstruction device is a 3D reconstruction device that performs 3D reconstruction of an indoor space, and includes an input unit that inputs a panoramic image of the target indoor space and 3D coordinates interpolated to connect corners of the subject to be photographed that are described in a drawing of the indoor space, an image encoding unit that extracts image features from the panoramic image, a shape encoding unit that extracts 3D shape features from the interpolated 3D coordinates, a concatenation calculation unit that generates features that concatenate the image features and the 3D shape features, an image coordinate decoding unit that generates image coordinates of the boundary by decoding the concatenated features, and a 3D reconstruction unit that performs 3D reconstruction of the indoor space based on the image coordinates of the boundary.

[0007] In order to solve the above problem, one embodiment of a 3D reconstruction method is a 3D reconstruction method for performing 3D reconstruction of an indoor space, and includes the steps of inputting a panoramic image of a target indoor space and 3D coordinates interpolated to connect corners of a subject to be photographed that are described on a drawing of the indoor space by a 3D reconstruction device, extracting image features from the panoramic image, extracting 3D shape features from the interpolated 3D coordinates, generating features that concatenate the image features and the 3D shape features, generating image coordinates of boundaries by decoding the concatenated features, and performing 3D reconstruction of the indoor space based on the image coordinates of the boundaries.

[0008] In order to achieve the above object, a program according to one embodiment causes a computer to function as the above three-dimensional reconstruction device. Effect of the Invention

[0009] According to the present disclosure, even in spaces where boundary detection from image features is difficult, such as when a large number of objects are present in an indoor space, highly accurate boundary detection can be achieved by combining panoramic images with 3D coordinates such as point clouds, thereby enabling highly accurate 3D reconstruction of indoor spaces. [Brief description of the drawings]

[0010] [Figure 1] 1 is a block diagram showing an example of the configuration of a 3D reconstruction device according to an embodiment. [Diagram 2] 1 is a flowchart illustrating an example of a 3D reconstruction method performed by a 3D reconstruction device according to an embodiment. [Diagram 3] 13 is a flowchart illustrating a procedure for a shape coding unit to extract three-dimensional shape features. [Figure 4] FIG. 1 is a block diagram showing a schematic configuration of a computer functioning as a three-dimensional reconstruction device. [Diagram 5] FIG. 1 is a diagram showing a conventional procedure for performing 3D reconstruction from a panoramic image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

[0012] Fig. 1 is a block diagram showing an example of the configuration of a 3D reconstruction device according to an embodiment. The 3D reconstruction device 1 shown in Fig. 1 includes an input unit 11, an image encoding unit 12, a shape encoding unit 13, a concatenation calculation unit 14, an image coordinate decoding unit 15, and a 3D reconstruction unit 16. The 3D reconstruction device 1 performs 3D reconstruction of an indoor space.

[0013] In this embodiment, by using a panoramic image 21 of an indoor space together with a drawing of the indoor space, highly accurate boundary detection is realized even for a space where it is difficult to obtain a sufficient image feature amount a. For this reason, it is necessary to prepare, as input data, one panoramic image 21 of the indoor space and three-dimensional coordinates 22 (with an arbitrary point as the origin) that are interpolated so as to connect corners of the subject depicted on the drawing of the indoor space.

[0014] The input unit 11 inputs a panoramic image 21 of a target indoor space and three-dimensional coordinates 22 that are interpolated to connect corners of the subject to be photographed that are depicted on a drawing of the indoor space. The input unit 11 needs to input one panoramic image and the corresponding three-dimensional coordinates 22 (obtained from the dimensions depicted on the drawing of the indoor space; an arbitrary point is set as the origin). The input unit 11 outputs the panoramic image 21 of the target indoor space to the image encoding unit 12, and outputs the interpolated three-dimensional coordinates 22 to the shape encoding unit 13.

[0015] The image encoding unit 12 extracts an image feature a represented by a matrix from the panoramic image 21. The image encoding unit 12 may extract the image feature a from the panoramic image 21 by using HorizonNet. The image encoding unit 12 outputs the image feature a to the concatenation calculation unit 14.

[0016] HorizonNet is a neural network that addresses the task of estimating a 3D room layout from a single panoramic image. HorizonNet's feature extractor inputs a single panoramic image 21 and extracts multiple features expressed as a matrix. Next, it inputs a feature map in which the multiple features are concatenated, and learns from it to generate image coordinates of boundaries representing the boundary positions between the floor and walls, the ceiling and walls, and the boundaries between walls. Then, through post-processing, the 3D room layout is reconstructed from the image coordinates of the boundaries.

[0017] The shape coding unit 13 extracts a three-dimensional shape feature value b represented by a matrix from the interpolated three-dimensional coordinates 22. The shape coding unit 13 may use PointNet to extract the three-dimensional shape feature value b from the interpolated three-dimensional coordinates 22. The shape coding unit 13 outputs the three-dimensional shape feature value b to the concatenation calculation unit 14. The procedure by which the shape coding unit 13 extracts the three-dimensional shape feature value b will be described later with reference to FIG. 3.

[0018] The concatenation calculation unit 14 generates feature c by concatenating image feature a extracted from the panoramic image 21 and three-dimensional shape feature b extracted from the interpolated three-dimensional coordinates 22, which are represented by respective matrices. The concatenation calculation unit 14 outputs the concatenated feature c to the image coordinate decoding unit 15.

[0019] The image coordinate decoding unit 15 generates image coordinates d of the boundary by decoding the linked feature amount c. The image coordinate decoding unit 15 may decode the linked feature amount c by using HorizonNet.

[0020] The three-dimensional reconstruction unit 16 performs three-dimensional reconstruction of the indoor space based on the image coordinates d of the boundary.

[0021] FIG. 2 is a flowchart illustrating an example of a 3D reconstruction method executed by a 3D reconstruction device according to an embodiment. <Step S01>

[0022] In step S01, the input unit 11 inputs the panoramic image 21. The panoramic image 21 prepared in advance before this step is executed is one in which the vertical direction of the image is aligned with the zenith direction. <Step S02>

[0023] In step S02, the input unit 11 inputs three-dimensional coordinates 22 that are interpolated to connect the corners of the subject of the panoramic image 21. Before executing this step, the three-dimensional coordinates of the subject of the panoramic image 21 (the three-dimensional coordinates of the corners obtained by setting the origin of the coordinate system to any one corner) are prepared in advance. The three-dimensional coordinates are interpolated to connect the corners to obtain the three-dimensional coordinates 22. The three-dimensional coordinates before the interpolation have three-dimensional coordinates of each point that forms a point cloud of only the corners, but the three-dimensional coordinates 22 also have three-dimensional coordinates of each point that forms a point cloud of the corners and the portions of the lines that are interpolated to connect the corners. <Step S03>

[0024] In step S03, the image encoding unit 12 extracts an image feature a represented by a matrix from the panoramic image 21. The above-mentioned HorizonNet or the like may be used to extract the image feature a. <Step S04>

[0025] In step S04, shape coding unit 13 extracts three-dimensional shape feature amount b represented by a matrix from three-dimensional coordinates 22. Shape coding unit 13 may extract three-dimensional shape feature amount b from three-dimensional coordinates 22 using PointNet. PointNet is a deep neural network to which a measured point cloud can be directly input. FIG. 3 is a flowchart explaining the procedure by which shape coding unit 13 extracts three-dimensional shape feature amount using PointNet. The procedure by which shape coding unit 13 extracts three-dimensional shape feature amount will be explained below, divided into steps S041 to S044.

[0026] In step S041, the T-net inputs a point cloud and outputs a 3×3 affine transformation matrix. The point cloud is expressed as a set of three-dimensional points {Pi| i = 1, ..., n}, and each point Pi has (x, y, z) coordinates. The T-net is a network that inputs a point cloud and outputs an affine transformation matrix. Affine transformation refers to performing image enlargement / reduction, rotation, translation, etc. together using a matrix. As shown in FIG. 3, the internal structure of the T-net has the same structure as that of the shape coding unit 13.

[0027] In step S042, the matrix multiplier performs an operation (Matrix multiply) to multiply the input point cloud by the affine transformation matrix output by the T-net. As a result, an n × 3 matrix is ​​obtained. This operation makes it possible to eliminate the influence of translation and rotation of the point cloud.

[0028] As mentioned above, a neural network is a mathematical model that models the nerve cells (neurons) in the human brain and their connections, in other words, the neural circuit network, and is reproduced on a computer for application in machine learning and other operations. A "perceptron" is the smallest unit that makes up a neural network, and is a function that outputs one value for multiple inputs. A "perceptron" is a function that outputs one signal for n input signals, and is expressed by the following formula (1). In formula (1), y is the output signal, function f() is the "activation function," and x i is the i-th input signal (i=0,1,---,n-1), w i is a variable that represents the weight applied to the i-th input signal, and b is a variable called the bias. y = f (w0*x0+ --- + w n-1 * x n-1 + b ) (1)

[0029] In step S043, the n×3 matrix obtained in step S042 is input to a multilayer perceptron (in FIG. 3, the multilayer perceptron is written as mlp). In FIG. 3, "mlp(64,128,256)" is a multilayer perceptron with output sizes of 64, 128, and 256. The multilayer perceptron is configured, for example, by connecting a fully connected layer and an activation function. The fully connected layer combines multiple numerical values ​​input to a certain node from multiple nodes (referring to modeled neurons) into one numerical value by linear conversion processing. The activation function is a function that nonlinearly converts an input value into another numerical value when outputting from a certain node to the next node. In this embodiment, ReLU is used as the activation function. ReLU (Rectified Linear Unit) is a function in which the output value is always 0 when the input value to the function is 0 or less, and the output value is the same as the input value when the input value is greater than 0.

[0030] In step S044, the 3D shape feature is obtained by Max Pooling. Max Pooling refers to selecting the maximum value from the output values ​​of each range and compressing them. This operation makes it possible to eliminate the influence of the order of each point in the point cloud. <Step S05>

[0031] In step S05, the concatenation calculation unit 14 concatenates the image feature a and the three-dimensional shape feature b to generate a concatenated feature c. Methods for concatenating features include a simple matrix concatenation method, an element sum or element product of matrices, and a bilinear model method. <Step S06>

[0032] In step S06, the image coordinate decoding unit 15 generates the image coordinates d of the boundary by decoding the linked feature amount c. A HorizonNet decoder or the like may be used to decode the linked feature amount c. <Step S07>

[0033] In step S07, the 3D reconstruction unit 16 performs 3D reconstruction based on the decoded image coordinates d of the boundary. The 3D reconstruction unit 16 performs processing such as determining wall surfaces by principal component analysis based on the Manhattan World Assumption. The Manhattan World Assumption is a hypothesis that artificial objects such as ceilings and walls in a 3D space have three mutually orthogonal dominant axes, and the surfaces of the ceilings, walls, etc. that constitute the artificial objects are arranged perpendicular or parallel to the three axes.

[0034] As described above, in order to achieve the object of the present invention of realizing highly accurate boundary detection even for indoor spaces where it is difficult to obtain sufficient image features a, a panoramic image 21 of the indoor space and a drawing of the indoor space are used in combination.

[0035] In the present disclosure, what has been implemented to realize highly accurate boundary detection in conjunction with the drawings is (i) preparing, as input data, three-dimensional coordinates 22 that are interpolated between corners of the subject of the panoramic image 21 shown in the drawings, (ii) adding a shape coding unit 13 that extracts three-dimensional shape feature amount (point cloud feature amount) b from the three-dimensional coordinates 22, and (iii) performing boundary estimation that takes into account three-dimensional information by decoding feature amount c that is a combination of image feature amount a and three-dimensional shape feature amount b. By implementing the above, boundary detection that takes into account three-dimensional shape feature amount b in addition to image feature amount a becomes possible, and the accuracy of three-dimensional reconstruction is improved.

[0036] In addition to the panoramic image 21, by utilizing 3D information described in a drawing or the like, boundary detection that is more in line with the drawing information becomes possible. Furthermore, if highly accurate boundary detection can be achieved, highly accurate 3D reconstruction becomes possible. Furthermore, highly accurate 3D reconstruction can reduce image reprojection errors, leading to improved visibility.

[0037] Furthermore, if an inspection image of a structure is used for the panoramic image 21, three-dimensional data including deterioration information can be obtained, making it possible to perform structural calculations and quantitatively evaluate the soundness of the structure.

[0038] Therefore, according to the 3D image reconstruction device 1 of the present disclosure, even for spaces where boundary detection from image feature a is difficult, such as when a large number of objects exist in an indoor space, highly accurate boundary detection can be achieved by combining the panoramic image 21 and 3D coordinates 22 such as a point cloud, thereby enabling highly accurate 3D reconstruction.

[0039] More specifically, according to the 3D image reconstruction device 1 of the present disclosure, (i) while in conventional 3D reconstruction techniques from panoramic images the reconstruction accuracy depended on image features, in the present disclosure, by combining a panoramic image with 3D coordinates such as a point cloud, it is possible to achieve high-precision boundary detection even in spaces where it is difficult to obtain sufficient image features, (ii) it is possible to detect boundaries between walls with high precision without requiring the camera position and orientation at the time the panoramic image is acquired, and (iii) it is possible to reconstruct 3D images with high precision even in indoor spaces where the walls are obstructed by objects contained therein.

[0040] The input unit 11, image coding unit 12, shape coding unit 13, concatenation calculation unit 14, image coordinate decoding unit 15 and 3D reconstruction unit 16 in the above 3D reconstruction device 1 constitute a part of a control device (controller). The control device may be constituted by dedicated hardware such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array), may be constituted by a processor, or may be constituted by including both.

[0041] A computer capable of executing program instructions can be used to function the above-mentioned three-dimensional reconstruction device 1. FIG. 4 is a block diagram showing a schematic configuration of a computer functioning as the three-dimensional reconstruction device. Here, the computer functioning as the three-dimensional reconstruction device 1 may be a general-purpose computer, a dedicated computer, a workstation, a PC (Personal Computer), an electronic notepad, or the like. The program instructions may be program code, code segments, or the like for performing necessary tasks.

[0042] 4, the computer 100 includes a processor 110, a memory unit including a ROM (Read Only Memory) 120, a RAM (Random Access Memory) 130, and a storage 140, an input unit 150, an output unit 160, and a communication interface (I / F) 170. Each component is connected to each other via a bus 180 so as to be able to communicate with each other.

[0043] The ROM 120 stores various programs and various data. The RAM 130 temporarily stores programs or data as a working area. The storage 140 is configured with a hard disk drive (HDD) or a solid state drive (SSD), and stores various programs including an operating system and various data. In the present disclosure, the programs related to the present disclosure are stored in the ROM 120 or the storage 140.

[0044] Specifically, the processor 110 is a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), a SoC (System on a Chip), etc., and may be composed of multiple processors of the same or different types. The processor 110 reads a program from the ROM 120 or the storage 140, and executes the program using the RAM 130 as a working area, thereby controlling each of the above components and performing various arithmetic processing. At least a part of the processing contents may be realized by hardware.

[0045] The program may be recorded in a recording medium readable by the three-dimensional reconstruction device 1. If such a recording medium is used, it is possible to install the program in the three-dimensional reconstruction device 1. Here, the recording medium on which the program is recorded may be a non-transitory recording medium. The non-transitory recording medium is not particularly limited, and may be, for example, a CD-ROM, a DVD-ROM, a USB (Universal Serial Bus) memory, or the like. In addition, the program may be in a form that is downloaded from an external device via a network.

[0046] The following supplementary notes are further disclosed regarding the above embodiment.

[0047] (Additional note 1) A 3D reconstruction device for performing 3D reconstruction of an indoor space, comprising: Memory, a controller coupled to the memory; Equipped with The controller: A panoramic image of a target indoor space and 3D coordinates interpolated to connect corners of the target depicted on a drawing of the indoor space are input; Extracting image features from the panoramic image; extracting a three-dimensional shape feature from the interpolated three-dimensional coordinates; generating a feature quantity by linking the image feature quantity and the three-dimensional shape feature quantity; generating image coordinates of the boundary by decoding the concatenated features; A 3D reconstruction device that performs 3D reconstruction of the indoor space based on image coordinates of the boundary. (Additional note 2) The controller: 2. The 3D reconstruction device according to claim 1, further comprising: a 3D reconstruction method for extracting image features from the panoramic image using HorizonNet. (Additional note 3) The controller: 3. The 3D reconstruction device according to claim 1 or 2, further comprising: a PointNet that extracts 3D shape features from the interpolated 3D coordinates. (Additional note 4) The controller: 4. The 3D reconstruction device according to claim 1, wherein the concatenated features are decoded using HorizonNet. (Additional note 5) A 3D reconstruction method for performing 3D reconstruction of an indoor space, comprising: Using a 3D reconstruction device, A step of inputting a panoramic image of a target indoor space and three-dimensional coordinates that are interpolated so as to connect corners of the target to be photographed that are shown on a drawing of the indoor space; extracting image features from the panoramic image; extracting a three-dimensional shape feature from the interpolated three-dimensional coordinates; generating a feature quantity by concatenating the image feature quantity and the three-dimensional shape feature quantity; generating image coordinates of the boundary by decoding the concatenated features; performing a 3D reconstruction of the indoor space based on the image coordinates of the boundary; A three-dimensional reconstruction method comprising: (Additional note 6) A non-transitory storage medium storing a program executable by a computer, the non-transitory storage medium storing the program causing the computer to function as a 3D reconstruction device described in any one of appendix 1 to 4.

[0048] Although the above-mentioned embodiment has been described as a representative example, it is obvious to those skilled in the art that many modifications and substitutions can be made within the spirit and scope of the present disclosure. Therefore, the present invention should not be interpreted as being limited by the above-mentioned embodiment, and various modifications or changes can be made without departing from the scope of the claims. For example, it is possible to combine multiple building blocks shown in the configuration diagram of the embodiment into one, or to divide one building block. [Explanation of symbols]

[0049] 1 3D reconstruction device 11 Input section 12 Image Encoding Unit 13 Shape encoding section 14 Concatenation operation section 15 Image coordinate decoding unit 16 3D reconstruction unit 21 Panoramic Images 22 3D coordinates (interpolated 3D coordinates) 23 3D reconstruction 100 Computers 110 Processor 120 ROM 130 RAM 140 Storage 150 Input section 160 Output section 170 Communication Interface (I / F) 180 Bus

Claims

1. A 3D reconstruction device for performing 3D reconstruction of an indoor space, an input unit for inputting a panoramic image of a target indoor space and three-dimensional coordinates that are interpolated so as to connect corners of the target to be photographed that are shown on a drawing of the indoor space; an image encoding unit that extracts image features from the panoramic image; a shape coding unit for extracting a three-dimensional shape feature from the interpolated three-dimensional coordinates; a concatenation calculation unit that generates a feature quantity by concatenating the image feature quantity and the three-dimensional shape feature quantity; an image coordinate decoding unit that generates image coordinates of a boundary by decoding the linked feature amount; a three-dimensional reconstruction unit that performs three-dimensional reconstruction of the indoor space based on image coordinates of the boundary; A three-dimensional reconstruction apparatus comprising:

2. The image encoding unit includes: The 3D reconstruction device according to claim 1 , wherein image features are extracted from the panoramic image using HorizonNet.

3. The shape encoding unit The 3D reconstruction device according to claim 1 , further comprising: a PointNet for extracting 3D shape features from the interpolated 3D coordinates.

4. The image coordinate decoding unit The 3D reconstruction device according to claim 1 , wherein the concatenated features are decoded using HorizonNet.

5. A 3D reconstruction method for performing 3D reconstruction of an indoor space, comprising: Using a 3D reconstruction device, A step of inputting a panoramic image of a target indoor space and three-dimensional coordinates that are interpolated so as to connect corners of the target to be photographed that are shown on a drawing of the indoor space; extracting image features from the panoramic image; extracting a three-dimensional shape feature from the interpolated three-dimensional coordinates; generating a feature quantity by linking the image feature quantity and the three-dimensional shape feature quantity; generating image coordinates of the boundary by decoding the concatenated features; performing a three-dimensional reconstruction of the indoor space based on the image coordinates of the boundary; A three-dimensional reconstruction method comprising:

6. A program for causing a computer to function as the three-dimensional reconstruction apparatus according to any one of claims 1 to 4.

Images