Learning device, three-dimensional reconstruction device, learning method, three-dimensional reconstruction method, and program

Abstract

A training device (1) according to the present disclosure trains a layout estimation model comprising an image feature extractor (11) that extracts features from a panoramic image, a feature processor (12) that has an encoder for performing global feature processing and a bidirectional LSTM for performing local feature processing, and receives input of the extracted features to generate an encoded vector for which an autocorrelation of the panoramic image is taken into account, and a feature decoder (13) that receives input of the encoded vector to generate an output vector indicating, for each column of the panoramic image, boundary positions between the ceiling and wall surfaces, boundary positions between the floor and the wall surfaces, and probabilities of existence of boundaries between the wall surfaces.

Description

[Technical Field]

[0001] The present disclosure relates to a learning device, a three-dimensional reconstruction device, a learning method, a three-dimensional reconstruction method, and a program. [Background technology]

[0002] Layout estimation, which estimates (reconstructs) the three-dimensional structure of an internal space from a panoramic image of the internal space of a structure, mainly consists of two processing units: a feature estimation unit and a three-dimensional reconstruction unit. The feature estimation unit inputs the panoramic image into a deep learning model, and detects the walls of the internal space from the panoramic image (see, for example, Non-Patent Documents 1 and 2). In addition, the three-dimensional reconstruction unit calculates the three-dimensional coordinates of each boundary, and estimates the three-dimensional structure of the internal space of the structure based on the calculated three-dimensional coordinates.

[0003] The feature estimation unit performs local feature processing, which processes the input by dividing it into parts, and global feature processing, which processes the input all at once. For example, LGT-Net, a layout estimation neural network that has achieved the highest accuracy, is configured to repeat twice in this order: a Swin Transformer (str) (see, for example, Non-Patent Document 3), which processes features locally, and a Transformer (tr) (see, for example, Non-Patent Document 4), which processes features globally. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Sun, C., et al., “HorizonNet: Learning Room Layout With 1D Representation and Pano Stretch Data Augmentation”, CVPR, 2019. [Non-patent document 2] Yang, ST., et al., “DuLa-Net: A Dual-Projection Network for Estimating Room Layouts from a Single RGB Panorama”, CVPR, 2019. [Non-patent document 3] Ze, L., et al., “Swin Transformer: Hierarchical Vision Transformer using Shifted Windows”, ICCV, 2021. [Non-patent document 4] Ashish, V., et al., “Attention Is All You Need”, CL, 2017. Summary of the Invention [Problem to be solved by the invention]

[0005] However, conventional layout estimation methods have not been optimized for the combination of global feature processing and local feature processing, and have not been able to properly link information on global and local features that are located far apart, resulting in insufficient accuracy.

[0006] In view of the above circumstances, an object of the present disclosure is to provide a learning device, a three-dimensional reconstruction device, a learning method, a three-dimensional reconstruction method, and a program that are capable of improving estimation accuracy. [Means for solving the problem]

[0007] A learning device according to one embodiment is a learning device that learns a layout estimation model that estimates features of a panoramic image of the interior space of a structure, and that learns a layout estimation model that includes: an image feature extractor that extracts features from the panoramic image; a feature processor that has an encoder that performs global feature processing, inputting the features and generating an encoded vector that takes into account the autocorrelation of the panoramic image; and a feature decoder that has a bidirectional LSTM that performs local feature processing, inputting the encoded vector and generating, for each column of the panoramic image, an output vector that indicates the boundary positions between the ceiling and wall surfaces, the boundary positions between the floor and wall surfaces, and the probability of existence of boundaries between the wall surfaces.

[0008] A three-dimensional reconstruction device according to one embodiment is a three-dimensional reconstruction device that reconstructs three-dimensional data from a panoramic image, and includes: a feature estimation unit that inputs the panoramic image into a layout estimation model learned by the learning device and estimates the output vector; a three-dimensional coordinate calculation unit that calculates the three-dimensional coordinates of each coordinate point that constitutes a boundary based on the output vector; and a post-processing unit that three-dimensionally reconstructs the internal space of a structure based on the three-dimensional coordinates and generates image-mapped three-dimensional data.

[0009] A learning method according to one embodiment is a learning method for using a learning device to learn a layout estimation model that estimates features of a panoramic image of the interior space of a structure, and includes the steps of: extracting features from the panoramic image so as to reduce an error between an output vector output by the layout estimation model and a vector generated from true values ​​of training data; having an encoder that performs global feature processing, inputting the features, and generating an encoded vector that takes into account the autocorrelation of the panoramic image; and having a bidirectional LSTM that performs local feature processing, inputting the encoded vector, and generating an output vector that indicates, for each column of the panoramic image, the boundary positions between the ceiling and wall surfaces, the boundary positions between the floor and wall surfaces, and the probability of the existence of boundaries between the wall surfaces.

[0010] A three-dimensional reconstruction method according to one embodiment is a three-dimensional reconstruction method for reconstructing three-dimensional data from a panoramic image using a three-dimensional reconstruction device, and includes the steps of inputting a panoramic image into a layout estimation model learned by the above-described learning method and estimating the output vector, calculating the three-dimensional coordinates of each coordinate point that constitutes a boundary based on the output vector, and three-dimensionally reconstructing the internal space of a structure based on the three-dimensional coordinates and generating image-mapped three-dimensional data.

[0011] A program according to one embodiment causes a computer to function as the learning device.

[0012] A program according to one embodiment causes a computer to function as the three-dimensional reconstruction apparatus. [Effects of the Invention]

[0013] According to the present disclosure, it is possible to estimate the layout of a room with high accuracy from an image of the room. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a block diagram illustrating an example of the configuration of a learning device according to an embodiment. [Figure 2] FIG. 1 is a diagram illustrating an example of a panoramic image. [Figure 3] FIG. 10 is a diagram illustrating an example of an output vector. [Figure 4] FIG. 1 is a diagram illustrating an overview of a neural network of a learning device according to an embodiment. [Figure 5] FIG. 1 illustrates an overview of a neural network for a multi-head attention layer according to one embodiment. [Figure 6] 10 is a flowchart illustrating an example of an operation of a learning device according to an embodiment. [Figure 7] FIG. 1 is a diagram illustrating an example of the configuration of a three-dimensional reconstruction apparatus according to an embodiment. [Figure 8] FIG. 10 is a diagram illustrating an example of three-dimensional data. [Figure 9] 10 is a flowchart illustrating an example of an operation of the three-dimensional reconstruction apparatus according to an embodiment. [Figure 10] FIG. 1 is a diagram illustrating an example of the hardware configuration of a learning device and a three-dimensional reconstruction device according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, one embodiment will be described in detail with reference to the drawings.

[0016] LGT-Net performs local feature processing before global feature processing. However, changing the order of feature processing showed that the evaluation index (3D IoU) increased by 2 points, improving accuracy, as shown in the table below. Here, 3D IoU (Intersection over Union) is an index that indicates the degree to which two spaces overlap, and is expressed as the ratio of the intersection between the estimated 3D space and the correct 3D space to the union of the 3D space estimated by the estimation model (estimated 3D space) and the actual 3D space (correct 3D space). The closer 3D IoU is to 100%, the higher the accuracy. [Table 1]

[0017] In the evaluation of Table 1, we used Matterport3D, a large-scale RGB-D dataset containing panoramic images (see, for example, Reference 1). [References 1] Chang, A., et.al., “Matterport3D: Learning from RGB-D Data in Indoor Environments”, 3DV, 2017.

[0018] In room layout estimation, it is necessary to link positional information on the boundaries of wall surfaces. Therefore, it is believed that first acquiring global features and then acquiring local features allows for better linking of information that is separated by positions, improving the evaluation index. Therefore, in the present invention, local feature processing is performed after global feature processing, thereby enabling effective feature processing in layout estimation.

[0019] <Learning device> First, a learning device according to an embodiment of the present disclosure will be described.

[0020] 1 is a diagram illustrating an example configuration of a learning device 1 according to an embodiment of the present disclosure. The learning device 1 learns a layout estimation model that estimates feature quantities of a panoramic image of the interior space (indoor space) of a structure. A panoramic image is, for example, an image obtained by capturing the interior space of a structure using a camera or the like at angles ranging from -90° to +90° in the up-down direction along the vertical direction and at angles ranging from 0° to 360° in the left-right direction along the horizontal direction.

[0021] 1, the layout estimation model includes a preprocessing unit 10, an image feature extractor 11, a feature processor 12, and a feature decoder 13. The learning device 1 learns this layout estimation model. The preprocessing unit 10, the image feature extractor 11, the feature processor 12, and the feature decoder 13 may form a control unit (controller), which may be configured with dedicated hardware such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array), or may be configured with a processor, or may be configured with both.

[0022] As a preprocessing step, the preprocessing unit 10 aligns the zenith directions of the input panoramic images. The preprocessing unit 10 may use a method such as that proposed in Reference 2 below. The learning device 1 can easily perform learning by performing preprocessing using the preprocessing unit 10. The preprocessing unit 10 outputs the preprocessed panoramic images to the image feature extractor 11. However, the preprocessing unit 10 is not a required component, and the learning device 1 does not necessarily have to include the preprocessing unit 10. [Reference 2] Zou, C., et al., “Layoutnet: Reconstructing the 3d room layout from a single rgb image”, CVPR, 2018

[0023] The image feature extractor 11 extracts features from the input panoramic image without considering autocorrelation. The image feature extractor 11 extracts features using a neural network such as ResNet (Residual Neural Networks). The image feature extractor 11 may use the feature extractor disclosed in Reference 3 below. [Reference 3] Sun, C., et al., “HoHoNet: 360 Indoor Holistic Understanding with Latent Horizontal Features”, CVPR, 2020

[0024] The feature processor 12 receives the features extracted by the image feature extractor 11 and generates an encoded vector that takes into account the autocorrelation of the panoramic image. By taking into account the autocorrelation of the panoramic image, the feature processor 12 can complement the features of occlusion regions (portions of the panoramic image that are obscured by other objects and not visible). The feature processor 12 outputs the generated encoded vector to the feature decoder 13.

[0025] The feature decoder 13 generates an output vector indicating the boundary position in the panoramic image based on the encoded vector input from the feature processor 12 and outputs the output vector to the outside of the learning device 1.

[0026] The learning device 1 repeatedly performs the above-described process to train the neural network so as to reduce the error between the output vector output by the layout estimation model and a vector generated from the true values ​​of pre-prepared training data, and generates a trained layout estimation model.

[0027] 2 is a diagram showing an example of a panoramic image. In this embodiment, the input to the learning device 1 is a single panoramic image having dimensions of 3×512×1024 (channels×height×width).

[0028] The size of the output vector output by the learning device 1 is 3×1×1024. The output vector is a vector obtained by dividing the boundary position (y c ) and the boundary position of the floor and wall (y f ) and the probability of existence of a boundary between walls (i.e., a corner) (y w ) y w can be a binary vector with labels of 0 or 1, but the number of 1s will be sparse (for example, 4 out of 1024), so y w (i)=c dx Here, i is the i-th column, dx is the distance from the i-th column to the nearest column where a boundary between the wall surfaces exists, and c is a constant (for example, c=0.96).

[0029] FIG. 3 is a diagram showing an example of output vectors when the panoramic image shown in FIG. 2 is input to the learning device 1. In order to make it easier to understand the correspondence between the panoramic image and the output vectors, y c and y f The position indicated by y is displayed on top of the panoramic image. w (i). In Figure 3, y w To make (i) easier to see, it is stretched in the vertical direction.

[0030] FIG. 4 is a diagram illustrating an overview of the neural network of the learning device 1 according to an embodiment of the present disclosure.

[0031] The image feature extractor 11 has block 1, block 2, block 3, and block 4. The size of the data generated by the subsequent block is half the size of the data generated by the previous block. The data generated by each block is transformed by a conversion process (convs), then reshaped to the same width and height by a reshape process (reshape), and concatenated by a concatenation process (concat). The image feature extractor 11 finally generates features (feature vectors) with a size of 512 x 1 x 256. The image feature extractor 11 outputs the generated feature vectors to the feature processor 12.

[0032] The feature processor 12 has N encoders 121 that perform global feature processing.

[0033] The encoder 121 has an attention mechanism. In the example shown in Fig. 4, the encoder 121 has a multi-head attention layer (Multi-Head Attention) 151, a first residual connection and normalization layer (Add & Norm) 152, a feed-forward layer (Feed-Forward) 153, and a second residual connection and normalization layer (Add & Norm) 154.

[0034] The multi-head attention layer 151 receives a query vector (hereinafter referred to as a "Q vector"), a key vector (hereinafter referred to as a "K vector"), and a value vector (hereinafter referred to as a "V vector"). The Q vector is obtained by multiplying the feature vector input to the encoder 121 by a weight matrix Wq. The K vector is obtained by multiplying the feature vector input to the encoder 121 by a weight matrix Wk. The V vector is obtained by multiplying the feature vector input to the encoder 121 by a weight matrix Wv. The learning device 1 learns the weight matrices Wq, Wk, and Wv.

[0035] Fig. 5 is a diagram showing an overview of the neural network of the multi-head attention layer 151. As shown in Fig. 5, the multi-head attention layer 151 includes h scaled dot-product attention layers 155, a linear layer, and a contact layer. The scaled dot-product attention layer 155 receives h divided Q vectors, K vectors, and V vectors.

[0036] The scaled dot-product attention layer 155 calculates the scaled dot-product attention using the formula (1) that uses the softmax function. In formula (1), d k is the number of dimensions of the Q vector and the K vector.

number

[0037] Referring again to FIG. 4, the multi-head attention layer 151 calculates the scaled dot-product attention by the scaled dot-product attention layer 155, and then calculates the multi-head attention by Equation (2). Here, i is expressed by equation (3), and W i Q is expressed by equation (4). In equation (4), d k is the dimension number of the Q vector and the K vector. v is the number of dimensions of the V vector. The multi-head attention layer 151 outputs the calculated multi-head attention to the first residual connection / normalization layer 152.

number

[0038] The first residual coupling and normalization layer 152 normalizes the vector input to the encoder 121 by adding the multi-head attention calculated by the multi-head attention layer 151. The first residual coupling and normalization layer 152 inputs the normalized vector to the feedforward layer 153.

[0039] The feedforward layer 153 is also called a Position-wise Feed-Forward Network. The feedforward layer 153 generates an output using an activation function such as a ReLU (Rectified Linear Unit) and the vector input from the first residual connection and normalization layer 152. The feedforward layer 153 generates an FFN(x) from the input vector x using equation (5). In equation (5), W1 and W2 are coefficients, and b1 and b2 are biases. The feedforward layer 153 outputs the generated FFN(x) to the second residual connection and normalization layer 154.

number

[0040] The second residual coupling and normalization layer 154 normalizes the vector output by the first residual coupling and normalization layer 152 by adding the FFN(x) generated by the feedforward layer 153. The second residual coupling and normalization layer 154 outputs the normalized vector to the bidirectional LSTM 131 as an encoded vector.

[0041] The encoder 121 repeats the above process N times. For example, N=6. The input and output dimensions of the encoder 121 are the same. That is, the size of the encoded vector is 512×1×256.

[0042] The feature decoder 13 includes a bidirectional LSTM 131 that performs local feature processing, a dropout layer 132, and a fully connected layer 133.

[0043] Bidirectional LSTM131 is a neural network that propagates the output of the intermediate layer in both forward and backward directions. LSTM (Long Short Term Memory) is a type of recurrent neural network (RNN) that learns long-term dependencies. Geometrically, any corner of a room can be roughly estimated from the positions of other corners. LSTM stores prediction information for other areas in the cell state, allowing it to accurately predict areas occluded by other objects based on the geometric pattern of the entire room. In other words, bidirectional LSTM131 uses the functions of RNN to capture rough information and long-term dependencies. For more information on LSTM, see the references below. [Reference 4] Hochreiter, S., et al., “Long Short-Term Memory, Neural Computation”, 1997

[0044] The bidirectional LSTM131 uses y' corresponding to the feature vector. c ,y' f ,y' w Predict y' column by column or by several columns (e.g., every four columns). c ,y' f ,y' w Since is related to both its left and right neighbors, the accuracy of feature estimation can be improved by incorporating information from both sides using a bidirectional LSTM.

[0045] The feature decoder 13 estimates the image coordinates of the wall boundary. The learning device 1 changes the output size in the fully connected layer 133 to obtain the final output, which is the probability of boundary existence and the image coordinates (height only) of the upper and lower wall boundaries for the input panoramic image width of 1024 pixels. Overfitting is thought to be particularly likely to occur when learning images with occlusions. Therefore, the feature decoder 13 is provided with a dropout layer 132, which makes it possible to suppress overfitting.

[0046] Next, the operation of the learning device 1 according to this embodiment will be described. Fig. 6 is a flowchart showing an example of the operation of the learning device 1.

[0047] In step S11, the pre-processing unit 10 aligns the zenith directions of the panoramic images.

[0048] In step S12, the image feature extractor 11 extracts a feature amount indicating the boundary position in the panoramic image.

[0049] In step S13, the feature processor 12 generates feature values ​​that take into account the correlation of the layout.

[0050] In step S14, the feature decoder 13 generates an output vector indicating the boundary position in the panoramic image.

[0051] In step S15, the learning device 1 determines whether to terminate learning based on the error between the output vector output by the generated layout estimation model and the vector generated from the true values ​​of the training data. If the determination result in step S15 is No, the processes from step S11 to step S14 are repeated.

[0052] As described above, the learning device 1 performs global feature processing by the encoder 121 in the feature processor 12, followed by local feature processing by the bidirectional LSTM 131. This makes it possible to link positional information of boundary parts (corners) of wall surfaces in room layout estimation, thereby improving the estimation accuracy of features.

[0053] <Three-dimensional reconstruction device> Next, a three-dimensional reconstruction apparatus according to an embodiment of the present disclosure will be described.

[0054] 7 is a diagram illustrating an example configuration of a three-dimensional reconstruction device 2 according to an embodiment of the present disclosure. The three-dimensional reconstruction device 2 reconstructs a three-dimensional image (three-dimensional data) from a two-dimensional panoramic image.

[0055] 7, the three-dimensional reconstruction device 2 includes a pre-processing unit 20, a feature amount estimation unit 21, a three-dimensional coordinate calculation unit 22, and a post-processing unit 23. The pre-processing unit 20, the feature amount estimation unit 21, the three-dimensional coordinate calculation unit 22, and the post-processing unit 23 may constitute a control unit (controller), and the control unit may be constituted by dedicated hardware such as an ASIC or an FPGA, or may be constituted by a processor, or may be constituted by including both.

[0056] Similar to the preprocessing unit 10 of the learning device 1, the preprocessing unit 20 aligns the zenith direction of the input panoramic image. The preprocessing unit 20 outputs the preprocessed panoramic image to the feature estimation unit 21. However, similar to the preprocessing unit 10, the preprocessing unit 20 is not an essential component, and the 3D reconstruction device 2 does not necessarily have to include the preprocessing unit 20.

[0057] The feature estimation unit 21 acquires a learned layout estimation model that has been learned by the learning device 1. The feature estimation unit 21 inputs a panoramic image to the layout estimation model, and estimates an output vector that indicates the boundary position in the panoramic image. The output vector is calculated based on the boundary position (y c ) and the boundary position of the floor and wall (y f ) and the probability of existence of a boundary between walls (y w The feature estimation unit 21 outputs the estimated output vector to the three-dimensional coordinate calculation unit 22.

[0058] The three-dimensional coordinate calculation unit 22 calculates the three-dimensional coordinates of each coordinate point constituting the boundary based on the output vector input from the feature estimation unit 21. The three-dimensional coordinate calculation unit 22 outputs the calculated three-dimensional coordinates to the post-processing unit 23. The three-dimensional coordinate calculation unit 22 calculates the three-dimensional coordinates using, for example, the algorithm proposed in Non-Patent Document 1.

[0059] The post-processing unit 23 three-dimensionally reconstructs the internal space of the structure and generates image-mapped three-dimensional data based on the three-dimensional coordinates input from the three-dimensional coordinate calculation unit 22. The post-processing unit 23 outputs the three-dimensional data by displaying it on a display device (not shown), for example.

[0060] When the panoramic image shown in FIG. 2 is input, the three-dimensional reconstruction device 2 generates three-dimensional data shown in FIG. 8, for example.

[0061] Next, the operation of the three-dimensional reconstruction device 2 according to this embodiment will be described.

[0062] In step S21, the pre-processing unit 20 aligns the zenith directions of the panoramic images.

[0063] In step S22, the feature estimation unit 21 estimates an output vector indicating the boundary position in the panoramic image.

[0064] In step S23, the three-dimensional coordinate calculation unit 22 calculates the three-dimensional coordinates of each of the coordinate points that form the boundary.

[0065] In step S24, the post-processing unit 23 generates and outputs three-dimensional data.

[0066] As described above, the three-dimensional reconstruction device 2 estimates the output vector using the layout estimation model learned by the learning device 1. This makes it possible to associate positional information of boundary parts (corners) of wall surfaces in room layout estimation, thereby improving the estimation accuracy of feature quantities.

[0067] Furthermore, the 3D reconstruction device 2 performs 3D reconstruction using highly accurate output vectors, enabling highly accurate image mapping. Accordingly, for example, by reflecting in the 3D image any deterioration of the surfaces defining the interior space of the structure, detected by an operator during inspection, the accuracy of modeling the structure, which indicates the deteriorated areas, is improved. Furthermore, by reflecting in the 3D image any furniture placed in the interior space of the structure, wallpaper pasted on the surfaces defining the interior space, etc., the accuracy of modeling the structure is improved.

[0068] <Programs and recording media> The above-described learning device 1 and three-dimensional reconstruction device 2 can be realized by a computer. Also, a program for causing each of the learning device 1 and the three-dimensional reconstruction device 2 to function may be provided. Also, the program may be stored in a storage medium or provided via a network.

[0069] 10 is a block diagram showing a schematic configuration of a computer 300 functioning as the learning device 1. The same applies to a computer functioning as the 3D reconstruction device 2. Here, the computer 300 may be a general-purpose computer, a dedicated computer, a workstation, a PC (Personal Computer), an electronic notepad, or the like. The program instructions may be program code, code segments, or the like for performing the necessary tasks.

[0070] 10, a computer 300 includes a processor 301, a read-only memory (ROM) 302, a random access memory (RAM) 303, a storage 304, an input unit 305, an output unit 306, and a communication interface (I / F) 307. Each component is connected to each other so as to be able to communicate with each other via a bus 308. The processor 301 is specifically a central processing unit (CPU), a micro processing unit (MPU), a graphics processing unit (GPU), a digital signal processor (DSP), a system on a chip (SoC), or the like, and may be configured by a plurality of processors of the same type or different types.

[0071] The processor 301 controls each component and executes various types of arithmetic processing. That is, the processor 301 reads a program from the ROM 302 or the storage 304 and executes the program using the RAM 303 as a work area. The processor 301 controls each component and executes various types of arithmetic processing in accordance with the program stored in the ROM 302 or the storage 304. In the above-described embodiment, the program according to the present disclosure is stored in the ROM 302 or the storage 304.

[0072] The program may be stored in a storage medium readable by the computer 300. Using such a storage medium, the program can be installed in the computer 300. Here, the storage medium on which the program is stored may be a non-transitory storage medium. The non-transitory storage medium is not particularly limited, and may be, for example, a CD-ROM, a DVD-ROM, or a USB (Universal Serial Bus) memory. Furthermore, the program may be downloaded from an external device via a network.

[0073] The ROM 302 stores various programs and various data. The RAM 303 temporarily stores programs or data as a working area. The storage 304 is configured with an HDD (Hard Disk Drive) or SSD (Solid State Drive) and stores various programs including the operating system and various data.

[0074] The input unit 305 includes one or more input interfaces that receive input operations from a user and acquire information based on the user operations. For example, the input unit 305 may be, but is not limited to, a pointing device, a keyboard, a mouse, etc.

[0075] The output unit 306 includes one or more output interfaces that output information. For example, the output unit 306 is a display that outputs information as a video or a speaker that outputs information as an audio, but is not limited to these. If the output unit 306 is a touch panel display, it also functions as the input unit 305.

[0076] The communication interface 307 is an interface for communicating with an external device.

[0077] The following additional notes are provided regarding the above-described embodiments.

[0078] [Additional note 1] A learning device that learns a layout estimation model that estimates feature quantities of a panoramic image of an interior space of a structure, comprising: a controller, the controller comprising: extracting features from the panoramic image; an encoder for performing global feature processing, and generating an encoded vector from the feature taking into account autocorrelation of the panoramic image; A learning device that has a bidirectional LSTM that performs local feature processing, and generates, from the encoded vector, an output vector that indicates the boundary positions between the ceiling and wall surfaces, the boundary positions between the floor and wall surfaces, and the probability of the existence of boundaries between the wall surfaces, for each column of the panoramic images. [Additional note 2] 2. The learning device according to claim 1, wherein the encoder has an attention mechanism. [Additional note 3] 2. The learning device according to claim 1, wherein the controller has a dropout layer and a fully connected layer, and generates the output vector from the encoded vector. [Additional note 4] A three-dimensional reconstruction device that reconstructs three-dimensional data from a panoramic image, comprising: a controller, the controller comprising: inputting a panoramic image into a layout estimation model trained by the learning device according to any one of Supplementary Items 1 to 3, and estimating the output vector; Calculating three-dimensional coordinates of each coordinate point constituting the boundary based on the output vector; A three-dimensional reconstruction device that three-dimensionally reconstructs the internal space of the structure based on the three-dimensional coordinates and generates image-mapped three-dimensional data. [Additional note 5] A learning method for learning a layout estimation model that estimates feature quantities of a panoramic image of an interior space of a structure, using a learning device, comprising: To minimize the error between the output vectors of the layout estimation model and the vectors generated from the true values ​​of the training data, extracting features from the panoramic image; an encoder for performing global feature processing, inputting the feature and generating an encoded vector taking into account the autocorrelation of the panoramic image; a step of generating an output vector indicating the boundary positions between the ceiling and the wall surfaces, the boundary positions between the floor and the wall surfaces, and the existence probability of the boundary between the wall surfaces, for each row of the panoramic images, the output vector including a bidirectional LSTM that performs local feature processing and inputting the encoded vector; A learning method that involves repeatedly executing the above steps. [Additional note 6] A three-dimensional reconstruction method for reconstructing three-dimensional data from a panoramic image by a three-dimensional reconstruction device, comprising: a step of inputting a panoramic image into a layout estimation model trained by the training method described in Supplementary Item 5, and estimating the output vector; calculating three-dimensional coordinates of each coordinate point constituting the boundary based on the output vector; a step of three-dimensionally reconstructing the internal space of the structure based on the three-dimensional coordinates and generating image-mapped three-dimensional data; A three-dimensional reconstruction method is performed. [Additional note 7] A non-transitory storage medium storing a program executable by a computer, A non-transitory storage medium storing a program that causes the computer to function as the learning device described in any one of appendixes 1 to 3. [Additional note 8] A non-transitory storage medium storing a program executable by a computer, A non-transitory storage medium storing a program that causes the computer to function as the three-dimensional reconstruction device described in appended claim 4.

[0079] Although the above-described embodiments have been described as typical examples, it will be apparent to those skilled in the art that many modifications and substitutions can be made within the spirit and scope of the present disclosure. Therefore, the present invention should not be interpreted as being limited by the above-described embodiments, and various modifications or alterations are possible without departing from the scope of the claims. For example, multiple building blocks shown in the block diagrams of the embodiments can be combined into one, or one building block can be divided. [Explanation of symbols]

[0080] 1 Learning device 2 Three-dimensional reconstruction device 10 Pretreatment section 11 Image Feature Extractor 12 Feature processor 13 Feature Decoder 20 Pretreatment section 21 Feature estimation unit 22 3D coordinate calculation section 23 Post-processing section 121 Encoder 131 Bidirectional LSTM 132 Dropout Layer 133 Fully connected layer 151 Multi-head Attention Layer 152 1st residual combination / normalization layer 153 Feedforward Layer 154 Second residual combination / normalization layer 155 Scaled Inner Product Attention Layer 300 Computers 301 processor 302 ROM 303 RAM 304 Storage 305 Input section 306 Output section 307 Communication Interface 308 Bus

Claims

1. A learning device that learns a layout estimation model that estimates feature quantities of a panoramic image of an interior space of a structure, comprising: an image feature extractor that extracts features from the panoramic image; a feature processor having an encoder for performing global feature processing, the feature processor receiving the feature and generating an encoded vector taking into account the autocorrelation of the panoramic image; a feature decoder having a bidirectional LSTM for performing local feature processing, which receives the encoded vector and generates, for each row of the panoramic images, an output vector indicating the boundary positions between the ceiling and the wall surfaces, the boundary positions between the floor and the wall surfaces, and the probability of existence of boundaries between the wall surfaces; A learning device for learning a layout estimation model comprising:

2. The learning device according to claim 1 , wherein the encoder comprises an attention mechanism.

3. The learning device according to claim 1 , wherein the feature decoder includes a dropout layer and a fully connected layer.

4. A three-dimensional reconstruction device that reconstructs three-dimensional data from a panoramic image, comprising: a feature estimation unit that inputs a panoramic image to a layout estimation model trained by the learning device according to claim 1 and estimates the output vector; a three-dimensional coordinate calculation unit that calculates three-dimensional coordinates of each coordinate point that constitutes the boundary based on the output vector; a post-processing unit that three-dimensionally reconstructs the internal space of the structure based on the three-dimensional coordinates and generates image-mapped three-dimensional data; A three-dimensional reconstruction device comprising:

5. A learning method for learning a layout estimation model that estimates feature quantities of a panoramic image of an interior space of a structure, using a learning device, comprising: To minimize the error between the output vectors of the layout estimation model and the vectors generated from the true values ​​of the training data, extracting features from the panoramic image; an encoder for performing global feature processing, inputting the feature and generating an encoded vector taking into account the autocorrelation of the panoramic image; a step of generating an output vector indicating the boundary positions between the ceiling and the wall surfaces, the boundary positions between the floor and the wall surfaces, and the probability of existence of the boundary between the wall surfaces, for each row of the panoramic images, the output vector including a bidirectional LSTM that performs local feature processing and receives the encoded vector; A learning method that involves repeatedly executing the above steps.

6. A three-dimensional reconstruction method for reconstructing three-dimensional data from a panoramic image by a three-dimensional reconstruction device, comprising: a step of inputting a panoramic image into a layout estimation model trained by the training method according to claim 5 and estimating the output vector; calculating three-dimensional coordinates of each coordinate point constituting the boundary based on the output vector; a step of three-dimensionally reconstructing the internal space of the structure based on the three-dimensional coordinates and generating image-mapped three-dimensional data; A three-dimensional reconstruction method is performed.

7. A program for causing a computer to function as the learning device according to claim 1.

8. A program for causing a computer to function as the three-dimensional reconstruction apparatus according to claim 4.

Images