Learning device, three-dimensional reconstruction device, learning method, three-dimensional reconstruction method, and program

Abstract

A learning device (100) according to the present disclosure includes a teacher model estimating unit (13) that outputs processed features upon receiving input of image features extracted from a panorama image (D1) and shape features extracted from shape information (D2), using a teacher model, a student model estimating unit (15) that outputs processed features upon receiving input of the image features extracted from the panorama image (D1), using a student model, and a student model learning unit (16) that performs learning with the student model as an estimation model, such that loss (Lfeature) indicating the difference between a distribution of processed features output using the teacher model and a distribution of processed features output using the student model assumes a minimal value in verification data.

Description

[Technical Field]

[0001] This disclosure relates to a learning device, a three-dimensional reconstruction device, a learning method, a three-dimensional reconstruction method, and a program. [Background technology]

[0002] Conventionally, it is known that a deep learning estimation model (layout estimation network) is used to output features extracted from panoramic images obtained by photographing the interior space of a structure, and these features are then used to reconstruct the interior space of the structure in three dimensions (Non-Patent Documents 1 and 2).

[0003] Furthermore, it is known that by inputting not only image features but also shape features, it is possible to reconstruct various objects (such as cars, bicycles, airplanes, and chairs, as in the example in Non-Patent Document 3) in 3D with high accuracy (Non-Patent Documents 3 and 4). [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Sun, C., et al., “HorizonNet: Learning Room Layout With 1D Representation and Pano Stretch Data Augmentation”, CVPR, 2019. [Non-Patent Document 2] Yang, ST., et al., “DuLa-Net: A Dual-Projection Network for Estimating Room Layouts from a Single RGB Panorama”, CVPR, 2019. [Non-Patent Document 3] Yang, X., et al. Pose from Shape: Deep Pose Estimation for Arbitrary 3D Objects, BMVC, 2019. [Non-Patent Document 4] Li, Y., et al. DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detection, CVPR, 2022. [Overview of the Initiative] [Problems that the invention aims to solve]

[0005] However, the estimation model described above reconstructs the layout of the internal space of a structure in three dimensions by inputting image features extracted from panoramic images. Therefore, due to the nature of the layout estimation model, it was not possible to input information about the internal space layout from the beginning and perform three-dimensional reconstruction. Consequently, the estimation model described above had difficulty reconstructing the internal space of a structure in three dimensions with high accuracy compared to the case where image features and shape features were input.

[0006] In view of these circumstances, the purpose of this disclosure is to provide a learning device, a three-dimensional reconstruction device, a learning method, a three-dimensional reconstruction method, and a program that can reconstruct the internal space of a structure in three dimensions with high accuracy. [Means for solving the problem]

[0007] The learning device relating to this disclosure is a learning device for learning an estimation model for three-dimensional reconstruction of the interior space of a structure from a panoramic image of the interior space, and uses a teacher model that, when the panoramic image and shape information indicating the shape of the interior space are input, outputs processed feature quantities indicating features relating to the correlation of the planes defining the interior space, decodes the processed feature quantities and outputs a feature quantity vector indicating the position of the corners of the planes defining the interior space in the panoramic image, and when image features extracted from the panoramic image and shape features extracted from the shape information are input, the teacher model estimation unit outputs the processed feature quantities, and the panoramic image The system includes a student model estimation unit that outputs processed feature quantities showing characteristics related to the correlation of the planes defining the internal space when an image is input, decodes the processed feature quantities to output feature vectors showing the positions of the corners of the planes defining the internal space in the panoramic image, and outputs the processed feature quantities when image feature quantities extracted from the panoramic image are input, and a student model learning unit that calculates a loss showing the difference between the distribution of processed feature quantities output using the teacher model and the distribution of processed feature quantities output using the student model, and learns the student model as the estimation model so that the loss takes the minimum value with verification data.

[0008] The three-dimensional reconstruction apparatus according to this disclosure is a three-dimensional reconstruction apparatus that reconstructs the interior space of a structure in three dimensions from a panoramic image of the interior space of a structure, and comprises: an image input unit that receives the panoramic image as input; an estimation unit that inputs the panoramic image to an estimation model learned by the learning device described above and outputs a feature vector indicating the position of the corner of the plane defining the interior space; a three-dimensional coordinate calculation unit that calculates the three-dimensional coordinates of the points constituting the corner based on the feature vector; and a post-processing unit that reconstructs the interior space in three dimensions based on the three-dimensional coordinates.

[0009] Furthermore, the learning method relating to this disclosure is a learning method performed by a learning device that learns an estimation model for three-dimensional reconstruction of the interior space of a structure from a panoramic image of the interior space of the structure, and includes a training model in which, when the panoramic image and shape information indicating the shape of the interior space are input, processed feature quantities indicating features relating to the correlation of the planes defining the interior space are output, and the processed feature quantities are decoded to output a feature vector indicating the position of the corner of the planes defining the interior space in the panoramic image, and when image features extracted from the panoramic image and shape features extracted from the shape information are input, the processed feature quantities are output. The method includes the steps of: using a student model that, when a panoramic image is input, outputs processed feature quantities showing characteristics related to the correlation of the planes defining the internal space, decodes the processed feature quantities to output feature vectors showing the positions of the corners of the planes defining the internal space in the panoramic image, and when image features extracted from the panoramic image are input, outputs the processed feature quantities; and calculating a loss that shows the difference between the distribution of processed feature quantities output using the teacher model and the distribution of processed feature quantities output using the student model, and training the student model as the estimation model so that the loss takes the minimum value on the verification data.

[0010] Furthermore, the three-dimensional reconstruction method according to this disclosure is a three-dimensional reconstruction method performed by a three-dimensional reconstruction device that reconstructs the interior space of a structure in three dimensions from a panoramic image of the interior space of a structure, and includes the steps of: receiving the panoramic image as input; inputting the panoramic image to an estimation model learned by the learning method described above and outputting a feature vector indicating the position of the corner of the plane defining the interior space; calculating the three-dimensional coordinates of the points constituting the corner based on the feature vector; and reconstructing the interior space in three dimensions based on the three-dimensional coordinates.

[0011] Furthermore, the program relating to this disclosure causes the computer to operate as the learning device described above.

[0012] In addition, the program according to the present disclosure causes a computer to operate as the three-dimensional reconstruction device described above.

Advantages of the Invention

[0013] According to the present disclosure, a learning device, a three-dimensional reconstruction device, a learning method, a three-dimensional reconstruction method, and a program can reconstruct the internal space of a structure in three dimensions with high accuracy.

Brief Description of the Drawings

[0014] [Figure 1] It is a diagram showing an example of the configuration of a learning device according to an embodiment of the present disclosure. [Figure 2A] It is a diagram showing an example of a panoramic image. [Figure 2B] It is a diagram showing an example of a feature vector. [Figure 3] It is a diagram showing an example of the configuration of a three-dimensional reconstruction device according to an embodiment of the present disclosure. [Figure 4] It is a diagram showing a three-dimensional image of a structure constituted by the three-dimensional reconstruction device shown in FIG. 3. [Figure 5] It is a flowchart showing an example of the operation of the learning device shown in FIG. 1. [Figure 6] It is a flowchart showing an example of the operation of the three-dimensional reconstruction device shown in FIG. 3. [Figure 7] It is a hardware block diagram of the learning device.

Embodiments for Carrying Out the Invention

[0015] Hereinafter, the present embodiment will be described with appropriate reference to the drawings. In each drawing, the same or corresponding parts are denoted by the same reference numerals. In the description of the present embodiment, the description of the same or corresponding parts will be omitted or simplified as appropriate. The embodiments described below are examples of the configuration of the present disclosure, and the present invention is not limited to the following embodiments.

[0016] <Configuration of the Learning Device> The learning device 100 learns an estimation model for three-dimensional reconstruction of the interior space of a structure from panoramic images taken of the interior space. As shown in Figure 1, the learning device 100 comprises an image input unit 11 for the teacher model, a shape information input unit 12, a teacher model estimation unit 13, an image input unit 14 for the student model, a student model estimation unit 15, and a student model learning unit 16. The learning device 100 is a computer such as a server belonging to a cloud computing system or other computing system.

[0017] The teacher model image input unit 11, the shape information input unit 12, and the student model image input unit 14 are configured by an input interface. The input interface may be, for example, a physical key, a capacitive key, a pointing device, a touchscreen integrated with a display, or a microphone. The input interface may also include a communication interface. For example, standards such as Ethernet®, FDDI (Fiber Distributed Data Interface), and Wi-Fi® may be used for the communication interface. The teacher model estimation unit 13, the student model estimation unit 15, and the student model learning unit 16 are configured by a controller. The controller may be configured by dedicated hardware such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field-Programmable Gate Array), or by a processor, or by including both.

[0018] The image input unit 11 for the training model receives input of a panoramic image D1 taken of the interior space of a structure.

[0019] Figure 2A shows an example of a panoramic image. In this embodiment, the input to the learning device 100 is a single panoramic image with dimensions of 3 × 512 × 1024 (channels × height × width). As shown in Figure 2A, the panoramic image D1 is an image obtained by capturing the interior space of a structure with a camera or the like at angles from -90° to +90° in the vertical direction and from 0° to 360° in the horizontal direction. Therefore, the panoramic image D1 shows surfaces that define the interior space of the structure (e.g., ceiling, walls, floor).

[0020] The teacher model image input unit 11 may accept the input of the panoramic image D1 by any method. For example, the teacher model image input unit 11 may accept the input of the panoramic image D1 output from an external device equipped with a camera. The teacher model image input unit 11 may also accept the input of the panoramic image D1 by reading the panoramic image D1 from memory.

[0021] The shape information input unit 12 receives shape information D2 that indicates the shape of the internal space of the structure. For example, shape information D2 may be a drawing that represents the shape of the inside of the structure. The drawing may include a plan view, a side view, etc.

[0022] The teacher model estimation unit 13 outputs processed features when it receives image features extracted from the panoramic image D1 and shape features extracted from shape information D2 using the teacher model. The teacher model is a model that, when it receives the panoramic image D1 and shape information D2 which shows the shape of the interior space, outputs processed features that show features related to the correlation of the planes defining the interior space, decodes these processed features, and outputs a feature vector that shows the position of the corner of the plane that defines the interior space in the panoramic image D1.

[0023] The feature vector is a vector indicating the position of the corners of the surfaces that define the interior space. The surfaces that define the interior space include the ceiling, walls, and floor. The corners may be the boundaries between walls, between walls and floors, and between walls and ceilings. Specifically, the feature vector is a vector indicating the position of the boundary between a wall and a ceiling, the position of the boundary between a wall and a floor, and the position of the boundary between walls for each column of the panoramic image. The position of the boundary between walls may be a position where the probability of existence of the boundary between walls is higher than a predetermined value. Alternatively, the feature vector may be a vector indicating the position of the boundary between a wall and a ceiling, the position of the boundary between a wall and a floor, and the probability of existence of the boundary between walls for each column of the panoramic image.

[0024] The training model is obtained by creating a neural network that takes a panoramic image D1 and shape information D2 as inputs and outputs a feature vector, and then training this neural network. The training model loss L on the validation data is also obtained. T The model may be trained to take the minimum value. Teacher model loss L S This value represents the difference between the distribution of processed features output by the training model feature processor 133 (which will be explained in detail later), calculated using a loss function, and the distribution of the true values.

[0025] The teacher model estimation unit 13 includes a teacher model image feature extractor 131, a shape feature extractor 132, a teacher model feature processor 133, and a teacher model feature decoder 134.

[0026] The training model image feature extractor 131 extracts image features from the panoramic image D1, which is received as input by the training model image input unit 11. Image features are quantities that represent the characteristics of the image, such as edges and corners, within the structure shown in the panoramic image D1. The training model image feature extractor 131 can extract image features using any method; for example, it may use the known Horizon Net to extract image features.

[0027] The shape feature extractor 132 extracts shape features from shape information D2, which represents the shape of the internal space of the structure, and is received as input by the shape information input unit 12. The shape features can be, for example, three-dimensional coordinates representing the shape of the internal space of the structure. The shape feature extractor 132 can extract shape features using any method.

[0028] The training model feature processor 133, upon receiving image features extracted from the panoramic image D1 and shape features extracted from shape information, outputs processed features that show characteristics related to the correlation of planes defining the internal space. Specifically, the training model feature processor 133, upon receiving image features extracted by the training model image feature extractor 131 and shape features extracted by the shape feature extractor 132, outputs processed features.

[0029] More specifically, the training model feature processor 133 can obtain processed features by processing image features and shape features using the methods described in References 1 to 3. Reference 1: Yu, Z., et al. Multi-modal Factorized Bilinear Pooling with Co-attention Learning for Visual Question Answering, ICCV, 2017. Document 2: Yu, Z., et al. Deep Modular Co-Attention Networks for Visual Question Answering, CVPR, 2019. Document 3: Ben-younes, H., et al. MUTAN: Multimodal Tucker Fusion for Visual Question Answering, ICCV, 2017.

[0030] The training model feature decoder 134 outputs a feature vector indicating the position of the corner of the plane defining the interior space of the structure in the panoramic image D1 by decoding the processed features output by the training model feature processor 133. The training model feature decoder 134 may output the feature vector using, for example, Horizon Net.

[0031] The student model image input unit 14 receives a panoramic image D1 taken inside a structure. The panoramic image D1 received by the student model image input unit 14 is the same as the panoramic image D1 received by the teacher model image input unit 11.

[0032] The student model estimation unit 15 outputs processed features when image features extracted from the panoramic image D1 are input using the student model. The student model is a model that, when the panoramic image D1 is input, outputs processed features that show features related to the correlation of the planes defining the internal space, and decodes these processed features to output a feature vector that shows the position of the corners of the planes defining the internal space. The student model is obtained by training a neural network that takes the panoramic image D1 as input and outputs a feature vector. The student model loss L on the validation data S The student model loss L may be trained to take the minimum value. S This is the difference between the distribution of processed features output by the student model feature processor 152 (which will be explained in detail later), calculated using a loss function, and the distribution of the true values.

[0033] The student model estimation unit 15 includes a student model image feature extractor 151, a student model feature processor 152, and a student model feature decoder 153.

[0034] The student model image feature extractor 151 extracts image features from the panoramic image D1, which is received as input by the student model image input unit 14.

[0035] The student model feature processor 152 outputs processed features that show features related to the correlation of planes defining the internal space when image features extracted from the panoramic image D1 are input to the student model. Specifically, the student model feature processor 152 outputs processed features when image features extracted by the student model image feature extractor 151 are input to the student model. Specifically, the student model feature processor 152 can output processed features by processing the image features output by the student model feature processor 152, for example, using a Transformer as described in Reference 4. Reference 4: Ashish, V., et al. Attention Is All You Need, CL, 2017.

[0036] The student model feature decoder 153 estimates a feature vector indicating the position of the corner of the plane defining the interior space of the structure by decoding the processed features output by the student model feature processor 152. Similar to the teacher model feature decoder 134, the student model feature decoder 153 may output a feature vector indicating the position of the corner of the interior space in the panoramic image D1 based on the processed features, for example, using Horizon Net.

[0037] The student model learning unit 16 learns the student model described above. At this time, the student model learning unit 16 sets a loss function based on the distribution of processed features output using the teacher model by the teacher model feature processor 133 and the distribution of processed features output using the student model by the student model feature processor 152. Specifically, the student model learning unit 16 sets a loss function L which represents the difference between the distribution of processed features output using the teacher model and the distribution of processed features output using the student model. feature The student model learning unit 16 calculates the loss L on the validation data. feature The student model is trained as an estimation model so that it takes the minimum value.

[0038] The size of the feature vector output by the estimation model learned by the learning device 100 is 3×1×1024. As described above, in the configuration where the feature vector indicates the probability of the existence of the boundary between the wall surfaces, the feature vector includes, for each column of the panoramic image, the boundary position (y c ) between the ceiling and the wall, the boundary position (y f ) between the floor and the wall, and the probability of the existence of the boundary between the wall surfaces (i.e., the corner) (y w ). y w may be a binary vector with labels of 0 or 1. However, since the number of 1s becomes sparse (e.g., 4 out of 1024), y w (i)=c dx may be used. Here, i is the i-th column, dx is the distance from the i-th column to the nearest column where the boundary between the wall surfaces exists, and c is a constant (e.g., c = 0.96).

[0039] FIG. 2B is a diagram showing an example of the feature vector output when the panoramic image shown in FIG. 2A is input to the learning device 100. To clearly show the correspondence between the panoramic image and the feature vector, the positions indicated by y c and y f are superimposed and displayed on the panoramic image. Also, y w (i) is shown above the panoramic image. In FIG. 2B, for ease of viewing y w (i), it is stretched and displayed in the height direction.

[0040] <Configuration of the three-dimensional reconstruction device> The three-dimensional reconstruction device 300 reconstructs the internal space in three dimensions from the panoramic image D1 obtained by photographing the internal space of the structure. The three-dimensional reconstruction device 300 is a computer such as a server belonging to a cloud computing system or other computing systems.

[0041] As shown in FIG. 3, the three-dimensional reconstruction device 300 includes a student model storage unit 31, an image input unit 32, an estimation unit 33, a three-dimensional coordinate calculation unit 34, and a post-processing unit 35.

[0042] The student model storage unit 31 is comprised of memory. The memory may be an HDD (Hard Disk Drive), SSD (Solid State Drive), EEPROM (Electrically Erasable Programmable Read-Only Memory), ROM (Read Only Memory), or RAM (Random Access Memory). The image input unit 32 is comprised of an input interface. The estimation unit 33 and the post-processing unit 35 are comprised of a controller. The controller may be comprised of dedicated hardware such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field-Programmable Gate Array), or it may be comprised of a processor, or it may include both.

[0043] The student model memory unit 31 stores the student model (estimated model) learned by the learning device 100.

[0044] The image input unit 32 receives a panoramic image D1, which is a photograph of the interior space of the structure.

[0045] The estimation unit 33 inputs the panoramic image D1 to the student model trained by the learning device 100 described above, and estimates the feature vector by outputting a feature vector that indicates the position of the corner of the interior space in the panoramic image D1.

[0046] The three-dimensional coordinate calculation unit 34 calculates the three-dimensional coordinates of the points constituting the corner based on the feature vector estimated by the estimation unit 33.

[0047] The post-processing unit 35 reconstructs the internal space in three dimensions based on the three-dimensional coordinates calculated by the three-dimensional coordinate calculation unit 34. Specifically, the post-processing unit 35 generates a three-dimensional image D3 as shown in Figure 4 based on the three-dimensional coordinates. The post-processing unit 35 also outputs the three-dimensional image D3 to a display device or the like.

[0048] <Operation of the learning device> Here, the operation of the learning device 100 according to this embodiment will be described with reference to Figure 5. Figure 5 is a flowchart showing an example of the operation of the learning device 100 according to this embodiment. The operation of the learning device 100 described with reference to Figure 5 corresponds to an example of the method performed by the learning device 100 according to this embodiment, which learns an estimation model for three-dimensional reconstruction of the interior space from a panoramic image D1 taken of the interior space of a structure.

[0049] First, in steps S11 to S15, the teacher model estimation unit 13 uses the teacher model to output processed features when it receives image features extracted from the panoramic image D1 and shape features extracted from the shape information D2. As described above, when the panoramic image D1 and shape information D2 indicating the shape of the internal space are input, the model outputs processed features indicating the features related to the correlation of the planes defining the internal space, and decodes these processed features to output a feature vector indicating the position of the corners of the planes defining the internal space.

[0050] Specifically, in step S11, the image input unit 11 for the training model receives input of a panoramic image D1 taken of the interior space of the structure.

[0051] In step S12, the image feature extractor 131 for the training model extracts image features from the panoramic image D1.

[0052] In step S13, the shape information input unit 12 receives shape information D2, which indicates the shape of the internal space of the structure.

[0053] In step S14, the shape feature extractor 132 extracts shape features from the shape information D2.

[0054] In step S15, the teacher model estimation unit 13 receives image features extracted from the panoramic image D1 and shape features extracted from the shape information D2 as input and outputs processed features that show characteristics related to the correlation of the planes defining the internal space. Specifically, the teacher model feature processor 133 inputs the image features extracted in step 12 and the shape features extracted in step S14 into the teacher model and outputs processed features.

[0055] Next, in steps S16 to S18, the student model estimation unit 15 outputs processed features when image features extracted from the panoramic image D1 are input using the student model. As described above, when the panoramic image D1 is input, the model outputs processed features that show features related to the correlation of the planes defining the interior space, and decodes these processed features to output a feature vector that shows the position of the corners of the planes defining the interior space in the panoramic image D1.

[0056] Specifically, in step S16, the student model image input unit 14 receives input of a panoramic image D1 taken inside the structure.

[0057] In step S17, the student model image feature extractor 151 extracts image features from the panoramic image D1.

[0058] In step S18, when the student model estimation unit 15 receives image features extracted from the panoramic image D1 as input, it outputs processed features that show characteristics related to the correlation of the planes defining the internal space. Specifically, the student model feature processor 152 inputs the image features extracted in step S17 into the student model and outputs processed features.

[0059] Next, in steps S19 and S20, the student model learning unit 16 learns the student model.

[0060] Specifically, in step S19, the student model learning unit 16 calculates the loss L, which represents the difference between the distribution of processed features output using the teacher model in step S15 and the distribution of processed features output using the student model in step S18. feature Calculate.

[0061] In step S20, the student model learning unit 16 determines the loss L feature The student model is trained to take the minimum value on the validation data.

[0062] In the above description, the learning device 100 executed steps S16 to S18 after executing steps S11 to S15, but it is not limited to this. For example, the learning device 100 may execute steps S11 to S15 after executing steps S16 to S18. Alternatively, the learning device 100 may execute steps S16 to S18 at the same time as executing steps S11 to S15.

[0063] Furthermore, while the learning device 100 performed steps S13 and S14 after performing steps S11 and S12, it is not limited to this. For example, the learning device 100 may perform steps S11 and S12 after performing steps S13 and S14. Alternatively, the learning device 100 may perform steps S13 and S14 at the same time as performing steps S11 and S12.

[0064] <Operation of the three-dimensional reconstruction device> Here, the operation of the three-dimensional reconstruction apparatus 300 according to this embodiment will be described with reference to Figure 6. Figure 6 is a flowchart showing an example of the operation of the three-dimensional reconstruction apparatus 300 according to this embodiment. The operation of the three-dimensional reconstruction apparatus 300 described with reference to Figure 6 corresponds to an example of the method performed by the three-dimensional reconstruction apparatus 300 for three-dimensional reconstruction of the interior space from a panoramic image D1 taken of the interior space of a structure, according to this embodiment.

[0065] In step S31, the image input unit 32 receives an input of a panoramic image D1 taken inside the structure.

[0066] In step S32, the estimation unit 33 inputs the panoramic image D1 received in step 31 into the student model trained by the learning device 100 described above, and estimates the feature vector by outputting a feature vector indicating the position of the corner of the interior space in the panoramic image D1.

[0067] In step S33, the three-dimensional coordinate calculation unit 34 calculates the three-dimensional coordinates of the points constituting the corner based on the feature vector estimated by the estimation unit 33.

[0068] In step S34, the post-processing unit 35 reconstructs the internal space in three dimensions based on the three-dimensional coordinates estimated in step S34.

[0069] As described above, the learning device 100 in this embodiment is a learning device 100 that learns an estimation model for three-dimensional reconstruction of the interior space from a panoramic image D1 of the interior space of a structure, and when the panoramic image D1 and shape information D2 indicating the shape of the interior space are input, it outputs processed feature quantities that show features related to the correlation of the planes defining the interior space, and decodes the processed feature quantities to output a feature quantity vector that shows the position of the corners of the planes defining the interior space in the panoramic image, using an instructional model, when image features extracted from the panoramic image D1 and shape features extracted from the shape information D2 are input The teacher model estimation unit 13 outputs processed features when a panoramic image D1 is input, and the student model estimation unit 15 outputs processed features when image features extracted from the panoramic image D1 are input, using a student model that outputs processed features showing features related to the correlation of planes defining the internal space when a panoramic image D1 is input, decodes the processed features and outputs a feature vector showing the position of the corner of the plane defining the internal space in the panoramic image D1, and the loss L is the difference between the distribution of processed features output by the teacher model estimation unit 13 and the distribution of processed features output by the student model estimation unit 15. feature The system includes a student model learning unit 16 that calculates and trains the student model as an estimation model so that the loss on the validation data is minimized.

[0070] As a result, the three-dimensional reconstruction device 300 can use the estimation model learned by the learning device 100 to estimate feature vectors with the same high accuracy as when shape features are used, without using shape features. Therefore, the three-dimensional reconstruction device 300 can calculate the three-dimensional coordinates of points constituting the corners of the internal space of a structure with high accuracy, and can reconstruct the internal space in three dimensions with high accuracy. Consequently, for example, by reflecting the deterioration of surfaces defining the internal space of a structure, detected by workers through inspections, etc., in the three-dimensional image D3, the modeling accuracy of the structure showing deteriorated areas is improved. Furthermore, by reflecting furniture placed in the internal space of a structure, wallpaper applied to surfaces defining the internal space, etc., in the three-dimensional image D3, the modeling accuracy of the structure is improved. In addition, because the three-dimensional reconstruction device 300 does not use shape features, it can estimate three-dimensional coordinates at high speed.

[0071] Furthermore, the three-dimensional reconstruction device 300 in this embodiment is a three-dimensional reconstruction device 300 that reconstructs the interior space of a structure in three dimensions from a panoramic image D1 taken of the interior space of the structure, and comprises an image input unit 32 that receives the panoramic image D1 as input, an estimation unit 33 that inputs the panoramic image D1 to an estimation model learned by the learning device 100 described above and outputs a feature vector indicating the position of the corners of the planes defining the interior space, a three-dimensional coordinate calculation unit 34 that calculates the three-dimensional coordinates of the points constituting the corners based on the feature vector, and a post-processing unit 35 that reconstructs the interior space in three dimensions based on the three-dimensional coordinates. As a result, the three-dimensional reconstruction device 300 can calculate the three-dimensional coordinates of the planes defining the interior space with high accuracy without using shape features. Therefore, the three-dimensional reconstruction device 300 can reconstruct the interior space in three dimensions with high accuracy based on the three-dimensional coordinates. Consequently, for example, by reflecting the deterioration of surfaces defining the internal space of a structure, detected by workers through inspections, etc., into the three-dimensional image D3, the modeling accuracy of the structure showing deteriorated areas is improved. Furthermore, by reflecting furniture placed in the internal space of the structure, wallpaper applied to surfaces defining the internal space, etc., into the three-dimensional image D3, the modeling accuracy of the structure is improved. In addition, since the three-dimensional reconstruction device 300 does not use shape features, it can reconstruct the internal space in three dimensions at high speed.

[0072] <Program> The learning device 100 and the three-dimensional reconstruction device 300 described above can be implemented by a computer. Furthermore, programs for functioning as the learning device 100 and the three-dimensional reconstruction device 300 may be provided. These programs may be stored on a storage medium or provided via a network. Figure 7 is a block diagram illustrating the schematic configuration of a computer 401 functioning as the learning device 100. The computer functioning as the three-dimensional reconstruction device 300 is similar. Here, the computer 401 may be a general-purpose computer, a dedicated computer, a workstation, a PC (Personal Computer), an electronic notepad, etc. Program instructions may be program code, code segments, etc., for executing the required tasks.

[0073] As shown in Figure 7, the computer 401 comprises a processor 410, a ROM (Read Only Memory) 420, a RAM (Random Access Memory) 430, storage 440, an input unit 450, an output unit 460, and a communication interface (I / F) 470. Each component is connected to each other via a bus 480 so as to be able to communicate with each other. The processor 410 is specifically a CPU (Central Processing Unit), MPU (Micro Processing Unit), GPU (Graphics Processing Unit), DSP (Digital Signal Processor), SoC (System on a Chip), etc., and may be composed of multiple processors of the same or different types.

[0074] The processor 410 controls each configuration and performs various arithmetic operations. Specifically, the processor 410 reads a program from the ROM 420 or storage 440 and executes the program using the RAM 430 as a working area. The processor 410 controls each configuration and performs various arithmetic operations according to the program stored in the ROM 420 or storage 440. In the embodiment described above, the program according to this disclosure is stored in the ROM 420 or storage 440.

[0075] The program may be stored on a storage medium readable by computer 401. Using such a storage medium, the program can be installed on computer 401. Here, the storage medium on which the program is stored may be a non-transitory storage medium. The non-transitory storage medium is not particularly limited, but may include, for example, a CD-ROM, DVD-ROM, or USB (Universal Serial Bus) memory. Alternatively, the program may be downloaded from an external device via a network.

[0076] ROM420 stores various programs and data. RAM430 temporarily stores programs or data as a working area. Storage440 consists of an HDD (Hard Disk Drive) or SSD (Solid State Drive) and stores various programs and data, including the operating system.

[0077] The input unit 450 includes one or more input interfaces that receive user input operations and acquire information based on the user operations. For example, the input unit 450 is a pointing device, a keyboard, a mouse, etc., but is not limited to these.

[0078] The output unit 460 includes one or more output interfaces for outputting information. For example, the output unit 460 is a display that outputs information as video, or a speaker that outputs information as audio, but is not limited to these. If the output unit 460 is a touch panel display, it also functions as an input unit 450.

[0079] The communication interface 470 is an interface for communicating with external devices.

[0080] The following additional information is disclosed regarding the embodiments described above. [Additional note 1] A learning device for learning an estimation model for three-dimensional reconstruction of the interior space of a structure from panoramic images of the interior space of the structure, The controller is equipped with a controller, Using a training model that, when the panoramic image and shape information indicating the shape of the internal space are input, outputs processed feature quantities showing features related to the correlation of the planes defining the internal space, decodes the processed feature quantities to output feature vectors indicating the positions of the corners of the planes defining the internal space in the panoramic image, when image features extracted from the panoramic image and shape features extracted from the shape information are input, the processed feature quantities are output. Using a student model that, upon inputting the panoramic image, outputs processed feature quantities showing characteristics related to the correlation of the planes defining the internal space, decodes the processed feature quantities to output feature vectors indicating the positions of the corners of the planes defining the internal space in the panoramic image, and then, upon inputting image features extracted from the panoramic image, outputs the processed feature quantities, A learning device that calculates a loss representing the difference between the distribution of processed features output using the teacher model and the distribution of processed features output using the student model, and trains the student model as the estimation model so that the loss takes its minimum value on the validation data. [Additional note 2] A three-dimensional reconstruction device for three-dimensionally reconstructing the interior space of a structure from a panoramic image of the interior space captured, The system includes an input interface for receiving the aforementioned panoramic image and a controller. The aforementioned controller, The panoramic image is input to the estimation model trained by the learning device described in Appendix 1, and a feature vector indicating the position of the corner of the plane defining the interior space is output. Based on the aforementioned feature vector, the three-dimensional coordinates of the points constituting the corner portion are calculated. A three-dimensional reconstruction device that reconstructs the internal space in three dimensions based on the aforementioned three-dimensional coordinates. [Additional note 3] A learning method performed by a learning device that learns an estimation model for three-dimensional reconstruction of the interior space of a structure from panoramic images of the interior space of the structure, Using a training model that, when the panoramic image and shape information indicating the shape of the internal space are input, outputs processed feature quantities showing features related to the correlation of the planes defining the internal space, decodes the processed feature quantities to output feature vectors indicating the positions of the corners of the planes defining the internal space in the panoramic image, when image features extracted from the panoramic image and shape features extracted from the shape information are input, the processed feature quantities are output. Using a student model that, upon inputting the panoramic image, outputs processed feature quantities showing characteristics related to the correlation of the planes defining the internal space, decodes the processed feature quantities to output feature vectors indicating the positions of the corners of the planes defining the internal space in the panoramic image, and then, upon inputting image features extracted from the panoramic image, outputs the processed feature quantities, A learning method comprising: calculating a loss that represents the difference between the distribution of processed features output using the teacher model and the distribution of processed features output using the student model; and training the student model as the estimation model so that the loss takes its minimum value on the validation data. [Additional note 4] A three-dimensional reconstruction method performed by a three-dimensional reconstruction apparatus that reconstructs the interior space of a structure in three dimensions from a panoramic image of the interior space of the structure, The panoramic image is received as input. The estimation model trained by the learning method described in Appendix 3 is input with the panoramic image and outputs a feature vector indicating the position of the corner of the plane defining the interior space. Based on the aforementioned feature vector, the three-dimensional coordinates of the points constituting the corner portion are calculated. A three-dimensional reconstruction method for three-dimensionally reconstructing the internal space based on the three-dimensional coordinates. [Additional note 5] A non-temporary computer-readable medium containing a program for operating a computer as a learning device as described in Appendix 1. [Additional note 6] A non-temporary computer-readable medium storing a program for operating a computer as a three-dimensional reconstruction device as described in Appendix 2.

[0081] All documents, patent applications, and technologies described herein are incorporated by reference to the same extent as if each individual document, patent application, and technology were specifically and individually described as being incorporated by reference.

[0082] Although the embodiments described above are representative examples, it will be apparent to those skilled in the art that many modifications and substitutions are possible within the spirit and scope of this disclosure. Therefore, the present invention should not be construed as being limited by the embodiments described above, and various modifications or changes are possible without departing from the claims. [Explanation of Symbols]

[0083] 11 Image input section for the training model 12 Shape Information Input Section 13. Teacher Model Estimation Unit 14. Image input section for student models 15 Student Model Estimation Department 16 Student Model Learning Department 31 Student Model Memory Department 32 Image Input Section 33 Estimation part 34 3D coordinate calculation section 35 Post-processing 100 Learning Devices 131 Image Feature Extractor for Training Models 132 Shape Feature Extractor 133 Feature processor for training models 134 Feature Decoder for Training Models 151 Image Feature Extractor for Student Models 152 Feature processor for student models 153 Feature Decoder for Student Models 300 Three-dimensional reconstruction device 401 Computer 410 Processor 420 ROM 430 RAM 440 storage 450 Input section 460 Output section 470 Communication Interfaces 480 bus

Claims

1. A learning device for learning an estimation model for three-dimensional reconstruction of the interior space of a structure from panoramic images of the interior space of the structure, A teacher model estimation unit that, upon inputting the panoramic image and shape information indicating the shape of the internal space, outputs processed feature quantities showing features relating to the correlation of the planes defining the internal space, decodes the processed feature quantities, and outputs feature vectors indicating the positions of the corners of the planes defining the internal space in the panoramic image, and upon inputting image features extracted from the panoramic image and shape features extracted from the shape information, outputs the processed feature quantities. A student model estimation unit that, when an image feature extracted from the panoramic image is input, outputs a processed feature quantity showing characteristics related to the correlation of the planes defining the internal space, decodes the processed feature quantity and outputs a feature quantity vector showing the position of the corner of the planes defining the internal space in the panoramic image, and when an image feature quantity extracted from the panoramic image is input, outputs the processed feature quantity, A student model learning unit calculates a loss that represents the difference between the distribution of processed features output using the teacher model and the distribution of processed features output using the student model, and trains the student model as the estimation model so that the loss takes the minimum value on the validation data. A learning device equipped with the following features.

2. A three-dimensional reconstruction device for three-dimensionally reconstructing the interior space of a structure from a panoramic image of the interior space captured, An image input unit that receives the aforementioned panoramic image input, An estimation unit that inputs the panoramic image to an estimation model learned by the learning device described in claim 1 and outputs a feature vector indicating the position of the corner of the plane defining the internal space, A three-dimensional coordinate calculation unit calculates the three-dimensional coordinates of the points constituting the corner portion based on the aforementioned feature vector, A post-processing unit that reconstructs the internal space in three dimensions based on the three-dimensional coordinates, A three-dimensional reconstruction device equipped with the following features.

3. A learning method performed by a learning device that learns an estimation model for three-dimensional reconstruction of the interior space of a structure from panoramic images of the interior space of the structure, Using a training model that, when the panoramic image and shape information indicating the shape of the internal space are input, outputs processed feature quantities showing features relating to the correlation of the planes defining the internal space, decodes the processed feature quantities to output feature vectors indicating the positions of the corners of the planes defining the internal space in the panoramic image, the step of outputting the processed feature quantities when image features extracted from the panoramic image and shape features extracted from the shape information are input, Using a student model that, when the panoramic image is input, outputs processed feature quantities showing characteristics related to the correlation of the planes defining the internal space, decodes the processed feature quantities to output feature vectors showing the positions of the corners of the planes defining the internal space in the panoramic image, the student model outputs the processed feature quantities when image features extracted from the panoramic image are input. The steps include: calculating a loss that represents the difference between the distribution of processed features output using the teacher model and the distribution of processed features output using the student model, and training the student model as the estimation model so that the loss takes its minimum value on the validation data; Learning methods that include this.

4. A three-dimensional reconstruction method performed by a three-dimensional reconstruction apparatus that reconstructs the interior space of a structure in three dimensions from a panoramic image of the interior space of the structure, The step of receiving the aforementioned panoramic image as input, The steps include inputting the panoramic image into an estimation model trained by the learning method described in claim 3 and outputting a feature vector indicating the position of the corner of the plane defining the interior space, The steps include: calculating the three-dimensional coordinates of the points constituting the corner portion based on the feature vector; The steps include: three-dimensional reconstruction of the internal space based on the three-dimensional coordinates; A three-dimensional reconstruction method, including...

5. A program for operating a computer as a learning device according to claim 1.

6. A program for operating a computer as a three-dimensional reconstruction apparatus as described in claim 2.

Images