Three-dimensional reconstruction device, three-dimensional reconstruction method and program

Abstract

A three-dimensional reconstruction device (10) according to the present disclosure comprises: a panorama image acquisition unit (111) for acquiring a panorama image obtained by imaging the interior of a subject structure; a drawing information acquisition unit (112) for acquiring a drawing of the interior of the structure and the ceiling height of the structure; a plan view creation unit (113) for creating a plan view of the interior of the structure on the basis of the panorama image and the ceiling height; a registration unit (116) for aligning the created plan view and the drawing, and determining coordinate values of the plan view; and a three-dimensional coordinates calculation unit (117) for carrying out a three-dimensional reconstruction of the structure on the basis of the coordinate values that have been determined for the plan view.

Description

[Technical Field]

[0001] The present disclosure relates to a three-dimensional reconstruction device, a three-dimensional reconstruction method, and a program. [Background technology]

[0002] Conventionally, in 3D reconstruction techniques based on images of the interior of a structure, 3D coordinates are reconstructed through preprocessing, wall detection, and 3D coordinate calculation. The 3D coordinate calculation is based on the Manhattan world hypothesis (see, for example, Non-Patent Documents 1 and 2). The "Manhattan world hypothesis" is a hypothesis that all walls inside a structure are orthogonal. FIG. 12A shows an example of a drawing of the interior of a structure based on the Manhattan world hypothesis, and FIG. 12B shows an example of a drawing of the interior of a structure that is not based on the Manhattan world hypothesis. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Sun, C., et al., “HorizonNet: Learning Room Layout With 1D Representation and Pano Stretch Data Augmentation”, CVPR, 2019. [Non-patent document 2] ] Yang, ST., et al., “DuLa-Net: A Dual-Projection Network for Estimating Room Layouts from a Single RGB Panorama”, CVPR, 2019. Summary of the Invention [Problem to be solved by the invention]

[0004] However, it has been difficult to perform 3D reconstruction for the interior of structures where all walls are not orthogonal, i.e., structures that are not based on the Manhattan world hypothesis. Thus, there has been a demand for technology that can accurately reconstruct 3D images of the interior of structures that do not assume an orthogonal system.

[0005] In view of the above circumstances, an object of the present disclosure is to provide a technology for accurately reconstructing three-dimensional images of the interior of a structure without assuming an orthogonal system. [Means for solving the problem]

[0006] The three-dimensional reconstruction device according to the present disclosure includes a panoramic image acquisition unit that acquires a panoramic image of the interior of a target structure, a drawing information acquisition unit that acquires a drawing of the interior of the structure and the ceiling height of the structure, a plan view creation unit that creates a plan view of the interior of the structure based on the panoramic image and the ceiling height, a registration unit that aligns the created plan view with the drawing and determines the coordinate values ​​of the plan view, and a three-dimensional coordinate calculation unit that performs three-dimensional reconstruction of the structure based on the determined coordinate values ​​of the plan view.

[0007] In addition, the three-dimensional reconstruction method according to the present disclosure is a three-dimensional reconstruction method executed by a three-dimensional reconstruction device, and includes a panoramic image acquisition step of acquiring a panoramic image of the interior of a target structure, a drawing information acquisition step of acquiring a drawing of the interior of the structure and the ceiling height of the structure, a plan view creation step of creating a plan view of the interior of the structure based on the panoramic image and the ceiling height, a registration step of aligning the created plan view with the drawing and determining coordinate values ​​of the plan view, and a three-dimensional coordinate calculation step of three-dimensionally reconstructing the structure based on the determined coordinate values ​​of the plan view.

[0008] Furthermore, a program according to the present disclosure causes a computer to function as a three-dimensional reconstruction apparatus according to the present disclosure. [Effects of the Invention]

[0009] According to the present disclosure, it is possible to provide a technology for accurately reconstructing three-dimensional images of the interior of a structure without assuming an orthogonal system. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of a three-dimensional reconstruction apparatus according to an embodiment of the present disclosure. [Figure 2] FIG. 2 is a diagram illustrating an example of a panoramic image acquired by a panoramic image acquisition unit. [Figure 3] 10 is a diagram showing an example of a drawing of the interior of a target structure acquired by a drawing information acquisition unit. FIG. [Figure 4] FIG. 10 is a simplified diagram illustrating an example in which coordinate axes are added to a panoramic image for the purpose of explanation. [Figure 5] FIG. 1 is a cross-sectional view of a structure showing a camera installed inside the structure that captured the panoramic image. [Figure 6] FIG. 10 is a diagram showing an example of a plan view formed by a point cloud acquired by a plan view creating unit; [Figure 7] 10 is a diagram showing an example of a plan view in which a line is detected by a first line detector. FIG. [Figure 8] 10 is a diagram showing an example of a drawing in which a line is detected by a second line detector. FIG. [Figure 9A] 10 is a flowchart illustrating an example of an operation of a three-dimensional reconstruction apparatus according to an embodiment of the present disclosure. [Figure 9B] 10 is a flowchart illustrating an example of an operation of a three-dimensional reconstruction apparatus according to an embodiment of the present disclosure. [Figure 10A] 10 is a flowchart showing an example of the operation of the three-dimensional reconstruction device according to the first modified example of the present disclosure. [Figure 10B] 10 is a flowchart showing an example of the operation of the three-dimensional reconstruction device according to the first modified example of the present disclosure. [Figure 11A] 10 is a flowchart showing an example of the operation of a three-dimensional reconstruction device according to a second modification of the present disclosure. [Figure 11B]10 is a flowchart showing an example of the operation of a three-dimensional reconstruction device according to a second modification of the present disclosure. [Figure 12A] FIG. 10 is a diagram showing an example of a drawing of the interior of a structure based on the Manhattan World Hypothesis. [Figure 12B] FIG. 10 shows an example of a drawing of the interior of a structure that is not based on the Manhattan World hypothesis. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings as appropriate. In each drawing, identical or corresponding parts are designated by the same reference numerals. In the description of the present embodiments, descriptions of identical or corresponding parts will be omitted or simplified as appropriate. The embodiments described below are examples of the configuration of the present disclosure, and the present invention is not limited to the following embodiments.

[0012] The three-dimensional reconstruction device 10 according to this embodiment is a computer such as a server belonging to a cloud computing system or other computing system.

[0013] <Configuration of the 3D reconstruction device 10> An example of the configuration of a three-dimensional reconstruction device 10 according to this embodiment will be described with reference to Fig. 1. As shown in Fig. 1, the three-dimensional reconstruction device 10 includes a control unit 11, a storage unit 12, a communication unit 13, an input unit 14, and an output unit 15.

[0014] The storage unit 12 includes one or more memories, and may include, for example, semiconductor memory, magnetic memory, optical memory, etc. Each memory included in the storage unit 12 may function, for example, as a main storage device, an auxiliary storage device, or a cache memory. The storage unit 12 stores any information used in the operation of the three-dimensional reconstruction device 10. The storage unit 12 does not necessarily need to be provided inside the three-dimensional reconstruction device 10, and may be configured to be provided outside the three-dimensional reconstruction device 10.

[0015] The communication unit 13 includes at least one communication interface. The communication interface is, for example, a LAN interface. The communication unit 13 receives information used in the operation of the three-dimensional reconstruction device 10 and transmits information obtained by the operation of the three-dimensional reconstruction device 10.

[0016] The communication unit 13 enables the 3D reconstruction device 10 to transmit and receive information to and from other devices via a network. The network includes the Internet, at least one wide area network (WAN), at least one metropolitan area network (MAN), or a combination thereof. The network may include at least one wireless network, at least one optical network, or a combination thereof. The wireless network may be, for example, an ad hoc network, a cellular network, a wireless local area network (LAN), a satellite communication network, or a terrestrial microwave network.

[0017] The input unit 14 includes at least one input interface. The input interface is, for example, a physical key, a capacitance key, a pointing device, a touch screen integrated with a display, or a microphone. The input unit 14 accepts an operation to input information used in the operation of the three-dimensional reconstruction device 10. The input unit 14 may be connected to the three-dimensional reconstruction device 10 as an external input device instead of being provided in the three-dimensional reconstruction device 10. As a connection method, any method such as USB (Universal Serial Bus), HDMI (High-Definition Multimedia Interface) (registered trademark), or Bluetooth (registered trademark) can be used.

[0018] The output unit 15 includes at least one output interface. The output interface is, for example, a display or a speaker. The display is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display. The output unit 15 may include a device that can be worn by a user, such as VR goggles. The output unit 15 outputs information obtained by the operation of the three-dimensional reconstruction device 10. The output unit 15 may be connected to the three-dimensional reconstruction device 10 as an external output device, instead of being provided in the three-dimensional reconstruction device 10. Any connection method can be used, for example, USB, HDMI (registered trademark), or Bluetooth (registered trademark).

[0019] The control unit 11 is realized by a control and arithmetic circuit (controller). The control and arithmetic circuit may be configured by dedicated hardware such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array), or may be configured by a processor, or may be configured to include both. The control unit 11 executes processing related to the operation of the three-dimensional reconstruction device 10 while controlling each part of the three-dimensional reconstruction device 10. The control unit 11 can send and receive information to and from external devices via the communication unit 13 and a network.

[0020] The control unit 11 includes a panoramic image acquisition unit 111, a drawing information acquisition unit 112, a plan view creation unit 113, a first line detection unit 114, a second line detection unit 115, a registration unit 116, and a three-dimensional coordinate calculation unit 117.

[0021] The panoramic image acquisition unit 111 acquires a panoramic image I capturing the interior of a target structure S. FIG. 2 shows an example of the panoramic image I acquired by the panoramic image acquisition unit 111. The panoramic image I is an image obtained by capturing the interior of the structure S using a camera or the like at angles ranging from −90° to +90° in the vertical direction and from 0° to 360° in the horizontal direction. In the panoramic image I of FIG. 2, area A represents the ceiling inside the structure S, area B represents the wall surface, and area C represents the floor surface. The panoramic image I includes a first boundary line LC, which is the boundary line between the wall surface and the ceiling, and a second boundary line LF, which is the boundary line between the wall surface and the floor surface, both of which have been detected in advance. In the panoramic image I of FIG. 2, the boundary line LC and the boundary line LF are indicated by solid lines. The boundary lines LC and LF may be detected using any image processing technology, such as deep learning.

[0022] Any method may be adopted to acquire the panoramic image I. For example, the panoramic image acquisition unit 111 may directly acquire the panoramic image I from an external device such as a terminal device equipped with a camera via the communication unit 13, or may directly acquire the panoramic image I input by the user via the input unit 14. The panoramic image acquisition unit 111 may acquire the panoramic image I by reading out the panoramic image I stored in advance in the storage unit 12.

[0023] The panoramic image acquisition unit 111 outputs the acquired panoramic image I to the plan view creation unit 113.

[0024] The drawing information acquisition unit 112 acquires drawing information showing a drawing D of the interior of the target structure S and ceiling height information showing the ceiling height of the target structure S. The drawing D is specifically a plan completion drawing of the target structure S. FIG. 3 shows an example of the drawing D of the target structure S acquired by the drawing information acquisition unit 112. The drawing information is specifically information including coordinate values ​​of a point cloud representing the drawing D.

[0025] Any method may be employed to acquire the plan information and the ceiling height information. For example, the plan information acquisition unit 112 may analyze an image representing the plan D input by the user from the input unit 14 using any image processing technology to acquire the plan information and the ceiling height information.

[0026] The drawing information acquisition unit 112 outputs the drawing information to the second line detection unit 115. The drawing information acquisition unit 112 also outputs the ceiling height information to the floor plan creation unit 113.

[0027] The floor plan creation unit 113 acquires the panoramic image I and the ceiling height information. The floor plan creation unit 113 creates a floor plan P of the interior of the structure S based on the panoramic image I and the ceiling height indicated by the ceiling height information. Specifically, the floor plan creation unit 113 calculates the coordinate values ​​of a point cloud representing the floor plan P. The floor plan creation unit 113 may create the floor plan P by calculating the coordinates from the panoramic image I using any image processing technology.

[0028] FIG. 4 is a simplified diagram for the purpose of explanation showing an example in which coordinate axes are assigned to a panoramic image I. The v-axis in FIG. 4 represents the angle (elevation angle) from -90° to +90° when the camera captures the interior of the structure S in the up-down direction along the vertical direction, and the u-axis represents the angle (azimuth angle) from 0° to 360° when the camera captures the left-right direction along the horizontal direction. In other words, the width W of the panoramic image I corresponds to the u-axis, and the height H corresponds to the v-axis, and the coordinates (u c ,v c ) and the coordinates of any pixel on the boundary line LF (u f ,v f ) can be obtained. u c, u f ∈[0,W], v c, v f ∈[0,H].

[0029] 5 is a cross-sectional view of the structure S, showing the state in which the camera that captured the panoramic image I is installed inside the structure S. θ in FIG. cθ indicates the camera's shooting angle from a plane parallel to the ceiling or floor at the height of the camera from the floor to the boundary line LC. f indicates the camera's shooting angle from a plane parallel to the ceiling or floor at the height of the camera from the floor to the boundary line LF. c is the v-axis element of the pixel coordinates mentioned above. c and θ f is v f corresponds to:

[0030] θ c and θ f In order to obtain the above, the plan view creation unit 113 first calculates θ' in FIG. c and θ' f Find ∈[-π / 2,π / 2] where θ' c and θ' f θ' including the above can be calculated using the following formula 1.

number

[0031] Therefore, the plan view creation unit 113 calculates θ from the following equation 2. c Calculate.

number

[0032] Furthermore, the plan view creation unit 113 calculates θ from the following equation 3. f Calculate.

number

[0033] The floor plan creating unit 113 uses the ceiling height h inside the structure S indicated by the ceiling height information acquired from the drawing information acquiring unit 112 to calculate the height h of the camera from the floor shown in FIG. f is calculated using the following formula 4.

number

[0034] Next, the floor plan creation unit 113 uses the ceiling height h to calculate the distance d from the camera height position above the floor surface inside the structure S to the wall surface according to the following formula 5.

number

[0035] The plan view creation unit 113 applies the above formula 5 to each u coordinate and plots a point at a position of the distance d to the wall surface. For each plotted point, the plan view creation unit 113 sets the position of the distance d as the x-axis and y-axis coordinate values, and calculates the height h of the camera from the floor. f as the coordinate value of the Z axis, it is possible to obtain the coordinate values ​​of the point group that constitutes the floor plan P. In this way, the floor plan creation unit 113 calculates the distance d from the camera that captured the panoramic image I to the wall surface, and creates the floor plan P by plotting the distance d for each horizontal shooting angle of the camera.

[0036] The method is not limited to the above, and the plan view creation unit 113 may calculate the distance d from the camera to the wall surface by any method. For example, the plan view creation unit 113 may calculate the distance d using the detection results of a distance sensor, a depth camera, or the like.

[0037] 6 is an example of a plan view P formed by the point cloud acquired by the plan view creation unit 113. The plan view creation unit 113 outputs plan view information indicating the created plan view P to the first line detection unit 114.

[0038] The first line detection unit 114 uses an LSD (Line Segment Detector) method to detect lines in the plan view P indicated by the plan view information acquired from the plan view creation unit 113. However, the method is not limited to this, and any method may be adopted for detecting lines.

[0039] 7 is a diagram showing a plan view P' in which lines PL1 to PL6 have been detected by the first line detection unit 114 from the plan view P indicated by the plan view information. The first line detection unit 114 identifies at least one intersection of the detected lines as a first intersection. In the plan view P' in FIG. 7, the intersections identified by the first line detection unit 114 are indicated by circles labeled A1 to A6. The first line detection unit 114 outputs first intersection information indicating the identified intersections to the registration unit 116 together with plan view information including point groups other than the intersections.

[0040] The second line detection unit 115 detects lines in the drawing D indicated by the drawing information acquired from the drawing information acquisition unit 112 by the LSD method, similar to the first line detection unit 114. However, the present invention is not limited to this, and any method may be adopted for detecting lines.

[0041] FIG. 8 is a diagram showing drawing D' in which lines DL1 to DL6 have been detected by the second line detection unit 115 from drawing D indicated by the drawing information. The second line detection unit 115 identifies at least one intersection of the detected lines as a second intersection. In drawing D' in FIG. 8, the intersections identified by the second line detection unit 115 are indicated by circles labeled B1 to B6. The second line detection unit 115 outputs second intersection information indicating the identified intersections to the registration unit 116 together with drawing information including point groups other than the intersections.

[0042] The registration unit 116 aligns the points representing the plan view P' with the points representing the drawing D' based on the plan view information output from the first line detection unit 114 and the drawing information output from the second line detection unit 115. In this embodiment, the registration unit 116 aligns the points using a method such as an ICP (Iterative Closest Point) algorithm. However, the method is not limited to this, and the registration unit 116 may align the points using any algorithm.

[0043] The ICP algorithm is an algorithm for aligning multiple 3D point groups. The ICP algorithm is disclosed in, for example, the following document: Reference 1: Besl, PJ and ND McKay (1992), "A method for registration of 3-D shapes", IEEE Transactions on Pattern Analysis and Machine Intelligence 14(2): 239-256.

[0044] The ICP algorithm relies on a dimensional point cloud P obtained from different positions. s and P t Given, P s P t Specifically, the registration unit 116 executes the following steps 1 to 4.

[0045] Step 1: The registration unit 116 uses a nearest neighbor detection algorithm to find the point cloud P s Each point P si For the point group P t The closest point P in tj Find the point P si The corresponding point of tj The corresponding points are set at the initial positions. Since the corresponding points will be reset in steps 3 and 4 described below, the initial corresponding points are referred to as the "initial positions." The registration unit 116 performs alignment by including the first intersection indicated by the first intersection information output from the first line detection unit 114 and the second intersection indicated by the second intersection information output from the second line detection unit 115 in the initial positions. The registration unit 116 performs iterative processing without using the first intersection and the second intersection in steps 3 and 4 below. The nearest neighbor detection algorithm is disclosed in the following document 2, so a detailed explanation will be omitted. Reference 2: Eggert, DW, et al. (1997), "Estimating 3-D rigid body transformations: a comparison of four major algorithms", Machine Vision and Applications 9(5): 272-290. Step 2: Define an objective function E between the corresponding points determined in step 1 above, and estimate R and T at which E converges. Step 3. Using R and T obtained in step 2 above, P s Move. Step 4. Repeat steps 1 to 3 above.

[0046] In the ICP algorithm, if the coordinates of the initial positions of the two point clouds to be matched are significantly different, alignment may fail and the point clouds may not be determined at their original three-dimensional coordinate positions. In this embodiment, the registration unit 116 can robustly set the initial positions by using the first intersection detected by the first line detection unit 114 and the second intersection detected by the second line detection unit 115 as landmarks.

[0047] In this way, the registration unit 116 performs alignment using the first intersection indicated by the first intersection information output from the first line detection unit 114 and the second intersection indicated by the second intersection information output from the second line detection unit 115 as initial positions.

[0048] The point cloud resulting from the alignment performed by the registration unit 116 represents the shape of the boundary lines between the walls, ceiling, and floor of the target structure S, i.e., the shape of the plan view P". In this way, the registration unit 116 aligns the created plan view P' with the drawing D' and determines the coordinate values ​​of the plan view P". The registration unit 116 outputs information indicating the determined coordinate values ​​of the plan view P" to the three-dimensional coordinate calculation unit 117.

[0049] The three-dimensional coordinate calculation unit 117 performs three-dimensional reconstruction of the target structure S. Specifically, the three-dimensional coordinate calculation unit 117 acquires information indicating the coordinate values ​​of the plan view P'' determined by the registration unit 116. Furthermore, the three-dimensional coordinate calculation unit 117 calculates three-dimensional coordinates for the entire internal shape of the structure S other than the shape of the boundary lines represented by the plan view P''. Any method may be used to acquire this information. The three-dimensional coordinate calculation unit 117 stores information indicating the three-dimensional coordinates of the target structure S reconstructed in this manner in the memory unit 12.

[0050] <Program> A computer capable of executing program instructions can also be used to function as the above-described three-dimensional reconstruction apparatus 10. Here, the computer may be a general-purpose computer, a dedicated computer, a workstation, a PC (Personal Computer), an electronic notepad, etc. The program instructions may be program code, code segments, etc. for performing the necessary tasks.

[0051] The computer includes a processor, a storage unit, an input unit, an output unit, and a communication interface. The processor may be a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an SoC (System on a Chip), or the like, and may be configured with multiple processors of the same or different types. The processor reads and executes programs from the storage unit to control the above components and perform various arithmetic processing. Note that at least a portion of these processing contents may be implemented by hardware. The input unit is an input interface that accepts user input operations and acquires information based on the user operations, such as a pointing device, keyboard, or mouse. The output unit is an output interface that outputs information, such as a display or speaker. The communication interface is an interface for communicating with external devices.

[0052] The program may be recorded on a computer-readable recording medium. Using such a recording medium, the program can be installed on a computer. Here, the recording medium on which the program is recorded may be a non-transitory recording medium. The non-transitory recording medium is not particularly limited, and may be, for example, a CD-ROM, a DVD-ROM, or a USB memory. Furthermore, the program may be downloaded from an external device via a network.

[0053] <Operation of the 3D reconstruction device 10> Next, the operation of the three-dimensional reconstruction device 10 according to this embodiment will be described with reference to Figures 4, 5, 9A, and 9B. The operation of the three-dimensional reconstruction device 10 corresponds to the three-dimensional reconstruction method according to this embodiment.

[0054] In step S1, the panoramic image acquisition unit 111 of the control unit 11 acquires a panoramic image I capturing the interior of the target structure S. The panoramic image I includes a first boundary line LC, which is the boundary line between the wall surface and the ceiling, and a second boundary line LF, which is the boundary line between the wall surface and the floor, both of which have been detected in advance. The boundary lines LC and LF may be detected by any image processing technology, such as deep learning. Any method may be adopted for acquiring the panoramic image. The panoramic image acquisition unit 111 outputs the acquired panoramic image I to the floor plan creation unit 113.

[0055] In step S2, the drawing information acquisition unit 112 acquires drawing information indicating a drawing D of the interior of the target structure S and ceiling height information indicating the ceiling height of the target structure S. Any method may be used to acquire the drawing information and ceiling height information. The drawing information acquisition unit 112 outputs the drawing information to the second line detection unit 115. The drawing information acquisition unit 112 also outputs the ceiling height information to the floor plan creation unit 113.

[0056] In step S3, the floor plan creation unit 113 acquires the panoramic image I and the ceiling height information. The floor plan creation unit 113 creates a floor plan P based on the panoramic image I and the ceiling height indicated by the ceiling height information. Specifically, the floor plan creation unit 113 calculates the coordinate values ​​of a point cloud representing the floor plan.

[0057] θ in Figure 5 c and θ f In order to obtain the above, the plan view creation unit 113 first calculates θ' in FIG. c and θ' f Find ∈[-π / 2,π / 2] where θ' c and θ' f The θ′ including the angle θ can be calculated by the above-mentioned formula 1. The plan view creation unit 113 calculates the angle θ from the above-mentioned formula 2. c Calculate θ from the above equation 3 f The floor plan creating unit 113 calculates the height h of the camera from the floor shown in FIG. 5 using the ceiling height h inside the structure S indicated by the ceiling height information acquired from the plan information acquiring unit 112. f is calculated by the above-mentioned formula 4. Next, the floor plan creation unit 113 uses the ceiling height h to calculate the distance d from the camera height position above the floor surface inside the structure S to the wall surface by the above-mentioned formula 5.

[0058] The plan view creation unit 113 applies the above formula 5 to each u coordinate in Fig. 4 and plots a point at a position of the distance d to the wall surface. For each plotted point, the plan view creation unit 113 sets the position of the distance d as the x-axis and y-axis coordinate values, and sets the height h of the camera from the floor. f as the coordinate value of the Z axis, it is possible to obtain the coordinate values ​​of the point cloud that constitutes the floor plan P. In this way, the floor plan creation unit 113 calculates the distance d from the camera that captured the panoramic image I to the wall surface, and plots this distance d for each horizontal shooting angle of the camera to create the floor plan P. The floor plan creation unit 113 outputs floor plan information indicating the created floor plan P to the first line detection unit 114.

[0059] In step S4, the first line detection unit 114 detects lines in the plan view P indicated by the plan view information acquired from the plan view creation unit 113 using the LSD method. Any method may be used to detect lines, without being limited to this. The first line detection unit 114 identifies at least one intersection of the detected lines as a first intersection. The first line detection unit 114 outputs first intersection information indicating the identified intersection to the registration unit 116, together with plan view information including point groups other than the intersection.

[0060] In step S5, the second line detection unit 115 detects lines in the drawing D indicated by the drawing information acquired from the drawing information acquisition unit 112 using the LSD method, as with the first line detection unit 114. Any method may be used to detect lines, without being limited to this. The second line detection unit 115 identifies at least one intersection of the detected lines as a second intersection. The second line detection unit 115 outputs second intersection information indicating the identified intersection to the registration unit 116, together with drawing information including point groups other than the intersection.

[0061] In step S6 of FIG. 9B , the registration unit 116 aligns the point cloud representing the plan view P′ with the point cloud representing the drawing D′ based on the plan view information output from the first line detection unit 114 and the drawing information output from the second line detection unit 115. Specifically, the registration unit 116 aligns the point cloud representing the plan view P′ with the point cloud representing the drawing D′ using, as initial positions, the first intersection indicated by the first intersection information output from the first line detection unit 114 and the second intersection indicated by the second intersection information output from the second line detection unit 115. In this embodiment, the registration unit 116 aligns the point cloud using a method such as an ICP algorithm. However, the method is not limited to this, and the registration unit 116 may align the point cloud using any algorithm.

[0062] The point cloud resulting from the alignment performed by the registration unit 116 represents the shape of the boundary lines between the walls, ceiling, and floor of the target structure S, i.e., the shape of the plan view P". In this way, the registration unit 116 aligns the created plan view P' with the drawing D' and determines the coordinate values ​​of the plan view P". The registration unit 116 outputs information indicating the determined coordinate values ​​of the plan view P" to the three-dimensional coordinate calculation unit 117.

[0063] In step S7, the three-dimensional coordinate calculation unit 117 performs three-dimensional reconstruction of the target structure S. Specifically, the three-dimensional coordinate calculation unit 117 acquires information indicating the coordinate values ​​of the plan view P'' determined by the registration unit 116. Furthermore, the three-dimensional coordinate calculation unit 117 calculates three-dimensional coordinates for the entire internal shape of the structure S other than the shape of the boundary lines represented by the plan view P''. Any method may be used to acquire this information. The three-dimensional coordinate calculation unit 117 stores information indicating the three-dimensional coordinates of the target structure S reconstructed in this manner in the memory unit 12.

[0064] In step S8, the control unit 11 displays an image representing the interior of the reconstructed three-dimensional structure S to the user in response to a user request. Any method may be used to display the image to the user. For example, the control unit 11 may display the image to the user via the output unit 15. The control unit 11 may communicate with a terminal device used by the user via the communication unit 13 and transmit the image to the terminal device. Thereafter, the operation of the three-dimensional reconstruction device 10 ends.

[0065] As described above, the three-dimensional reconstruction device 10 according to this embodiment includes a panoramic image acquisition unit 111 that acquires a panoramic image of the interior of a target structure, a drawing information acquisition unit 112 that acquires a drawing of the interior of the structure and the ceiling height of the structure, a plan view creation unit 113 that creates a plan view of the interior of the structure based on the panoramic image and the ceiling height, a registration unit 116 that aligns the created plan view with the drawing and determines the coordinate values ​​of the plan view, and a three-dimensional coordinate calculation unit 117 that performs three-dimensional reconstruction of the structure based on the determined coordinate values ​​of the plan view.

[0066] According to this embodiment, by aligning the created floor plan with the drawing, corrections can be made using the drawing, ultimately determining accurate floor plan coordinates. High-precision 3D reconstruction can ultimately be achieved, leading to improved visibility for users. For example, when an inspection image of the interior of a structure such as infrastructure equipment is used as a panoramic image, the user can obtain 3D reconstructed data including information indicating internal deterioration areas. This data can be used to perform structural calculations and quantitatively evaluate the integrity of the structure. Thus, this embodiment enables 3D reconstruction of a panoramic image of the interior of a structure with complex interior wall shapes that does not follow the Manhattan world assumption. Therefore, it is possible to improve the technology for accurately reconstructing 3D images of the interior of a structure that does not assume an orthogonal system.

[0067] As described above, the 3D reconstruction device 10 according to this embodiment further includes a first line detector 114 that detects straight lines in the created plan view and identifies at least one intersection of the straight lines as a first intersection, and a second line detector 115 that detects straight lines in the acquired drawing and identifies at least one intersection of the straight lines as a second intersection. The registration unit 116 performs registration using the first intersection and the second intersection as initial positions.

[0068] While the accuracy of alignment using the ICP algorithm depends heavily on the initial positions of the point groups, this embodiment enables alignment using the first intersection point and the second intersection point detected by the first line detection unit 114 and the second line detection unit 115, respectively, as landmarks. Since the first intersection point and the second intersection point are automatically identified as corresponding landmarks, alignment accuracy is likely to improve. This makes it possible to improve the technology for accurately reconstructing 3D images of the interior of a structure without assuming an orthogonal system.

[0069] As described above, in the three-dimensional reconstruction device 10 according to this embodiment, the plan view creation unit 113 calculates the distance from the camera that captured the panoramic image to the wall surface of the structure, and creates a plan view by plotting the distance for each horizontal shooting angle of the camera.

[0070] According to this embodiment, based on the assumption that the ceiling height inside the structure S is constant, the distance from the camera to the wall can be calculated and a floor plan can be created. This floor plan can also be created for a panoramic image of the interior of a structure whose walls are not orthogonal, i.e., a structure that does not conform to the Manhattan world hypothesis. By performing alignment based on the created floor plan, it is possible to reduce calculation costs and calculate three-dimensional coordinates even for structures that do not conform to the Manhattan world hypothesis. This improves the technology for accurately reconstructing three-dimensional images of the interior of a structure that does not assume an orthogonal system.

[0071] Although the present disclosure has been described based on the drawings and embodiments, it should be noted that various modifications and alterations can be easily made by those skilled in the art based on the present disclosure. Therefore, it should be noted that these modifications and alterations are included in the scope of the present disclosure.

[0072] (First Modification) Next, as a first modified example of the embodiment of the present disclosure, a case will be described in which the three-dimensional reconstruction device 10 is applied to a structure S2 that is different from the structure S of the above-described embodiment. In this modified example, the configuration of each part of the three-dimensional reconstruction device 10 is the same as in the above-described embodiment, and therefore description thereof will be omitted.

[0073] The difference between the operation of the 3D reconstruction device 10 according to the above embodiment and the operation of the 3D reconstruction device 10 according to this modification will be described below with reference to Figures 10A and 10B. Steps S21 to S28 in Figures 10A and 10B correspond to steps S1 to S8 in Figures 9A and 9B of the above embodiment.

[0074] In this example, the walls inside the structure S2 are occluded by an object, and the detection accuracy of the walls and boundaries in the panoramic image I2 acquired by the panoramic image acquisition unit 111 in step S21 is low. In step S22, the drawing information acquisition unit 112 acquires drawing information indicating a drawing D2 of the interior of the target structure S2 and ceiling height information indicating the ceiling height of the target structure S2, and outputs the drawing information to the second line detection unit 115 and the ceiling height information to the floor plan creation unit 113. In this case, in step S23, a distorted floor plan P2 is created by the floor plan creation unit 113.

[0075] The plan view P2' in step S24 is obtained by detecting the first intersection point and the straight line in the plan view P2 by the first line detection unit 114. The plan view D2' in step S25 is obtained by detecting the second intersection point and the straight line in the plan view D2 acquired by the plan view information acquisition unit 112 by the second line detection unit 115.

[0076] In step S26 of FIG. 10B, the registration unit 116 performs alignment using the detected first intersection point and second intersection point as initial positions. As a result, the registration unit 116 determines the coordinate values ​​of the plan view P2''. In step S27, the three-dimensional coordinate calculation unit 117 performs three-dimensional reconstruction of the target structure S2, as in the above-mentioned embodiment. In step S28, the control unit 11 displays to the user an image representing the interior of the reconstructed three-dimensional structure S2 in response to a user request, as in the above-mentioned embodiment.

[0077] As described above, in this modification, even if the plan view creation unit 113 creates a distorted plan view P2, the registration unit 116 can finally determine the coordinate values ​​of the undistorted plan view P2'' by correcting it using the drawing D2 provided as input. Therefore, it is possible to perform 3D reconstruction even from a panoramic image I2 in which the walls are occluded by objects and the detection accuracy of the wall surfaces or boundary lines is not high.

[0078] (Second Modification) Next, as a second modified example of the embodiment of the present disclosure, a case will be described in which the three-dimensional reconstruction device 10 is applied to a structure S3 that is different from the structure S of the above-described embodiment. In this modified example, the configuration of each part of the three-dimensional reconstruction device 10 is the same as in the above-described embodiment, and therefore description thereof will be omitted.

[0079] The following describes the differences between the operation of the 3D reconstruction device 10 according to the above-described embodiment and the operation of the 3D reconstruction device 10 according to this modification, with reference to Figures 11A and 11B. Steps S31 to S38 in Figures 11A and 11B correspond to steps S1 to S8 in the above-described embodiment.

[0080] In this example, the interior of structure S3 has pillars on the walls, and the detection accuracy of the walls and boundaries in panoramic image I3 acquired by panoramic image acquisition unit 111 in step S31 is low. In step S32, drawing information acquisition unit 112 acquires drawing information indicating a drawing D3 of the interior of target structure S3 and ceiling height information indicating the ceiling height of target structure S3, and outputs the drawing information to second line detection unit 115 and the ceiling height information to floor plan creation unit 113. In this case, in step S33, floor plan P3 with low accuracy is created by plan plan creation unit 113.

[0081] The plan view P3' in step S34 is obtained by detecting the first intersection point and the straight line in the plan view P3 by the first line detection unit 114. The plan view D3' in step S35 is obtained by detecting the second intersection point and the straight line in the plan view D3 acquired by the plan view information acquisition unit 112 by the second line detection unit 115.

[0082] In step S36 of FIG. 11B, the registration unit 116 performs alignment using the detected first intersection point and second intersection point as initial positions. As a result, the registration unit 116 determines the coordinate values ​​of the plan view P3''. In step S37, the three-dimensional coordinate calculation unit 117 performs three-dimensional reconstruction of the target structure S3, as in the above-described embodiment. In step S38, the control unit 11 displays to the user an image showing the interior of the reconstructed three-dimensional structure S3 in response to a user request, as in the above-described embodiment.

[0083] As described above, in this modification, even if the plan view creation unit 113 creates a plan view P3 with low accuracy, the registration unit 116 can finally determine the coordinate values ​​of the accurate plan view P3'' by correcting it using the drawing D3 that reflects the positions of the pillars, which is given as input. Therefore, it is possible to perform three-dimensional reconstruction even from a panoramic image I3 that has pillars on the wall and is expected to be difficult to accurately estimate by wall surface or boundary detection.

[0084] The following additional notes are provided regarding the above-described embodiments.

[0085] (Additional note 1) A panoramic image of the interior of the target structure is acquired, Obtaining a drawing of the interior of the structure and the ceiling height of the structure; creating a plan view of the interior of the structure based on the panoramic image and the ceiling height; Aligning the created plan view with the drawing to determine the coordinate values ​​of the plan view; a control unit that performs three-dimensional reconstruction of the structure based on the determined coordinate values ​​of the plan view; A three-dimensional reconstruction device comprising: (Additional note 2) the control unit detects straight lines in the created plan view and identifies at least one intersection point of the straight lines as a first intersection point; Detecting straight lines in the acquired drawing and identifying at least one intersection point of the straight lines as a second intersection point; 2. The three-dimensional reconstruction device according to claim 1, wherein alignment is performed using the first intersection point and the second intersection point as initial positions. (Additional note 3) 3. The 3D reconstruction device according to claim 1, wherein the control unit calculates a distance from the camera that captured the panoramic image to a wall surface of the structure, and creates the plan view by plotting the distance for each horizontal shooting angle of the camera. (Additional note 4) A three-dimensional reconstruction method executed by a three-dimensional reconstruction device, comprising: a panoramic image acquisition step of acquiring a panoramic image of the interior of the target structure; a drawing information acquisition step of acquiring a drawing of the interior of the structure and a ceiling height of the structure; a plan view creation step of creating a plan view of the interior of the structure based on the panoramic image and the ceiling height; a registration step of aligning the created plan view with the drawing and determining coordinate values ​​of the plan view; a three-dimensional coordinate calculation step of three-dimensionally reconstructing the structure based on the determined coordinate values ​​of the plan view; A three-dimensional reconstruction method comprising: (Additional note 5) a first line detection step of detecting lines in the created plan view and identifying at least one intersection point of the lines as a first intersection point; a second line detection step of detecting straight lines in the acquired drawing and identifying at least one intersection point of the straight lines as a second intersection point; 5. The three-dimensional reconstruction method according to claim 4, wherein the registration step further includes a step of performing alignment using the first intersection point and the second intersection point as initial positions. (Additional note 6) 6. The 3D reconstruction method according to claim 4 or 5, wherein the plan view creation step further includes a step of calculating a distance from a camera that captured the panoramic image to a wall surface of the structure, and plotting the distance for each horizontal shooting angle of the camera to create the plan view. (Additional note 7) A non-transitory computer-readable medium storing a program for causing a computer to function as the three-dimensional reconstruction device described in appended claim 1 or 2. [Explanation of symbols]

[0086] 10 Three-dimensional reconstruction device 11 Control section 12 Storage section 13 Communications Department 14 Input section 15 Output section 111 Panoramic image acquisition unit 112 Drawing information acquisition unit 113 Floor Plan Creation Department 114 First straight line detector 115 Second straight line detector 116 Registration Section 117 Three-dimensional coordinate calculation section

Claims

1. a panoramic image acquisition unit that acquires a panoramic image of the inside of a target structure; a drawing information acquisition unit that acquires a drawing of the interior of the structure and a ceiling height of the structure; a plan view creating unit that creates a plan view of the interior of the structure based on the panoramic image and the ceiling height; a registration unit that aligns the created plan view with the drawing and determines coordinate values ​​of the plan view; a three-dimensional coordinate calculation unit that performs three-dimensional reconstruction of the structure based on the determined coordinate values ​​of the plan view; A three-dimensional reconstruction device comprising:

2. a first line detection unit that detects straight lines in the created plan view and identifies at least one intersection point of the straight lines as a first intersection point; a second line detection unit that detects straight lines in the acquired drawing and identifies at least one intersection point of the straight lines as a second intersection point; The three-dimensional reconstruction apparatus according to claim 1 , wherein the registration unit performs registration using the first intersection point and the second intersection point as initial positions.

3. 3. The three-dimensional reconstruction device according to claim 1, wherein the plan view creation unit calculates a distance from a camera that captured the panoramic image to a wall surface of the structure, and creates the plan view by plotting the distance for each horizontal shooting angle of the camera.

4. A three-dimensional reconstruction method executed by a three-dimensional reconstruction device, comprising: a panoramic image acquisition step of acquiring a panoramic image of the interior of the target structure; a drawing information acquisition step of acquiring a drawing of the interior of the structure and a ceiling height of the structure; a plan view creation step of creating a plan view of the interior of the structure based on the panoramic image and the ceiling height; a registration step of aligning the created plan view with the drawing and determining coordinate values ​​of the plan view; a three-dimensional coordinate calculation step of three-dimensionally reconstructing the structure based on the determined coordinate values ​​of the plan view; A three-dimensional reconstruction method comprising:

5. a first line detection step of detecting lines in the created plan view and identifying at least one intersection point of the lines as a first intersection point; a second line detection step of detecting lines in the acquired drawing and identifying at least one intersection point of the lines as a second intersection point; The three-dimensional reconstruction method according to claim 4 , wherein the registration step further includes a step of performing alignment using the first intersection point and the second intersection point as initial positions.

6. 6. The three-dimensional reconstruction method according to claim 4, wherein the plan view creation step further includes a step of calculating a distance from a camera that captured the panoramic image to a wall surface of the structure, and plotting the distance for each horizontal shooting angle of the camera to create the plan view.

7. A program for causing a computer to function as the three-dimensional reconstruction apparatus according to claim 1 or 2.

Images