Three-dimensional model generation device, three-dimensional model generation method, three-dimensional model generation program, drawing creation display device, drawing creation display method, and drawing creation display program

Abstract

[PROBLEM] To make it possible to generate a three-dimensional model capable of easily measuring an inter-surface distance or joining of a plurality of three-dimensional models. [SOLUTION] In the present invention, a coordinate axis along a wall surface of a target location is set with respect to a point group of the target location. A co-normal clustering process is executed to extract, from the target location point group, as a co-normal cluster, a set of points for which a prescribed direction and a normal direction with respect to the coordinate axis match within a prescribed allowable range. A rectangular process is executed to convert the co-normal cluster into a rectangular plane. A three-dimensional model in which the target location is represented by a set of a plurality of rectangular planes is generated. In particular, before the co-normal clustering process, an individual clustering process is executed to extract, as an individual cluster, a plane in an arbitrary direction from the target location point group, and the rectangular process is executed on the individual cluster and the co-normal cluster. In addition, the co-normal cluster is divided into a plurality of sub-clusters so as to correspond to the shape of the co-normal cluster, and the co-normal cluster and the sub-clusters are converted into rectangular planes.

Description

3D model generation device, 3D model generation method, 3D model generation program, drawing creation display device, drawing creation display method, and drawing creation display program

[0001] The present disclosure relates to a three-dimensional model generation device, a three-dimensional model generation method, and a three-dimensional model generation program that use a processor to perform a three-dimensional reconstruction process of a target location and generate a three-dimensional model based on point cloud data obtained.

[0002] A 3D reconstruction technique is known that generates point cloud data as 3D spatial information about a target location based on captured images of the target location. Furthermore, a technique is known that generates a 3D model of the target location from the point cloud data of the target location (see Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2022-142994

[0004] According to conventional technology, even a user who is not familiar with the algorithms for generating a three-dimensional model from point cloud data can easily and quickly generate a three-dimensional model suitable for radio wave propagation simulation.

[0005] However, three-dimensional models generated from point cloud data can be used for a variety of purposes, not just radio wave propagation simulations. In particular, measurements such as the distance between two opposing walls can be performed based on the three-dimensional model of a target location. Furthermore, multiple three-dimensional models can be combined to generate a larger three-dimensional model. Therefore, there is a demand for technology that can easily measure the distance between walls of a target location and combine multiple three-dimensional models, but conventional technology has not addressed this need.

[0006] Therefore, the main object of the present disclosure is to provide a three-dimensional model generation device, a three-dimensional model generation method, and a three-dimensional model generation program that can generate three-dimensional models that can easily measure inter-surface distances and combine multiple three-dimensional models.

[0007] The three-dimensional model generation device disclosed herein is a three-dimensional model generation device that uses a processor to execute a process of generating a three-dimensional model based on point cloud data acquired by performing a three-dimensional reconstruction process on a target location, and the processor sets coordinate axes along the wall surfaces of the target location for the point cloud of the target location, executes a conormal clustering process to extract from the point cloud of the target location a set of points whose normal directions coincide with a predetermined direction based on the coordinate axes within a predetermined tolerance range as a conormal cluster, and executes a rectangularization process to convert the conormal cluster into a rectangular plane, thereby generating the three-dimensional model that represents the target location as a set of multiple rectangular planes.

[0008] Furthermore, the three-dimensional model generation method disclosed herein is a three-dimensional model generation method in which a processor executes a process of generating a three-dimensional model based on point cloud data acquired by performing three-dimensional reconstruction processing of a target location, and is configured to set coordinate axes along the wall surfaces of the target location for the point cloud of the target location, execute a conormal clustering process to extract from the point cloud of the target location a set of points whose normal directions coincide with a predetermined direction based on the coordinate axes within a predetermined tolerance range as a conormal cluster, and execute a rectangularization process to convert the conormal cluster into a rectangular plane, thereby generating the three-dimensional model in which the target location is represented by a set of multiple rectangular planes.

[0009] Furthermore, the three-dimensional model generation program of the present disclosure is a three-dimensional model generation program that causes a processor to execute a process of generating a three-dimensional model based on point cloud data acquired by performing three-dimensional reconstruction processing of a target location, and is configured to set coordinate axes along the wall surfaces of the target location for the point cloud of the target location, execute a conormal clustering process that extracts, from the point cloud of the target location, a set of points whose normal direction coincides with a predetermined direction based on the coordinate axes within a predetermined tolerance range, as a conormal cluster, and execute a rectangularization process that converts the conormal cluster into a rectangular plane, thereby generating the three-dimensional model that represents the target location as a set of multiple rectangular planes.

[0010] According to the present disclosure, a 3D model can be obtained in which the parallelism of parallel planes at the reconstruction location is ensured. This allows for easy measurement of inter-surface distances. Furthermore, multiple 3D models can be easily combined. Furthermore, since a moderately abstracted 3D model is generated, the processing load on the processor can be reduced.

[0011] FIG. 1 is a block diagram showing the overall configuration of a 3D model generation system according to this embodiment; FIG. 2 is a block diagram showing the schematic configuration of a user terminal and a server; FIG. 3 is a flow diagram showing the procedure for 3D model generation processing performed on the server; FIG. 4 is a flow diagram showing the procedure for pre-processing performed on the server; FIG. 5 is an explanatory diagram showing the status of point clouds at the restoration location before and after the coordinate axis setting processing performed on the server; FIG. 6 is an explanatory diagram showing the status of first and second coordinate axis setting processing performed on the server; FIG. 7 is an explanatory diagram showing the procedure for second coordinate axis setting processing performed on the server; FIG. 8 is an explanatory diagram showing the procedure for second coordinate axis setting processing performed on the server; FIG. 9 is a flow diagram showing the procedure for clustering processing performed on the server; FIG. 10 is an explanatory diagram showing the status of conormal clustering processing performed on the server; FIG. 11 is an explanatory diagram showing an overview of conormal clustering processing performed on the server; FIG. 12 is an explanatory diagram showing an overview of individual clustering processing performed on the server; FIG. 13 is an explanatory diagram showing the status of axis rotation processing performed on the server; FIG. 14 is an explanatory diagram showing the status of sub-clustering processing performed on the server; FIG. 15 is an explanatory diagram showing the status of sub-clustering processing performed on the server; Diagram showing the outline of clustering processingDiagram showing the outline of sub-clustering processing performed on the serverDiagram showing the outline of sub-clustering processing performed on the serverDiagram showing the outline of sub-clustering processing performed on the serverDiagram showing the outline of sub-clustering processing performed on the serverDiagram showing the outline of sub-clustering processing performed on the serverDiagram showing the outline of sub-clustering processing performed on the serverDiagram showing the outline of sub-clustering processing performed on the serverFlow diagram showing the procedure of rectangularization processing performed on the serverDiagram showing the status of rectangularization processing performed on the serverDiagram showing a 3D model generated by rectangularization processing performed on the serverDiagram showing a 3D model generated by rectangularization processing performed on the serverDiagram showing an overview of alignment processing performed on the serverDiagram showing an overview of snap processing between first and second groups performed on the serverDiagram showing an overview of snap processing within a group performed on the serverDiagram showing the status of floor plan generation processing performed on the serverDiagram showing the status of dimension line insertion processing performed on the serverDiagram showing a viewing screen displayed on a user terminal

[0012] A first invention made to solve the above problem is a 3D model generation device that uses a processor to execute a process of generating a 3D model based on point cloud data acquired by performing a 3D reconstruction process on a target location, wherein the processor sets coordinate axes along the wall surfaces of the target location for the point cloud of the target location, executes a conormal clustering process to extract from the point cloud of the target location a set of points whose normal directions coincide with a predetermined direction based on the coordinate axes within a predetermined tolerance range as a conormal cluster, and executes a rectangularization process to convert the conormal cluster into a rectangular plane, thereby generating the 3D model that represents the target location as a set of multiple rectangular planes.

[0013] This allows for the creation of a 3D model in which the parallelism of parallel planes at the reconstruction location is ensured. This makes it easy to measure the distance between surfaces. It also makes it easy to combine multiple 3D models. Furthermore, since a moderately abstract 3D model is generated, the processing load on the processor can be reduced.

[0014] In addition, in a second invention, the processor estimates the directions of the four wall surfaces of the restoration location based on the distribution of normal directions for the entire point cloud of the restoration location, which is rectangular in plan view, and sets the coordinate axes based on the estimation results.

[0015] According to this method, the coordinate axes are set based on the distribution of normal directions for the entire point cloud of the restoration location, and therefore, compared to a method of setting coordinate axes based on a specific plane, errors due to distortion occurring in the point cloud of the restoration location are dispersed, making it possible to appropriately set coordinate axes that follow the wall surfaces of the target location. Furthermore, even if the point cloud of the restoration location mainly represents furniture such as desks and shelves placed at the restoration location and there is no large-area wall surface to serve as a reference, it is possible to appropriately set coordinate axes.

[0016] In addition, in a third invention, the processor is configured to obtain, as a normal angle, the radial angle when the normal direction of each point constituting the point cloud is expressed in cylindrical coordinates, multiply the normal angle of each point by four to obtain a quadrupled value of each point, estimate an optimal value of the quadrupled value for each point using an estimation method that excludes the influence of outliers, and multiply the optimal value of the quadrupled value by one-fourth to obtain a rotation angle that defines the direction of the coordinate axes.

[0017] This allows the coordinate axes to be appropriately set in a simple procedure based on the distribution of normal directions for the entire point cloud at the restoration location.

[0018] In addition, a fourth invention is configured such that, before the conormal clustering process, the processor performs an individual clustering process to extract planes of any direction from the point cloud of the target location as individual clusters, and then performs the rectangularization process on the individual clusters and the conormal clusters.

[0019] According to this, because individual clustering processing can extract planes in any direction, it compensates for the drawback of conormal clustering processing, which can only extract planes in a specified direction. On the other hand, while individual clustering processing cannot detect small planes, conormal clustering processing can detect even small planes as long as they are in a specified direction, so the drawback of individual clustering processing is compensated for by conormal clustering processing. As a result, by combining both conormal clustering processing and individual clustering processing, it is possible to appropriately extract the required clusters.

[0020] In a fifth aspect of the present invention, the processor divides the conormal cluster into a plurality of sub-clusters corresponding to the shape of the conormal cluster, and converts the conormal cluster and the sub-clusters into the rectangular plane.

[0021] This allows for improved fidelity of the three-dimensional model of the target location.

[0022] A sixth invention is a three-dimensional model generation method in which a processor executes a process of generating a three-dimensional model based on point cloud data acquired by performing three-dimensional reconstruction processing of a target location, the method including: setting coordinate axes along the wall surfaces of the target location for the point cloud of the target location; executing a conormal clustering process to extract, from the point cloud of the target location, a set of points whose normal direction coincides with a predetermined direction based on the coordinate axes within a predetermined tolerance range, as a conormal cluster; and executing a rectangularization process to convert the conormal cluster into a rectangular plane, thereby generating the three-dimensional model in which the target location is represented by a set of multiple rectangular planes.

[0023] As with the first invention, this method can obtain a 3D model in which the parallelism of parallel planes at the reconstruction location is ensured. This allows for easy measurement of inter-surface distances. Furthermore, it also allows for easy merging of multiple 3D models. Furthermore, since a moderately abstracted 3D model is generated, it is possible to reduce the processing load on the processor.

[0024] A seventh invention is a three-dimensional model generation program that causes a processor to execute a process of generating a three-dimensional model based on point cloud data acquired by performing three-dimensional reconstruction processing of a target location, the program being configured to set coordinate axes along the wall surfaces of the target location for the point cloud of the target location, execute a conormal clustering process to extract from the point cloud of the target location a set of points whose normal directions coincide with a predetermined direction based on the coordinate axes within a predetermined tolerance range as a conormal cluster, and execute a rectangularization process to convert the conormal cluster into a rectangular plane, thereby generating the three-dimensional model that represents the target location as a set of multiple rectangular planes.

[0025] As with the first invention, this method can obtain a 3D model in which the parallelism of parallel planes at the reconstruction location is ensured. This allows for easy measurement of inter-surface distances. Furthermore, it also allows for easy merging of multiple 3D models. Furthermore, since a moderately abstracted 3D model is generated, it is possible to reduce the processing load on the processor.

[0026] An eighth invention made to solve the above problem is a 3D model generation device that uses a processor to execute a process of generating a 3D model based on point cloud data acquired by performing 3D reconstruction processing of a target location, wherein the processor is configured to extract a plane from the point cloud of the target location, divide the plane into a plurality of circumscribing rectangles so as to correspond to the shape of the plane, convert the point cloud included in the circumscribing rectangle into a rectangular plane corresponding to the circumscribing rectangle, and generate the 3D model that represents the target location as a collection of a plurality of the rectangular planes.

[0027] This allows a plane (a set of points distributed on a plane at a predetermined density) representing the object to be restored placed at the target location to be expressed in the 3D model as multiple rectangular planes corresponding to the shape of the object to be restored, making it possible to generate a moderately abstract 3D model.Furthermore, since a moderately abstract 3D model is generated, the processing load on the processor can be reduced.

[0028] In addition, in a ninth aspect of the invention, the processor sets coordinate axes along the wall surfaces of the target location for the point cloud of the target location, extracts a set of points from the point cloud of the target location whose set direction based on the coordinate axes and normal directions match within a predetermined tolerance range as a conormal cluster, divides the conormal cluster into multiple sub-clusters as the circumscribing rectangles, and converts the conormal cluster and the sub-clusters into rectangular planes.

[0029] This method obtains a 3D model in which the parallelism of parallel planes at the reconstruction location is ensured. This makes it easy to measure the distance between surfaces. It also makes it easy to combine multiple 3D models. Furthermore, compared to when only conormal clusters are converted into rectangular planes, it is possible to generate a 3D model that faithfully represents the shape of the object to be reconstructed to a certain degree.

[0030] In addition, in a tenth invention, the processor sets two circumscribing rectangles so as to divide the plane in two, searches in the direction of the coordinate axes for a division position where the amount of area reduction of the circumscribing rectangle due to the division is maximum, and when the division position is found, divides the plane in two at that division position and sets two circumscribing rectangles.

[0031] According to this method, the plane representing the object to be restored is appropriately divided into two and converted into a rectangular plane, so that a suitably abstracted three-dimensional model can be generated.

[0032] In an eleventh aspect of the present invention, the processor is configured not to divide the plane when the amount of area reduction is less than a predetermined threshold.

[0033] By appropriately adjusting the threshold value for the amount of area reduction, it is possible to avoid randomly subdividing planes, and to generate a moderately abstract three-dimensional model.

[0034] In addition, in a twelfth invention, the processor sets two circumscribing rectangles to divide the plane in two, and attempts a bisection process to search in the direction of the coordinate axes for a division position that maximizes the amount of area reduction of the circumscribing rectangle due to the division; when the division position is found, the processor divides the plane in two at that division position to set one circumscribing rectangle; and then attempts the bisection process to search for a division position in the same search direction for the remaining point cloud excluding the range of the set circumscribing rectangle.

[0035] In this case, since the division into two parts is performed multiple times, it is possible to represent an object to be restored with a more complex shape in a three-dimensional model, compared to when the division into two parts is performed only once.

[0036] In a thirteenth aspect of the present invention, the processor sets a search box of a predetermined width for the plane, sets two circumscribing rectangles so as to divide the cloud of points within the search box into two, and attempts a bisection process in which the division position in the direction of the coordinate axes is searched for at a position where the amount of area reduction of the circumscribing rectangle due to the division is greatest; if the division position is not found, the width of the search box is gradually increased by a predetermined expansion width, and the bisection process is attempted again to search for a division position using the cloud of points within the search box; if the division position is found, the cloud of points within the search box is divided into two and one circumscribing rectangle is set; next, the bisection process is attempted using the remaining cloud of points excluding the range of the circumscribing rectangle that has already been set; and the attempts to perform the bisection process are repeated until the entire plane is completed.

[0037] This allows even complexly shaped planes to be divided appropriately, making it possible to generate a 3D model that represents complexly shaped planes. In this case, by appropriately adjusting the expansion width when gradually widening the search box, it is possible to avoid randomly subdividing the plane, and to generate a moderately abstract 3D model.

[0038] In addition, in a fourteenth aspect of the present invention, the processor is configured to execute a first stage of division processing for searching for division positions in the direction of a first coordinate axis on the plane, and a second stage of division processing for searching for division positions in the direction of a second coordinate axis.

[0039] This makes it possible to represent an object to be restored with a more complex shape in a three-dimensional model, compared to when searching for division positions in only one direction.

[0040] In addition, a fifteenth invention is configured such that, before dividing the conormal cluster into the sub-clusters, the processor performs an axis rotation process to rotate the coordinate axes relative to the conormal cluster in accordance with the inclination of an oriented bounding box set for the conormal cluster.

[0041] This makes it possible to prevent the shape of the object to be restored, which is represented by a rectangular plane in the three-dimensional model, from becoming unnatural. For example, in the case of an elongated object that extends in a direction significantly deviating from the direction of the coordinate axes, it is possible to prevent the object to be restored from being represented in a stepped shape in the three-dimensional model, which would be significantly different from the actual shape.

[0042] In addition, a sixteenth aspect of the present invention is configured such that the processor determines the necessity of the axis rotation processing by comparing the ratio of the orthogonal projection areas of the oriented bounding box and the axis-parallel bounding box set for the conormal cluster onto a horizontal plane with a predetermined threshold.

[0043] This makes it possible to avoid the generation of a complicated and difficult-to-view three-dimensional model caused by the axis rotation process being performed indiscriminately.

[0044] A seventeenth aspect of the present invention is a three-dimensional model generation method in which a processor executes a process for generating a three-dimensional model based on point cloud data acquired by performing three-dimensional reconstruction processing of a target location, the method comprising: extracting a plane from the point cloud of the target location; dividing the plane into a plurality of circumscribing rectangles so as to correspond to the shape of the plane; converting the point cloud included in the circumscribing rectangle into a rectangular plane corresponding to the circumscribing rectangle; and generating the three-dimensional model in which the target location is represented by a collection of a plurality of the rectangular planes.

[0045] According to this, as in the eighth aspect of the present invention, a plane (a set of points distributed on a plane at a predetermined density) representing an object to be restored placed at a target location is expressed in a 3D model by a plurality of rectangular planes corresponding to the shape of the object to be restored, so that a moderately abstract 3D model can be generated. Furthermore, since a moderately abstract 3D model is generated, the processing load on the processor can be reduced.

[0046] An eighteenth invention is a three-dimensional model generation program that causes a processor to execute a process for generating a three-dimensional model based on point cloud data obtained by performing three-dimensional reconstruction processing of a target location, and is configured to extract a plane from the point cloud of the target location, divide the plane into a plurality of circumscribing rectangles so as to correspond to the shape of the plane, convert the point cloud included in the circumscribing rectangle into a rectangular plane corresponding to the circumscribing rectangle, and generate the three-dimensional model that represents the target location as a collection of a plurality of the rectangular planes.

[0047] According to this, as in the eighth aspect of the present invention, a plane (a set of points distributed on a plane at a predetermined density) representing an object to be restored placed at a target location is expressed in a 3D model by a plurality of rectangular planes corresponding to the shape of the object to be restored, so that a moderately abstract 3D model can be generated. Furthermore, since a moderately abstract 3D model is generated, the processing load on the processor can be reduced.

[0048] A nineteenth invention made to solve the above-mentioned problems is a drawing creation display device that uses a processor to execute a process of creating and displaying a layout drawing from a three-dimensional model generated based on point cloud data obtained by performing a three-dimensional reconstruction process on a target location, wherein the processor sets coordinate axes for the point cloud of the target location, extracts from the point cloud of the target location a set of points whose set direction based on the coordinate axes and whose normal direction match within a predetermined tolerance range as a conormal cluster, converts the conormal cluster into a rectangular plane, and generates the three-dimensional model that represents the target location as a set of multiple rectangular planes, generates a two-dimensional layout drawing of the target location based on the three-dimensional model, and displays the two-dimensional layout drawing on a display.

[0049] This allows for the easy creation of a layout diagram that outlines the location to be restored with sufficient accuracy. Furthermore, since a moderately abstracted 3D model is generated and then converted into a 2D layout diagram, the processing load on the processor can be reduced.

[0050] In addition, in a twentieth invention, the processor is configured to generate the two-dimensional layout drawing by orthogonally projecting the three-dimensional model onto a horizontal plane after removing a floor portion and a ceiling portion from the three-dimensional model.

[0051] This makes it possible to generate a two-dimensional layout diagram that is easy to see and that allows for an overview of the restoration location.

[0052] In a twenty-first aspect of the present invention, the processor extracts a representative plane from the three-dimensional model, and inserts dimension lines relating to the representative plane into the two-dimensional layout drawing.

[0053] This eliminates the need for a user to insert dimension lines into a two-dimensional layout drawing using a CAD application, and also prevents the two-dimensional layout drawing from becoming cluttered and difficult to read due to the insertion of a large number of dimension lines into the two-dimensional layout drawing.

[0054] In addition, a 22nd invention is configured such that the processor displays on the display a layout drawing that visualizes the three-dimensional model composed of the rectangular planes, and in response to a user's operation of selecting two planes on the layout drawing, measures the inter-plane distance between the two selected planes based on the three-dimensional model, and displays the inter-plane distance on the display.

[0055] This makes it possible to easily measure the inter-surface distance between two opposing wall surfaces, etc., at the restored location and display it on a display. Furthermore, since the parallelism of two planes arranged parallel to each other at the restored location is also ensured in the 3D model by the conormal clustering process, the inter-surface distance between the two planes can be easily measured based on the user's simple operation of selecting two points on the target plane. Note that the layout diagram visualized from the 3D model may be either a 2D layout diagram or a 3D layout diagram.

[0056] In addition, a 23rd invention is configured such that, in response to a user's operation of selecting two planes on the layout drawing, the processor measures the inter-plane distance between the two selected planes based on the three-dimensional model, calculates the parallelism of the two selected planes based on the point cloud data, and displays the parallelism in addition to the inter-plane distance on the display.

[0057] This allows the user to easily check the parallelism of the two planes.

[0058] A 24th invention is a drawing creation and display method in which a processor executes a process of creating and displaying a layout drawing from a 3D model generated based on point cloud data obtained by performing a 3D reconstruction process on a target location, the method comprising the steps of: setting coordinate axes for the point cloud of the target location; extracting from the point cloud of the target location a set of points whose set direction based on the coordinate axes and whose normal direction coincide within a predetermined tolerance range as a conormal cluster; converting the conormal cluster into a rectangular plane; generating the 3D model that represents the target location as a set of multiple rectangular planes; generating a 2D layout drawing of the target location based on the 3D model; and displaying the 2D layout drawing on a display.

[0059] According to this, as in the nineteenth invention, a layout diagram showing an outline of the restoration location can be easily created with sufficient accuracy. In addition, since a moderately abstracted three-dimensional model is generated and then converted into a two-dimensional layout diagram, the processing load on the processor can be reduced.

[0060] A 25th invention is a drawing creation and display program that causes a processor to execute a process of creating and displaying a layout drawing from a 3D model generated based on point cloud data obtained by performing a 3D reconstruction process on a target location, the program setting coordinate axes for the point cloud of the target location, extracting from the point cloud of the target location a set of points whose set direction based on the coordinate axes and whose normal direction match within a predetermined tolerance range as a conormal cluster, converting the conormal cluster into a rectangular plane to generate the 3D model that represents the target location as a set of multiple rectangular planes, generating a 2D layout drawing of the target location based on the 3D model, and displaying the 2D layout drawing on a display.

[0061] According to this, as in the nineteenth invention, a layout diagram showing an outline of the restoration location can be easily created with sufficient accuracy. In addition, since a moderately abstracted three-dimensional model is generated and then converted into a two-dimensional layout diagram, the processing load on the processor can be reduced.

[0062] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0063] FIG. 1 is a diagram showing the overall configuration of a three-dimensional model generation system according to this embodiment.

[0064] This system comprises a user terminal 1 and a server 2 (a three-dimensional model generating device and a drawing creating and displaying device). The user terminal 1 and the server 2 are connected via a network such as a LAN or the Internet.

[0065] The user terminal 1 includes a device main body 11 and a camera 12. The user terminal 1 can be configured as a tablet terminal or a notebook PC.

[0066] The camera 12 is a visible camera, i.e., a monocular camera that detects visible light to capture an image of a subject, and outputs a captured image, for example, an RGB color image. Note that, for the three-dimensional reconstruction performed by the server 2, the user terminal 1 may be provided with sensors such as a depth camera that measures the distance to the subject and an IMU (Inertial Measurement Unit) that detects three-dimensional angular velocity and acceleration, in addition to the camera 12 (visible camera).

[0067] The user (worker) walks through the target location while holding the device main body 11 of the user terminal 1. At this time, the target location is photographed by the camera 12 of the user terminal 1, and photographed images of each point in the target location are sequentially acquired. Note that, although an example in which photographed images are acquired by the user walking through the target location while holding the device main body 11 has been shown, photographed images of the target location may also be acquired using a self-propelled robot or the like equipped with the user terminal 1.

[0068] The server 2 is configured as a PC. Based on the captured images received from the user terminal 1, the server 2 generates point cloud data as a 3D reconstruction result, which is 3D spatial information about the target location. The server 2 also generates a 3D model of the target location based on the point cloud data of the target location. The server 2 also generates a floor plan (2D layout diagram) of the target location based on the 3D model of the target location. The server 2 also displays the generated point cloud data, 3D model, and floor plan of the target location on the user terminal 1.

[0069] Next, a description will be given of the general configuration of the user terminal 1 and the server 2. Fig. 2 is a block diagram showing the general configuration of the user terminal 1 and the server 2. Fig. 3 is a flow diagram showing the procedure of the three-dimensional model generation process.

[0070] In addition to the camera 12 , the user terminal 1 includes a display 13 (display unit), an input device 14 , a storage unit 15 , a processor 16 (CPU), and a communication unit 17 .

[0071] The display 13 displays various operation screens related to shooting and editing operations, a setting screen for setting various processing conditions, and a viewing screen for checking the generated 3D reconstruction results (point cloud data) and 3D models. The input device 14 is used by the user to perform input operations. Note that when the user terminal 1 is configured as a tablet terminal, a touch panel display in which a touch panel as the input device 14 and a display panel as the display 13 are integrated is provided.

[0072] The storage unit 15 stores programs executed by the processor 16. The storage unit 15 also stores images captured by the camera 12. The captured images are transmitted to the server 2 at appropriate times.

[0073] The processor 16 executes the programs stored in the storage unit 15 to perform various processes related to photographing by the camera 12, screen display by the display 13, acquisition of operation information by the input device 14, and the like.

[0074] The communication unit 17 communicates with the server 2. Specifically, the communication unit 17 transmits to the server 2 images captured by the camera 12 and operation information acquired by a user operating the input device 14. The communication unit 17 also receives display information for various screens transmitted from the server 2.

[0075] The server 2 includes a communication unit 21 , a storage unit 22 , and a processor 23 .

[0076] The communication unit 21 communicates with the user terminal 1. Specifically, the communication unit 21 receives captured images and operation information transmitted from the user terminal 1. The communication unit 21 also transmits to the user terminal 1 display information for various screens to be displayed on the user terminal 1.

[0077] The storage unit 22 stores programs and the like to be executed by the processor 23. The storage unit 22 also stores point cloud data generated by the processor 23, data of three-dimensional models and two-dimensional models, and floor plans (layout drawings).

[0078] The processor 23 performs various processes by executing the programs stored in the storage unit 22. In this embodiment, the processor 23 performs 3D reconstruction processing, 3D model generation processing, drawing generation processing, drawing display processing, and the like.

[0079] In the three-dimensional reconstruction process, the processor 23 generates point cloud data as three-dimensional spatial information about the target location based on the captured image received from the user terminal 1. The three-dimensional reconstruction process can be realized, for example, by determining the camera position and orientation at the time of capture in a world coordinate system using the SLAM (Simultaneous Localization and Mapping) method, and then superimposing depth information obtained from a depth camera on the world coordinate system based on this camera position and orientation.

[0080] In the 3D model generation process, the processor 23 generates a 3D model of the target location based on the point cloud data generated in the 3D reconstruction process. In the 3D model generation process, the point cloud of the target location is sequentially subjected to preprocessing (ST101), clustering (ST102), and rectangularization (ST103) to generate the 3D model (see FIG. 3).

[0081] In the pre-processing, the processor 23 performs processes such as estimating the normals of each point that makes up the point cloud, extracting the floor surface, setting coordinate axes, and extracting flat surfaces on the point cloud of the target location generated by the 3D reconstruction process.

[0082] In the clustering process, the processor 23 detects a plane (a set of points distributed on a plane at a predetermined density) from the point cloud of the target location, and extracts the set of points that make up the plane as a cluster. In the clustering process, the processor 23 also divides the cluster into multiple sub-clusters.

[0083] In the rectangularization process, the processor 23 converts the clusters and sub-clusters obtained in the clustering process into rectangular planes, and generates a three-dimensional model that represents the target location as a collection of multiple rectangular planes.

[0084] The drawing generation process includes a plan view generation process and a dimension line insertion process.

[0085] In the floor plan generation process, the processor 23 generates a floor plan (2D layout diagram) of the restoration location based on the 3D model generated in the 3D model generation process. Specifically, the floor and ceiling portions are removed from the 3D model, and the 3D model is orthogonally projected onto the XY plane (horizontal plane), i.e., orthogonally projected in the Z direction, to generate the floor plan.

[0086] In the dimension line insertion process, the processor 23 extracts a representative plane from the three-dimensional model and inserts dimension lines related to the representative plane into the plan view. For example, a wall surface of the restoration location is extracted as a representative plane, and a dimension line representing the inter-surface distance between two opposing wall surfaces is inserted into the plan view.

[0087] The drawing display process includes a plan view display process and a measurement display process.

[0088] In the floor plan display process, the processor 23 displays the floor plan generated in the drawing generation process on the display 13 of the user terminal 1.

[0089] In the measurement display process, the processor 23 executes a measurement process related to the specified measurement object in response to a user operation to specify the measurement object, and displays the measurement results on the display 13 of the user terminal 1. Specifically, the inter-plane distance between two planes is measured based on a three-dimensional model, and the inter-plane distance is displayed. In addition, the parallelism of the two planes is calculated based on the point cloud data, and the parallelism of the two planes is displayed.

[0090] Next, a description will be given of the pre-processing (ST101 in FIG. 3) performed by the server 2. FIG. 4 is a flow chart showing the procedure of the pre-processing.

[0091] First, the server 2 performs normal estimation processing (ST201). In the normal estimation processing, normals of each point constituting the point cloud of the restoration location are estimated.

[0092] Next, floor surface estimation processing is performed (ST202). In the floor surface estimation processing, a horizontal plane that will become the floor surface is extracted from the point cloud at the restoration location. Specifically, points where the inner product of the normal vector at each point that makes up the point cloud and the Z-direction reference vector [0,0,1] exceeds a predetermined threshold (e.g., 0.9) are extracted as the horizontal plane. In addition, a histogram of the z coordinates of each point that makes up the point cloud is obtained, and the first and second peaks correspond to the floor and ceiling surfaces, and the one with the smaller z coordinate is recognized as the floor surface.

[0093] Next, a coordinate axis setting process is performed (ST203). In the coordinate axis setting process, coordinate axes (X-axis and Y-axis) are set for the point cloud of the restoration location in directions along the wall surface of the target location.

[0094] Next, flat surface extraction processing is performed (ST204). In the flat surface extraction processing, local features are calculated for the point cloud at the restoration location, and flat surfaces are extracted based on the local features. Principal component analysis is used to calculate the local features. First, a covariance matrix is ​​calculated from neighboring points of each point, and the eigenvalues ​​of the covariance matrix are calculated. The eigenvalues ​​are sorted in descending order and, assuming λ1, λ2, and λ3 in that order, points where (λ2-λ3) / λ1>0.3 are extracted. Subsequent processing, such as clustering, is performed on the extracted flat surfaces.

[0095] In addition, the preprocessing also includes a process for removing noise (isolated points) contained in the point cloud at the restoration location and a process for downsampling the point cloud. In the downsampling process, for example, the voxel size is set to 0.01 (1 cm). Furthermore, the entire point cloud is translated and rotated so that the floor is at z = 0, the center of the point cloud is the center, and the maximum vertical plane is in the X-axis direction.

[0096] Next, the coordinate axis setting process performed by the server 2 will be described. Fig. 5 is an explanatory diagram showing the state of the point cloud at the restoration location before and after the execution of the coordinate axis setting process. Fig. 6 is an explanatory diagram showing the state of the first and second coordinate axis setting processes. Figs. 7 and 8 are explanatory diagrams showing the gist of the second coordinate axis setting process.

[0097] In the coordinate axis setting process, coordinate axes (X-axis, Y-axis, and Z-axis) of a three-dimensional Cartesian coordinate system are set for the point cloud data of the restoration location acquired by the three-dimensional restoration process. In this embodiment, the coordinate axes (X-axis and Y-axis) are set for the point cloud of the restoration location in a direction along the wall surface of the target location. Note that the vertical direction of the point cloud can be set based on the detection data during imaging by the IMU provided in the user terminal 1, so the Z-axis is known. In other words, the direction perpendicular to the floor surface acquired in the floor surface estimation process is set as the Z-axis.

[0098] In the example shown in Fig. 5, the restoration location is a room with four walls constructed to form a rectangle in a plan view. Specifically, the restoration location is a conference room, with desks, chairs, etc. arranged inside the room. Note that the example shown in Fig. 5 shows the point cloud generated for the restoration location as viewed from directly above, but the point cloud that constitutes the ceiling has been deleted from the point cloud of the restoration location so that the interior state of the room can be seen.

[0099] In the example shown in Fig. 5A, the coordinate axes are provisionally set for the point cloud of the restoration location in a state inclined with respect to the wall surfaces of the restoration location. In this state, subsequent processing such as clustering processing cannot be performed appropriately. Therefore, in this embodiment, as shown in Fig. 5B, the coordinate axes (X-axis and Y-axis) are set so as to follow the wall surfaces of the restoration location. In other words, the coordinate axes (X-axis and Y-axis) are set in two directions perpendicular to the four wall surfaces that form a rectangle in a plan view.

[0100] The coordinate axis setting process includes a first coordinate axis setting process and a second coordinate axis setting process, and either the first coordinate axis setting process or the second coordinate axis setting process is executed. That is, it is determined whether the first coordinate axis setting process is appropriate, and if it is determined that the first coordinate axis setting process is appropriate, the first coordinate axis setting process is executed, and if it is determined that the first coordinate axis setting process is inappropriate, the second coordinate axis setting process is executed.

[0101] In the first coordinate axis setting process, the plane with the largest area detected from the point cloud of the restoration location is used as a reference plane, and the coordinate axes are set along this reference plane. At this time, a vertical plane is extracted from the point cloud of the target location. Specifically, points are extracted where the dot product of the normal of each point constituting the point cloud and the vertical normal [0,0,1] based on the coordinate axis is less than a threshold value (e.g., 0.5).

[0102] In the second coordinate axis setting process, the directions of the four wall surfaces (vertical surfaces) of the restoration location are estimated based on the distribution of normal directions across the entire point cloud of the restoration location, which forms a rectangle in plan view, and the coordinate axes are set based on the estimation results. In a typical indoor space, the four wall surfaces are constructed to form a rectangle in plan view. Furthermore, many indoor objects, such as desks and shelves, are arranged parallel to the walls. Therefore, the directions of the four wall surfaces of the restoration location can be estimated based on the distribution of normal directions across the entire point cloud of the restoration location.

[0103] Furthermore, in determining whether the first coordinate axis setting process is appropriate, it is determined whether the plane with the largest area detected from the point cloud of the restoration location is appropriate as a reference plane. For example, it is determined whether the plane with the largest area is sufficiently large compared to the overall size of the target location. Here, if it is determined that the first coordinate axis setting process is appropriate, the first coordinate axis setting process is executed, and if it is determined that the first coordinate axis setting process is inappropriate, the second coordinate axis setting process is executed.

[0104] 6A, the plane with the largest area is determined to be appropriate as the reference plane, and the first coordinate axis setting process is determined to be appropriate, so that the first coordinate axis setting process is executed, thereby appropriately setting the coordinate axes (X-axis and Y-axis) for the point cloud of the restoration location.

[0105] 6B, the plane with the largest area is inappropriate as the reference plane, and when the first coordinate axis setting process is performed, the coordinate axes (X-axis and Y-axis) are inappropriately set for the point cloud of the restoration location. That is, in the first coordinate axis setting process, the influence of the distortion of the reference plane spreads to the entire image, and there is a significant error between the directions of the coordinate axes (X-axis and Y-axis) and the directions of the wall surfaces of the restoration location.

[0106] In contrast, in the second coordinate axis setting process, the coordinate axes can be set appropriately for the entire wall surface, as shown in Fig. 6C. In this case, the error due to distortion occurring in the point cloud of the restored location is dispersed, and the coordinate axes are set so that the error between the directions of the coordinate axes (X-axis and Y-axis) and the directions of the wall surface of the restored location is minimized.

[0107] In the second coordinate axis setting process, the directions of the four wall surfaces (vertical surfaces) of the restoration location are estimated based on the distribution of normal directions for the entire point cloud of the restoration location, which is rectangular in plan view, and the coordinate axes are set based on the estimation results. Note that in the second coordinate axis setting process, since horizontal coordinate axes (X-axis and Y-axis) are set, only points whose normal directions are approximately horizontal, i.e., points representing vertical surfaces, are extracted as processing targets from among the points constituting the point cloud of the target location.

[0108] Specifically, as shown in Figure 7(A), if the normal vectors of each point constituting the point cloud at the restoration location are positioned so that their starting points are located at the origin of the coordinate axis, the end points of the normal vectors of each point, i.e., the points representing the normal directions of each point, will be lined up on a circumference (sphere), and the distribution of the normal directions of each point can be grasped.

[0109] The example shown in Figure 7(B) shows the distribution of the normal directions of each point constituting the point cloud of a restoration location where four wall surfaces are arranged to form a rectangle in a plan view. Here, the normal direction of each point constituting the point cloud is expressed in cylindrical coordinates (r, θ, z), and the normal angle θ (the azimuth angle of the normal vector) is expressed by the radial angle of the cylindrical coordinates. Furthermore, the points representing the normal directions of each point constituting the point cloud are concentrated in four locations, spaced 90 degrees apart, corresponding to the directions of the four wall surfaces.

[0110] Next, as shown in Fig. 8, the normal angle θ of each point on the cylindrical coordinate system is multiplied by four to obtain a quadrupled value 4θ. Here, four concentrated distribution points (see Fig. 7B) where points representing the normal directions of each point constituting the point cloud are gathered are aggregated into one concentrated distribution point.

[0111] Next, the optimal value of the quadruple value 4θ is estimated. Here, the points representing the normal directions of each point constituting the point cloud contain many outliers. Therefore, an estimation method that removes the influence of the outliers is applied to the quadruple value 4θ of each point to estimate the optimal value of the quadruple value 4θ. One possible estimation method is to remove the outliers using, for example, the RANSAC (Random Sample Consensus) method, find the center of gravity in cylindrical coordinates, and obtain the radial angle of the center of gravity.

[0112] Once the optimal value of the quadruple value 4θ is found in this way, the rotation angle φ that defines the direction of the coordinate axes is found by multiplying the optimal value of the quadruple value 4θ by ¼. The rotation angle φ thus found represents the inclination of a typical normal angle relative to the provisionally set horizontal coordinate axes (X-axis and Y-axis), i.e., the inclination of the four wall surfaces of the restored location. Therefore, by rotating the provisionally set horizontal coordinate axes (X-axis and Y-axis) by the rotation angle φ, the correct coordinate axes are set to align with the four wall surfaces of the restored location. The rotation angle φ is found within the range of -45 degrees to 45 degrees.

[0113] Furthermore, by determining the rotation angle φ, the directions of the two horizontal coordinate axes (X-axis and Y-axis) are determined, but the directions of the X-axis and Y-axis are indeterminate. Therefore, the X-axis and Y-axis are preferably set so that the point cloud of the restoration location becomes horizontally elongated after rotation. At this time, a bounding box (circumscribed rectangle) is preferably set for the point cloud of the restoration location, and it is determined whether the point cloud of the restoration location is horizontally elongated or vertically elongated. Here, if the point cloud of the restoration location is vertically elongated, the X-axis and Y-axis are set after rotating it by an additional 90 degrees from the rotation angle φ.

[0114] In this way, in the second coordinate axis setting process, coordinate axes are set based on the distribution of normal directions for the entire point cloud of the restoration location, and therefore, compared to the first coordinate axis setting process in which coordinate axes are set based on a specific plane, errors in the point cloud are dispersed, making it possible to appropriately set coordinate axes that follow the wall surfaces of the target location. Furthermore, because the coordinate axes are set with high precision for the point cloud, clusters can be appropriately extracted in the clustering process performed based on the coordinate axes.

[0115] However, depending on the restoration location, there may be no plane with an area suitable for the reference plane. In this case, it is difficult to set the coordinate axes using the first coordinate axis setting process, which sets the coordinate axes using the plane with the largest area as the reference plane. For example, in the case of a restoration location where a large number of furniture such as shelves and desks are arranged in a large room (e.g., a warehouse, office, or store), if the furniture such as shelves and desks are mainly photographed, the generated point cloud of the restoration location will contain almost no points representing the walls of the room. In this case, the point cloud of the restoration location does not contain a vertical plane with an area suitable for the reference plane.

[0116] On the other hand, furniture such as shelves and desks placed in a room are generally arranged parallel to the walls of the room, and the normal direction of each point constituting the point cloud representing the furniture such as shelves and desks is often perpendicular to or parallel to the walls of the room. Therefore, even if the point cloud of the restored location contains almost no points representing the walls of the room, the second coordinate axis setting process can appropriately set horizontal coordinate axes (X-axis and Y-axis) that are assumed to be along the walls of the target location. Furthermore, even if there is distortion in the point cloud representing the furniture such as shelves and desks, the distortion can be appropriately absorbed, allowing the coordinate axes to be appropriately set.

[0117] Next, the clustering process (ST102 in FIG. 3) performed by the server 2 will be described. Fig. 9 is a flow diagram showing the procedure of the clustering process. Fig. 10 is an explanatory diagram showing the status of the conormal clustering process. Fig. 11 is an explanatory diagram showing an overview of the conormal clustering process.

[0118] In the clustering process, as shown in Fig. 9A, first, individual clustering is performed (ST301), and then conormal clustering is performed (ST302).

[0119] In the individual clustering process, a plane (a set of points distributed in a plane at a predetermined density) in an arbitrary direction is detected from the point cloud of the target location, and the set of points that make up the plane is extracted as an individual cluster. In the individual clustering process, the plane in an arbitrary direction is detected using an estimation method that excludes the influence of outliers, such as RANSAC.

[0120] In the conormal clustering process, a group of planes (a set of points distributed in a planar shape at a predetermined density) in a predetermined direction based on the coordinate axes is detected from the point cloud of the target location (the predetermined direction will be described later). That is, a set of points whose predetermined direction and the point normal direction of the point cloud coincide within a predetermined tolerance range is detected as a group of planes. Furthermore, each plane is extracted from the group of planes as a conormal cluster. For example, a total of 21 predetermined directions are set in increments of 22.5 degrees.

[0121] The predetermined direction can be expressed as the components of a normal vector, for example, as follows: [1.,0.,0.],[0.,1.,0.],[0.7071,0.7071,0.],[-0.7071,0.7071,0.],[0.9239,0.3827,0.],[-0.3827,0.9239,0.],[0.3827,0.9239,0.],[-0.9239,0.3827,0.],[0.9239,0.,0.3827],[0.,0.9239,0.3827],[-0.9239, 0.,0.3827],[0.,-0.9239,0.3827],[0.7071,0.,0.7071],[0.,0.7071,0.7071],[-0.7071,0.,0.7071],[0.,-0.7071,0.7071],[0.3827,0.,0.9239],[0.,0.3827,0.9239],[-0.3827,0.,0.9239],[0.,-0.3827,0.9239],[0.,0.,1.].

[0122] In the conormal clustering process (ST302 in FIG. 9A), each process is executed in the procedure shown in FIG. 9B.

[0123] First, a conormal plane extraction process is performed (ST401). In the conormal plane extraction process, a plane (conormal plane) is extracted from the point cloud of the restoration location, which is a collection of points whose normal direction coincides with a predetermined direction based on the coordinate axes within an allowable range. Specifically, for each point constituting the point cloud of the restoration location, a point where the dot product of the point normal and the set normal exceeds a threshold value (e.g., 0.985) is extracted as a conormal plane. Here, the point normal refers to the normal of each point constituting the point cloud. Furthermore, the set normal refers to a normal (one of 21 possible directions) in a predetermined direction based on the coordinate axes. In other words, a group of planes whose normal is the set normal is extracted as a conormal plane in the predetermined direction.

[0124] Next, density-based clustering processing is performed (ST402). In the density-based clustering processing, the conormal planes extracted in the conormal plane extraction processing are grouped using a density-based clustering (DBSCAN: Density-based spatial clustering of applications with noise) technique, and conormal clusters are generated for each predetermined direction.

[0125] Next, a minimum cluster discarding process is performed (ST403). In the minimum cluster discarding process, among the conormal clusters obtained by the density-based clustering process, conormal clusters whose constituent points are less than a predetermined threshold are discarded. The threshold is set to, for example, 200 points. In this case, if voxelization is performed in units of 1 cm in the preprocessing, for example, clusters whose area is less than 200 cm are generally discarded.

[0126] Next, it is determined whether or not axis rotation processing is necessary (ST404). If it is determined that axis rotation processing is necessary (Yes in ST404), processing is performed to rotate the coordinate axes of the target cluster (axis rotation processing) (ST405).

[0127] Next, sub-clustering processing is performed (ST406). In the sub-clustering processing, the conormal clusters obtained in the density-based clustering processing (ST402) that have been subjected to the minimum cluster discarding processing (ST403) are divided into sub-clusters. Note that if axial rotation processing has been performed, after the sub-clustering processing is completed, the clusters are rotated in the opposite direction to the initial axial rotation.

[0128] Next, a cluster group generation process is performed (ST407). In the cluster group generation process, the conormal clusters (including those decomposed into sub-clusters) are grouped for each predetermined direction to generate conormal cluster groups. The conormal cluster groups are generated for each predetermined direction (set normal) based on the coordinate axes.

[0129] In the conormal clustering process, each time a conormal cluster is detected, the points included in that conormal cluster are excluded from the point cloud to be processed thereafter. At this time, the points included in the conormal cluster, i.e., the points included in the bounding box when the conormal cluster was detected, specifically the axis-aligned bounding box (AABB) or the oriented bounding box (OBB), are excluded. This prevents the same point from being included in different conormal clusters. Furthermore, since the points included in the conormal cluster are sequentially excluded from the point cloud of the target location as conormal clusters are detected, the order of detection of conormal clusters can be controlled to prioritize the detected plane directions. For example, if priority is given to the detection of vertical planes, the detection of conormal clusters for vertical planes may be started first, followed by the detection of conormal clusters for horizontal planes.

[0130] 9B shows the sub-clustering process in the conormal clustering process, but the sub-clustering process is also performed in the individual clustering process. Specifically, the sub-clustering process is performed on points included in the oriented bounding boxes detected as individual clusters in the individual clustering process.

[0131] The example shown in Fig. 10 is a case where a conormal clustering process is performed on the point cloud of the restored location shown in Fig. 5(B). In the conormal clustering process, the point cloud of the restored location is decomposed into conormal planes, and the conormal planes are grouped and extracted as conormal clusters. The conormal clusters are further grouped by predetermined direction (set normal). Note that in Fig. 10, the conormal clusters are drawn in different colors (densities).

[0132] The example shown in Figure 10(A) is a group of clusters whose normal direction is the X direction [1,0,0], i.e., clusters having conormal planes parallel to one wall surface. The example shown in Figure 10(B) is a group of clusters whose normal direction is the Y direction [0,1,0], i.e., clusters having conormal planes parallel to the other wall surface. The example shown in Figure 10(C) is a group of clusters whose normal direction is the Z direction [0,0,1], i.e., clusters having conormal planes parallel to the floor surface.

[0133] In the example shown in FIG. 11(A), the object to be restored is a cylinder with two planes joined in an L-shape and arranged inside it. The example shown in FIG. 11(B) is a group of clusters whose normal direction extracted from the point cloud of the target location shown in FIG. 11(A) is a predetermined direction [0,1,0]. In this case, three clusters belong to the group. The example shown in FIG. 11(C) is a group of clusters whose normal direction extracted from the point cloud of the target location is a predetermined direction [-1,1,0]. In this case, two clusters belong to the group. The example shown in FIG. 11(D) is a group of clusters whose normal direction extracted from the point cloud of the target location is a predetermined direction [0,0,1]. In this case, one cluster belongs to the group.

[0134] Next, a description will be given of the individual clustering process performed by the server 2. Fig. 12 is an explanatory diagram showing the status of the individual clustering process.

[0135] In the example shown in FIG. 12(A), partitions and a table are arranged in a room. One partition 51 is arranged in a different orientation from the other partitions and tables. The coordinate axes (X-axis and Y-axis) are set to follow the orientations of the partitions and tables other than partition 51. In this case, since partition 51 deviates from the set orientation based on the coordinate axes, when conormal clustering processing is performed, partition 51 is not properly represented in the 3D model after rectangularization processing, as shown in FIG. 12(B). In other words, the rectangular plane representing partition 51 has no continuity, and partition 51 is decomposed into rectangular planes 52 in two predetermined directions (e.g., 22.5 degrees and 45 degrees).

[0136] Therefore, in this embodiment, an individual clustering process is performed before the conormal clustering process, in which planes in any direction are individually extracted from the point cloud of the target location. As a result, in the 3D model after the rectangularization process, rectangular planes 53 representing the partitions 51 are arranged on a single plane, improving the state to be closer to the actual shape, as shown in Figure 12(C) .

[0137] In the individual clustering process, a plane of an arbitrary direction is extracted as an individual cluster from the point cloud of the target location. At this time, an oriented bounding box is set for the point cloud of the restored location, and points included in the oriented bounding box are extracted as an individual cluster. In addition, in the individual clustering process, for example, the RANSAC method is used as an estimation method to remove the influence of outliers.

[0138] In addition, the individual clustering process uses a planar patch detection technique. In planar patch detection, planar patches are detected based on the coplanarity angle. The coplanarity angle (coplanarity deg) defines the angle between the vector drawn from the center of the surface to a point cloud constituent point and the normal vector. A coplanarity angle of 90 degrees is the strictest, and in this case, only points completely on the plane are allowed. On the other hand, if the coplanarity angle is reduced, points outside the plane are also allowed. The coplanarity angle is set to, for example, 85 degrees. This plane detection technique is described in the literature "A. Araujo and M. Oliveira, A robust statistics approach for plane detection in unorganized point clouds, Pattern Recognition, 2020."

[0139] Here, in the individual clustering process, planes of any direction can be detected by setting an oriented bounding box, and in this case, a probabilistic search is performed using the RANSAC method. Therefore, in the individual clustering process, large planes can be detected from the point cloud of the restored location, but small planes may not be detected. On the other hand, the conormal clustering process cannot detect planes in directions that are significantly different from a predetermined direction based on the coordinate axes, but it can also detect small planes.

[0140] In this way, because individual clustering processing can extract planes in any direction, it compensates for the drawback of conormal clustering processing, which can only extract planes in a specified direction. On the other hand, while individual clustering processing cannot detect small planes, conormal clustering processing can detect even small planes as long as they are in a specified direction, so the drawback of individual clustering processing is compensated for by conormal clustering processing. As a result, by combining both conormal clustering processing and individual clustering processing, it is possible to appropriately extract the required clusters.

[0141] Furthermore, in this embodiment, the individual clustering process is performed before the conormal clustering process, and first, a large plane in an arbitrary direction is detected by the individual clustering process. Next, the conormal clustering process is performed. At this time, the points constituting the individual clusters acquired by the individual clustering process are excluded from the point cloud of the target location, and then the conormal clustering process is performed. This prevents the points constituting the individual clusters detected by the individual clustering process from being re-detected by the conormal clustering process. This makes it possible to avoid the generation of incorrect clusters caused by forcibly decomposing a plane in a direction deviating from a predetermined direction based on the coordinate axes into a plane in the predetermined direction.

[0142] Furthermore, although conormal clustering may detect many small planes, these are limited to planes in a specific direction based on the coordinate axes, so the three-dimensional model obtained by the rectangularization process is generated in an orderly manner without being overly complicated.

[0143] In the individual clustering process, a plane of any direction can be detected by setting a directional bounding box for the point cloud. However, since the normal direction obtained from the directional bounding box is based on the random number value provided by RANSAC and has low accuracy, it is advisable to redo the plane estimation for each point included in the directional bounding box to obtain the correct normal direction.

[0144] In the coordinate axis setting process, the coordinate axes (X-axis and Y-axis) are set so as to be parallel to the walls of a room that is rectangular in plan view. However, in the case of a room that is not rectangular in plan view, some of the walls may not be parallel to the coordinate axes. In this case, even the conormal clustering process may not be able to extract some of the walls. On the other hand, the individual clustering process can detect planes in any direction that deviates from the predetermined direction based on the coordinate axes, and therefore can be used to detect the walls of a room that is not rectangular in plan view.

[0145] In addition, if there is an object, such as partition 51 in the example shown in Figure 12, that is arranged in an orientation that is significantly different from the wall of the room and from many other objects that are arranged parallel to that wall, such an object will be an outlier in the second coordinate axis setting process based on the distribution of normal directions, and will not affect the setting of the coordinate axes.

[0146] Next, a description will be given of the axis rotation process performed by the server 2. Fig. 13 is an explanatory diagram showing the state of the axis rotation process.

[0147] In the example shown in FIG. 13A , a long, narrow handrail 61 is placed diagonally along a wall surface. The handrail 61 extends in a direction significantly different from the direction of the coordinate axes. Meanwhile, in the conormal clustering process, one cluster is set in the point cloud representing the handrail, and the cluster is then divided into multiple subclusters in the subclustering process. At this time, a bounding box (circumscribed rectangle) with sides aligned with the coordinate axes is set, and the cluster set for the entire handrail 61 is decomposed into bounding boxes aligned with the coordinate axes. Therefore, when the rectangularization process is performed after the clustering process, the handrail 61 is represented in a stepped shape by multiple rectangular planes 62, as shown in FIG. 13B , which significantly deviates from the actual shape.

[0148] Therefore, in this embodiment, before starting the sub-clustering process, an axis rotation process is performed to rotate the coordinate axes relative to the target cluster. As a result, as shown in Figure 13(C), multiple rectangular planes 63 representing the handrail 61 are aligned straight in the direction in which the handrail 61 extends, improving the shape to be closer to the actual shape.

[0149] The necessity of axial rotation processing is determined, and if axial rotation processing is necessary, axial rotation processing is performed. The necessity of axial rotation processing is determined based on the fitting rate of an oriented bounding box set in an arbitrary direction relative to the point cloud of the cluster. That is, axial rotation processing is performed if the fitting rate of the oriented bounding box is high. Furthermore, when axial rotation processing is performed, the orientation of the sides of the rectangular plane corresponding to the axially rotated cluster in the 3D model after rectangularization processing differs from that of other rectangular planes, which may make the 3D model appear cluttered and unsightly. Therefore, it is advisable to adjust a threshold value related to the fitting rate of the oriented bounding box to prevent excessive axial rotation processing. This allows the 3D model to be generated in an orderly and easy-to-view state.

[0150] In the axis rotation process, the angle by which the coordinate axes are rotated relative to the cluster is set based on the tilt of the axial direction of the oriented bounding box set for the point cloud of the cluster.

[0151] Specifically, first, the cluster point cloud is rotated so that the normal faces upward. Next, an axis-aligned bounding box and an oriented bounding box are set for the cluster point cloud. Next, the orthogonal projection areas of the axis-aligned bounding box and the oriented bounding box onto the XY plane (horizontal plane) are calculated. Next, if the ratio of the orthogonal projection areas of the axis-aligned bounding box and the oriented bounding box exceeds a predetermined threshold (e.g., 1.5), it is determined that axial rotation is necessary. If it is determined that axial rotation is necessary, the coordinate axes are rotated relative to the cluster according to the inclination of the oriented bounding box. After the sub-clustering process is completed, the cluster point cloud is subjected to axial rotation in the opposite direction to the initial rotation.

[0152] Next, the sub-clustering process performed by the server 2 will be described. Figures 14, 15, 16, and 17 are explanatory diagrams showing the status of the sub-clustering process. Note that in Figures 14, 15, 16, and 17, the sub-clusters are drawn in different colors (densities).

[0153] In the sub-clustering process, the conormal clusters obtained in the density-based clustering process and subjected to the smallest cluster discarding process are divided into a plurality of sub-clusters.

[0154] The example shown in Fig. 14 is a cluster whose normal direction is the X direction [1,0,0], i.e., a group of conormal clusters having a conormal plane parallel to one wall surface. When the subclustering process is performed on the conormal cluster shown in Fig. 14(A), the conormal cluster is divided into subclusters as shown in Fig. 14(B).

[0155] The example shown in Fig. 15 is a group of clusters whose normal direction is the Y direction [0,1,0], i.e., clusters having conormal planes parallel to the other wall surface. When the sub-clustering process is performed on the conormal cluster shown in Fig. 15(A), the conormal cluster is divided into sub-clusters as shown in Fig. 15(B).

[0156] The example shown in Fig. 16 is a group of clusters whose normal direction is the Z direction [0,0,1], i.e., clusters having conormal planes parallel to the floor surface. When the sub-clustering process is performed on the conormal cluster shown in Fig. 16(A), the conormal cluster is divided into sub-clusters as shown in Fig. 16(B).

[0157] The example shown in Figure 17 is a group of clusters whose normal direction is the Z direction [0,0,1], similar to the example shown in Figure 16, but is drawn as viewed from a different direction than the example shown in Figure 16.

[0158] Next, the procedure for the sub-clustering process will be explained. Figures 18, 19, 20, 21, 22, 23, 24, and 25 are explanatory diagrams showing the procedure for the sub-clustering process. In the examples shown in Figures 18 to 25, the clusters are assumed to be on the XY plane.

[0159] Clusters (conormal clusters and individual clusters) are planar and irregular in shape, and often have uneven density and holes. Therefore, in this embodiment, a cluster is divided into multiple sub-clusters that correspond to the shape of the cluster. This makes it possible to generate a 3D model that faithfully represents the shape of the cluster.

[0160] For example, if a cluster has an L-, T-, or U-shape, simply setting a bounding box (circumscribing rectangle) for the cluster results in the L-, T-, or U-shape being represented as a simple rectangle, resulting in a significantly insufficient ability to represent the shape. On the other hand, if the cluster is too faithful to its shape and a rectangular model with a sawtooth-like outline is obtained, this is undesirable for the original purpose, and therefore it is desirable to abstract it appropriately. Therefore, in this embodiment, one of the following two-division mode, N-division mode, and two-stage division mode is executed:

[0161] First, the two-division mode will be described. As shown in FIG. 18 , in the two-division mode, two bounding boxes BB1 and BB2 (circumscribed rectangles) are set for cluster C so as to divide cluster C into two. In addition, in the two-division mode, a division position that maximizes the amount of area reduction due to division is searched for. Specifically, a division position that maximizes the difference (S0-S1-S2) between the area S0 of the bounding box BB0 corresponding to the entire point cloud of the cluster before division and the sum of the areas S1 and S2 of the first and second bounding boxes BB1 and BB2 corresponding to the point cloud after division is searched for. In the example shown in FIG. 18 , the division position is searched for in the X direction.

[0162] The example shown in Figure 18(A) is a case where cluster C is L-shaped. In this case, the area reduction amount (S0-S1-S2) due to division is the largest and most optimal at division position b shown in Figure 18(B-2), compared to division position a shown in Figure 18(B-1) and division position c shown in Figure 18(B-3). Therefore, cluster C is divided into two sub-clusters SC at division position b. In this case, the L-shape of cluster C is represented by two rectangular planes set corresponding to the two bounding boxes BB1 and BB21 in the rectangularization process.

[0163] Here, even if the amount of area reduction by division is small, if a cluster is divided into sub-clusters, the resulting 3D model will be complex and difficult to see. Therefore, if the amount of area reduction by division (S0-S1-S2) is less than a predetermined threshold, division is not performed. The threshold is, for example, 0.25 m. ２ is set to

[0164] In this way, in the bisection mode, two bounding boxes (circumscribing rectangles) are set to divide the cluster in half along the coordinate axis, and the division position along the coordinate axis is searched for to maximize the area reduction of the bounding boxes due to the division. By dividing the cluster so as to reduce the total area of ​​the bounding boxes, it is possible to generate a 3D model that faithfully represents the shape of the cluster to a certain extent.

[0165] 18, the division positions are searched for in the X direction, but the division positions may also be searched for in the Y direction. Alternatively, the division positions may be searched for in both the X and Y directions, and the division position that results in the largest amount of area reduction may be adopted.

[0166] Next, an example will be described in which an appropriate division position cannot be set by a single search in one direction (X direction) in the two-division mode. The example shown in Fig. 19A is a case in which cluster C is T-shaped.

[0167] In this case, the area of ​​the bounding box is reduced in the case of division position a shown in Fig. 19(B-1) and division position c shown in Fig. 19(B-3). However, in the case of division position a shown in Fig. 19(B-1), division position b shown in Fig. 19(B-2), and division position c shown in Fig. 19(B-3), the T-shape of the cluster cannot be represented, and therefore, these are not optimal.

[0168] Next, an example will be described in which an appropriate division position can be set by searching twice in one direction (X direction) in the two-division mode. The example shown in Fig. 20(A) is a case in which cluster C is T-shaped, similar to the example shown in Fig. 19(A).

[0169] In this case, first, as shown in Figure 20(B), the point cloud of cluster C is divided a first time, and bounding boxes BB1 and BB2 are set. Next, as shown in Figures 20(C-1) and (C-2), the point cloud of bounding box BB2 is divided a second time, and bounding boxes BB21 and BB22 are set. As a result, as shown in Figure 20(D), three bounding boxes BB1, BB21, and BB22 are set for the point cloud of cluster C, and the point cloud of cluster C is divided into three sub-clusters SC. In this case, the T-shape of cluster C is represented by three rectangular planes set corresponding to the three bounding boxes BB1, BB21, and BB22 in the rectangularization process.

[0170] In this way, by repeating the search and division in one direction twice, a T-shaped cluster is expressed. In the example shown in Fig. 20, the search in the X direction is performed twice and the division is performed twice. On the other hand, the second search may be performed in a direction (Y direction) different from the first search. Alternatively, the first search may be performed in the Y direction and the second search in the X direction.

[0171] Next, an example in which an appropriate division position cannot be set in the two-division mode will be described. In the example shown in Fig. 21A, cluster C is U-shaped.

[0172] In this case, the areas of the bounding boxes BB1 and BB2 are not reduced and the U-shape of cluster C cannot be expressed in any of the cases of division position a shown in Fig. 21(B-1), division position b shown in Fig. 21(B-2), and division position c shown in Fig. 21(B-3), which are not optimal. Furthermore, the results are the same even if the search direction is changed.

[0173] Next, the N-division mode will be described. As shown in Figure 22, in the N-division mode, a search box SB (such as SB1 in the figure) is set for cluster C, and a division position that allows for division into two is searched for within the search box SB. If no division position is found within the search box SB, the width (search length) of the search box SB is gradually widened to L, 2L, and 3L. Furthermore, if a division position that allows for division into two is found within the search box SB, a bounding box BB (such as BB1 in the figure) is set so that the point cloud is divided at that division position. Next, a division position search is performed using the search box SB for the remaining points of cluster C, and this process is repeated throughout cluster C.

[0174] The example shown in FIG. 22(A) is a case where cluster C is U-shaped. In this case, first, as shown in FIG. 22(B-1), a search box SB1 having a width of search length L is set, and it is determined whether or not there is a division position by dividing the cluster into two. Here, it is determined that there is no division position. Next, as shown in FIG. 22(B-2), a search box SB2 having a width of search length 2L is set, and it is determined whether or not there is a division position by dividing the cluster into two. Here, it is also determined that there is no division position. Next, as shown in FIG. 22(B-3), a search box SB3 having a width of search length 3L is set, and it is determined whether or not there is a division position by dividing the cluster into two. Here, it is determined that there is a division position. Therefore, as shown in FIG. 22(C), the point cloud included in search box SB3 is divided into two. This results in setting a bounding box BB1.

[0175] Next, as shown in FIG. 23(A), a division position is searched for within the remaining portion of cluster C after removing the bounding box BB1. First, as shown in FIG. 23(B-1), a search box SB1 having a width of search length L is set, and it is determined whether or not there is a division position resulting from division into two, and it is determined that there is no division position. Next, as shown in FIG. 23(B-2), a search box SB2 having a width of search length 2L is set, and it is determined whether or not there is a division position resulting from division into two, and it is also determined that there is no division position. Next, as shown in FIG. 23(B-3), a search box SB3 having a width of search length 3L is set, and it is determined whether or not there is a division position resulting from division into two, and it is determined that there is a division position. Therefore, as shown in FIG. 23(C), the point cloud included in search box SB3 is divided into two. This results in a bounding box BB21 being set.

[0176] Next, as shown in FIG. 24(A), a division position is searched for in the portion remaining after removing bounding boxes BB1 and BB21 from cluster C. First, as shown in FIG. 24(B-1), a search box SB1 having a width of search length L is set, and it is determined whether or not there is a division position resulting from division into two, and it is determined that there is no division position. Next, as shown in FIG. 24(B-2), a search box SB2 having a width of search length 2L is set, and it is determined whether or not there is a division position resulting from division into two, and it is also determined that there is no division position. Next, as shown in FIG. 24(B-3), a search box SB3 having a width of search length 3L is set, and it is determined whether or not there is a division position resulting from division into two, and it is also determined that there is no division position. At this time, since there are no points beyond search box SB3 in the X direction, it is determined that the search has ended. Therefore, as shown in FIG. 24(C), a bounding box BB22 is set for the portion remaining after removing bounding boxes BB1 and BB21 from cluster C. As a result, three bounding boxes BB1, BB21, and BB22 are set for cluster C, and cluster C is divided into three sub-clusters SC. In this case, the U-shape of cluster C is expressed by three rectangular planes set in the rectangularization process corresponding to the three bounding boxes BB1, BB21, and BB22.

[0177] In this way, in the N-division mode, the width (search length) of the search box SB (such as SB1 in the figure) is gradually widened by the expansion width L, and a two-division process is attempted; that is, when a division position is found within the search box SB, the two-division process is performed at that division position and one bounding box BB (such as BB1 in the figure) is set, and then the same process is repeated for the remaining point clouds, and this process is repeated until a bounding box BB is set for the entire point cloud of cluster C.

[0178] In addition, in the N-division mode, when searching for a division position, a search box SB is set, and the division position is searched while gradually widening the width (search length) of the search box SB to L, 2L, and 3L. Therefore, only one division position can be set for each set search box SB. As a result, it may not be possible to express shapes smaller than the width of the search box SB, but this prevents the cluster from being randomly subdivided, allowing the shape of the cluster to be expressed at an appropriate level of abstraction. The expansion width L when gradually widening the width of the search box SB may be set appropriately depending on the restoration location, etc., but may be set to, for example, 0.5 m.

[0179] Note that the examples shown in Figures 22, 23, and 24 are for cases where the cluster is U-shaped, but in the N-division mode, even for clusters other than U-shaped, it is possible to divide the cluster by appropriately setting multiple bounding boxes (sub-clusters).

[0180] Next, the two-stage division mode will be described. In the two-stage division mode, the cluster is divided into two stages by changing the search direction. That is, the cluster is divided in a first stage along a first direction (X direction or Y direction) and in a second stage along a second direction different from the first direction.

[0181] In the example shown in FIG. 25(A), cluster C has a complex shape. In this case, as shown in FIGS. 25(B-1) and 25(C-1), a single division in only one direction, either the X direction or the Y direction, fails to set an appropriate bounding box for the cluster. On the other hand, as shown in FIGS. 25(B-2) and 25(C-2), when a second division is performed in a direction different from the first division, cluster C is appropriately divided into multiple sub-clusters SC. Here, in the example shown in FIGS. 25(B-1) and (B-2), division is first performed in the X direction, and then in the Y direction. Also, as shown in FIGS. 25(C-1) and (C-2), division is first performed in the Y direction, and then in the X direction.

[0182] Here, in the two-stage division mode, as in the two-stage division mode, a threshold value for the amount of area reduction when determining whether or not division is possible is set, and if the amount of area reduction due to division is less than the threshold value, division is not performed. Also, if the threshold value for the second stage division is set smaller than that for the first stage division, division is more likely to occur. For this reason, it is preferable to set separate threshold values ​​for determining whether or not division is possible in the first and second stages. For example, the threshold value is 0.25 m for the first division. ２ , 0.05m in the second division ２ may be set to

[0183] In this embodiment, the sub-clustering process for dividing a cluster (a conormal cluster or an individual cluster) obtained by the conormal clustering process or the individual clustering process into multiple sub-clusters can be performed in two-division mode, N-division mode, or two-stage division mode. However, these processes are not limited to the application of dividing a cluster into sub-clusters (sub-clustering).

[0184] That is, a clustering process may be performed in which a plane (a set of points distributed on a plane at a predetermined density) is extracted from the point cloud of the target location, and the plane is divided into clusters using a two-division mode, an N-division mode, or a two-stage division mode. This type of clustering process can be used, for example, to create a drawing (such as a front view or a plan view) of a single object to be restored from a point cloud of the object at the target location. Specifically, it may be used, for example, to create a schematic diagram of the exterior of a building.

[0185] Next, the rectangularization process (ST103 in FIG. 3) performed by the server 2 will be described. FIG. 26 is a flow diagram showing the procedure of the rectangularization process. Note that the process of this flow is performed in groups of conormal clusters and individual clusters generated in the clustering process. FIG. 27 is an explanatory diagram showing the status of the rectangularization process. FIGS. 28 and 29 are explanatory diagrams showing a three-dimensional model (rectangle model) generated in the rectangularization process.

[0186] In the rectangularization process, first, the clusters obtained in the clustering process (see ST102 in FIG. 3) are converted into rectangular planes (initial rectangularization process) (ST501). Here, the process is performed on individual clusters and their sub-clusters, and conormal clusters and their sub-clusters.

[0187] In the initial rectangularization process, the cluster is first rotated around the origin so that the normal points upward at [0,0,1]. This causes the rotated cluster to extend horizontally, simplifying subsequent processing. Next, the bounding box and center of gravity of the cluster are obtained, and the z coordinate of the bounding box is replaced with the z coordinate of the center of gravity. This results in a horizontal rectangular plane. Next, the rectangular plane is rotated in the opposite direction to the initial rotation.

[0188] In the example shown in Fig. 27 , similar to the example shown in Fig. 11 , the object to be restored is a cylinder in which two planes are arranged in an L-shape. In the initial rectangularization process, as shown in Fig. 27(A), clusters extracted from the point cloud of the object to be restored are converted into rectangular planes. Next, as shown in Fig. 27(A), a process is performed to adjust the position and shape of the rectangular planes. In the example shown in Fig. 27(A), an end of one rectangular plane protrudes from the other rectangular plane. In this case, by removing the protruding portion of one rectangular plane, the shape of the rectangular planes is adjusted so that two adjacent rectangular planes share a side, as shown in Fig. 27(B).

[0189] 26, as a process for adjusting the position and shape of rectangular planes, an alignment process is first performed (ST502). In the alignment process, rectangular planes belonging to the same group are targeted, and multiple rectangular planes that should be located on one plane are aligned on the same plane.

[0190] Next, a first inter-group snapping process is performed (ST503). In the first inter-group snapping process, two rectangular planes belonging to different groups are targeted, and if there is a gap between the two rectangular planes, the gap is filled to adjust the shapes of the rectangular planes so that the two rectangular planes share a common side.

[0191] Next, a second inter-group snapping process is performed (ST504). In the second inter-group snapping process, two rectangular planes belonging to different groups are targeted, and if there is a protrusion between the two rectangular planes, the protrusion is removed to adjust the shapes of the rectangular planes so that the two rectangular planes share a common side.

[0192] Next, intra-group snap processing is performed (ST505). In intra-group snap processing, rectangular planes belonging to the same group are targeted, and the shapes of the rectangular planes are adjusted by filling in gaps between two rectangular planes or cutting off portions of one rectangular plane that extend beyond the other rectangular plane.

[0193] In addition, the groups in the alignment processing, the first and second inter-group snap processing, and the intra-group snap processing represent groups of rectangular models (rectangle groups) when the clusters are converted into rectangular planes by the initial rectangularization processing, and correspond to groups of clusters (conormal clusters and individual clusters).

[0194] When the sub-clustering process is not performed, the rectangularization process is performed on the individual clusters and conormal clusters obtained by the clustering process, as shown in Fig. 28. On the other hand, when the sub-clustering process is performed, the rectangularization process is performed on the individual clusters and their sub-clusters, and the conormal clusters and their sub-clusters, as shown in Fig. 29.

[0195] As shown in Fig. 28, when the sub-clustering process is not performed, the fidelity of the objects at the restored location is low, and for example, a U-shaped desk placed in the center of a room is not sufficiently represented. On the other hand, as shown in Fig. 29, when the sub-clustering process is performed, the fidelity of the objects at the restored location is high, and for example, a U-shaped desk placed in the center of a room is sufficiently represented.

[0196] Next, a description will be given of the alignment processing performed by the server 2. Fig. 30 is an explanatory diagram showing the state of the alignment processing.

[0197] The alignment process involves aligning adjacent rectangular planes that belong to the same group (conormal rectangle group). The rectangular planes of each cluster obtained by the initial rectangle creation process may not be located on the same plane, even though they originally belong to a single plane (e.g., a wall, floor, or ceiling). If this is left unchecked, they will not be represented as a single line when drawn on a plan view (two-dimensional layout diagram). Therefore, the alignment process is performed.

[0198] When the alignment process is performed, for example, an unnatural state where there is a step between rectangular planes that make up indoor wall surfaces, i.e., rectangular planes that should be located on the same plane, is eliminated. In the example shown in Figure 30(A), there is a step between rectangular planes RP1 and RP2. When the alignment process is performed, as shown in Figure 30(B), the step between rectangular planes RP1 and RP2 disappears, and the rectangular planes RP1 and RP2 are aligned on the same plane.

[0199] In the alignment process, first, the rectangular planes are rotated so that their normals point upward. Next, the proximity between rectangular planes is determined, and adjacent planes are grouped together. Next, the z coordinates of the rectangular planes in each group are weighted averaged, and the z coordinate of the rectangular plane is replaced with this average value. At this time, the weighting is set according to the number of constituent points of the original cluster. Next, the rectangular planes are rotated in the opposite direction to the initial rotation. As a result, rectangular planes belonging to the same group are aligned on the same plane.

[0200] Next, a description will be given of the snap processing between the first and second groups performed by the server 2. Fig. 31 is an explanatory diagram showing the status of the snap processing between the first and second groups.

[0201] In the first inter-group snapping process, when two rectangular planes in different groups (conormal rectangle groups) are targeted and there is a gap between the two rectangular planes, the shapes of the rectangular planes are adjusted so that the two rectangular planes share an edge. Specifically, one edge (the adjacent edge) of the two rectangular planes is joined to the other face (the adjacent face) to eliminate the gap between the two rectangular planes.

[0202] When the first inter-group snapping process is performed, for example, an unnatural state in which there are gaps between rectangular planes representing each surface (such as the top surface or side surfaces) of a box-shaped object is eliminated. In the example shown in Fig. 31 (A-1), there are gaps between the rectangular planes RP1, RP2, and RP3. When the first inter-group snapping process is performed, as shown in Fig. 31 (A-2), the gaps between the rectangular planes RP1, RP2, and RP3 disappear, and the rectangular planes RP1, RP2, and RP3 are reshaped so that they share a side.

[0203] In the first inter-group snapping process, first, the rectangular planes are rotated so that their normals point upward. Next, proximity between the rectangular planes is determined. If the rectangular planes intersect, that is, if one of the rectangular planes protrudes from the other, this process is skipped and the process proceeds to the next, second inter-group snapping process. If there is a gap between the rectangular planes, the side (nearby side) of one rectangular plane is extended to join with the face (nearby face) of the other rectangular plane. At this time, the near face is extended as necessary. Next, the rectangular planes are rotated in the opposite direction to the initial rotation.

[0204] In the second inter-group snapping process, when two rectangular planes in different groups (conormal rectangle groups) are targeted and one of the two rectangular planes protrudes from the other, a process is performed to adjust the shape of the rectangular plane so that the two rectangular planes share an edge. Specifically, a process is performed to cut off the protruding portion. In this case, if the protruding amount is large and the protruding edge is not close to the surface of the other side, the second inter-group snapping process is not performed.

[0205] When the second inter-group snapping process is performed, for example, an unnatural state in which one of the rectangular planes representing each face (such as the top or side) of a box-shaped object protrudes from the other is eliminated. In the example shown in Fig. 31 (B-1), the rectangular plane RP1 protrudes from the rectangular plane RP2. When the second inter-group snapping process is performed, as shown in Fig. 31 (B-2), the protruding portion of the rectangular plane RP1 is removed, and the shape is adjusted so that the rectangular plane RP1 and the rectangular plane RP2 share a side.

[0206] In the second inter-group snapping process, first, the rectangular planes are rotated so that their normals point upward. Next, proximity between the rectangular planes is determined. If the rectangular planes do not intersect, that is, if one of the rectangular planes does not protrude beyond the other, this process is skipped. Next, the protruding portion of the edge (neighboring edge) of one of the rectangular planes is cut off. Next, the rectangular planes are rotated in the opposite direction to the initial rotation.

[0207] Next, a description will be given of the intra-group snap processing performed by the server 2. Fig. 32 is an explanatory diagram showing the status of intra-group snap processing.

[0208] In the intra-group snapping process, two adjacent rectangular planes that belong to the same group (conormal rectangle group) are targeted, and the shapes of the rectangular planes are adjusted.

[0209] In the example shown in Fig. 32(A-1), of two rectangular planes RP1 and RP2 belonging to the same group, one side of the rectangular plane RP1 slightly extends into the other rectangular plane RP2. In this case, as shown in Fig. 32(A-2), a process is performed to remove the extending portion of one rectangular plane RP1, thereby correcting the sides of the two rectangular planes RP1 and RP2 to be in contact with each other.

[0210] 32(B-1), among the rectangular planes RP1, RP2, and RP3 belonging to the same group, there is a small gap between the rectangular plane RP1 and the rectangular planes RP2 and RP3. In this case, as shown in FIG. 32(B-2), a process is performed to extend one of the rectangular planes RP1 so as to eliminate the gap between the rectangular planes, thereby correcting the sides of the three rectangular planes RP1, RP2, and RP3 to be in contact with each other.

[0211] Next, a description will be given of the floor plan generation process carried out by the server 2. Fig. 33 is an explanatory diagram showing the status of the floor plan generation process.

[0212] The server 2 generates a floor plan (2D layout drawing) by deleting the floor and ceiling parts, i.e., the rectangular planes representing the floor and ceiling, from the 3D model (rectangular model) and then orthogonally projecting the 3D model onto the XY plane (horizontal plane) (floor plan generation process). The floor and ceiling parts in the 3D model can be identified based on the z coordinate.

[0213] As shown in Fig. 33(A), the three-dimensional model is composed of a plurality of rectangular planes. Note that in the three-dimensional model shown in Fig. 33(A), the rectangular planes constituting the ceiling portion have been removed so that the interior state of the room where the image is restored can be shown. As shown in Fig. 33(B), the floor plan depicts the walls of the room where the image is restored, as well as furniture such as desks, chairs, and shelves arranged in the room.

[0214] In this embodiment, the majority of objects in the restoration location in the 3D model are represented by rectangular planes aligned along horizontal coordinate axes (X-axis and Y-axis), and the shapes of the objects to be restored are represented fairly faithfully, making it possible to obtain a reasonably abstract, orderly, and easy-to-read plan view (2D layout view).

[0215] Next, a description will be given of the dimension line insertion process performed by the server 2. Fig. 34 is an explanatory diagram showing the state of the dimension line insertion process.

[0216] The server 2 extracts a representative plane from the three-dimensional model and inserts dimension lines related to the representative plane into the floor plan (dimension line insertion process). In this process, for example, two planes spaced apart are extracted as representative planes in the three-dimensional model (rectangular model). For example, in the case of a room that is rectangular in plan view, the walls of the room are extracted. Furthermore, dimension lines representing the inter-plane distance between two opposing planes are inserted into the floor plan as dimension lines related to the representative planes. In the example shown in FIG. 34 , dimension lines representing the inter-plane distance between two pairs of wall surfaces that are opposed to each other in the directions of the coordinate axes (X-axis and Y-axis) in a rectangular room in plan view are inserted into the floor plan.

[0217] In this case, a combination of a predetermined number of planes may be selected in descending order of inter-plane distance. Alternatively, a combination of planes whose area is equal to or greater than a predetermined threshold may be selected. The threshold may be set based on the overall size of the restored location. This makes it possible to avoid the plan view becoming cluttered due to the insertion of numerous dimension lines into the plan view.

[0218] Next, a description will be given of the drawing display process performed by the server 2. Fig. 35 is an explanatory diagram showing the viewing screens 101 and 201 displayed on the user terminal 1.

[0219] In the user terminal 1, viewing screens 101 and 201 are displayed on the display 13 under the control of the server 2. Fig. 35(A) shows the viewing screen 101 in the plan view display mode, and Fig. 35(B) shows the viewing screen 201 in the measurement display mode.

[0220] As shown in Fig. 35(A), the viewing screen 101 in the plan view display mode displays a plan view 111 (two-dimensional layout drawing) generated by the drawing generation process. In the example shown in Fig. 35(A), the plan view 111 with dimension lines inserted is displayed. Note that a CAD mode may be provided so that editing operations can be performed on the plan view 111, such as erasing unnecessary lines and adding necessary lines.

[0221] A visualization of a three-dimensional model is displayed on the viewing screen 201 in the measurement display mode. In the example shown in Fig. 35(B), a three-dimensional layout drawing 211 is displayed, which visualizes a three-dimensional model (rectangular model) made up of multiple rectangular planes as viewed from a predetermined viewpoint.

[0222] Furthermore, the viewing screen 201 displays measurement results related to the specified measurement object in response to a user operation of specifying the measurement object on the three-dimensional layout drawing 211. In the example shown in Fig. 35(B) , a display box 212 for the inter-surface distance and a display box 213 for the parallelism are displayed at the time the user specifies two wall surfaces to be measured by a predetermined operation (e.g., a drag operation).

[0223] 35(B), a three-dimensional layout drawing 211 is displayed as a layout drawing that visualizes a three-dimensional model, but a two-dimensional layout drawing (floor plan) may also be displayed. Furthermore, on the viewing screen 201, when a user selects a plane to be measured (e.g., a wall surface of a room), the selected plane may be highlighted on the layout drawing. For example, the drawing color of the selected plane on the layout drawing may change.

[0224] Furthermore, rectangular planes may be drawn in different colors according to their normal directions in the three-dimensional layout diagram 211. For example, rectangular planes whose normal direction is the X direction may be drawn in red, rectangular planes whose normal direction is the Y direction may be drawn in green, and among rectangular planes whose normal direction is the Z direction, rectangular planes representing the floor may be drawn in gray, and rectangular planes representing surfaces other than the floor may be drawn in blue.

[0225] In the server 2, measurement and display processing for the inter-surface distance and measurement and display processing for the parallelism are performed in response to user operations on the user terminal 1, and a display box 212 for the inter-surface distance and a display box 213 for the parallelism are displayed on the viewing screen 201 of the user terminal 1.

[0226] In the measurement and display process for inter-surface distance, in response to the user selecting two planes on the three-dimensional layout drawing 211, the inter-surface distance between the two selected planes is measured based on the three-dimensional model, and the inter-surface distance is displayed in the display box 212 on the viewing screen 201.

[0227] In this case, the user simply designates one point on each of two planes (e.g., the walls of a room) to be used for measuring the inter-plane distance. The user simply selects a point on each plane as appropriate, and the direction connecting the two points designated by the user may deviate from the normal to the two planes. In this embodiment, the parallelism of two planes arranged parallel to each other at the restored location is ensured in the 3D model by the conormal clustering process. Therefore, once the user selects the two planes to be measured, the inter-plane distance between the two planes, i.e., the separation distance in the normal direction between the two planes, can be easily measured.

[0228] In this embodiment, the distance between two planes is measured, but the distance between one plane and a line parallel to that plane may also be measured. For example, the distance between a wall of a room and the edge (straight line) of a piece of furniture, such as a desk, placed parallel to that wall may also be measured.

[0229] Furthermore, in the parallelism measurement and display process, when a user selects two planes on the three-dimensional layout drawing, the parallelism of the two selected planes is calculated based on the point cloud data, and the parallelism of the two planes is displayed in a display box 213 on the viewing screen 201. In actual construction, there are cases where two planes that should be parallel are not constructed parallel, and in such cases, the user can easily check the parallelism of the two planes by selecting the two planes to be measured.

[0230] In this embodiment, when the user selects points on two parallel planes, the inter-plane distance between the two planes is measured and displayed on the viewing screen 201; however, when the user selects two points, the distance between the two points, i.e., the length of the straight line connecting the two points, may also be measured and displayed on the viewing screen 201.

[0231] In this embodiment, a 3D model is obtained in which the parallelism of parallel planes at the reconstruction location is ensured. This not only makes it easy to measure inter-surface distances but also makes it easy to combine 3D models. As a specific example, consider a case in which partially obtained 3D models of a square building are combined to construct a 3D model of the entire building. In the 3D model generated by the 3D model generation device according to the present disclosure, the coordinate axes are set in accordance with the wall surfaces of the building, and the constituent planes are limited to a predetermined direction. Therefore, in combining, only translation and three types of rotation (90-degree, 180-degree, and -90-degree rotations about the Z axis) need to be considered, making it easier to combine compared to models in which the orientation of the constituent planes is not limited.

[0232] The present embodiment is expected to be widely used as a fundamental technology for creating digital twins of real spaces. For example, the technology disclosed herein can be used to generate 3D models of factories, warehouses, stores, etc., and for a variety of purposes, such as facility and inventory management, renovations, layout change planning, and acquiring spatial data for simulations.

[0233] As described above, the embodiments have been described as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited to these, and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made. Furthermore, it is also possible to combine the components described in the above embodiments to create new embodiments.

[0234] The three-dimensional model generation device, three-dimensional model generation method, and three-dimensional model generation program disclosed herein have the effect of being able to generate a three-dimensional model that allows for easy measurement of inter-surface distances and combining of multiple three-dimensional models, and are useful as a three-dimensional model generation device, three-dimensional model generation method, and three-dimensional model generation program that use a processor to execute a process of generating a three-dimensional model based on point cloud data acquired by performing a three-dimensional reconstruction process on a target location.

[0235] 1: User terminal 2: Server 12: Camera 13: Display 14: Input device 16: Processor 51: Partition 52, 53: Rectangular plane 61: Handrail 62, 63: Rectangular plane 101: Viewing screen 111: Plan view (2D layout drawing) 201: Viewing screen 211: 3D layout drawing 212: Display box for inter-surface distance 213: Display box for parallelism BB, BB0, BB1, BB2, BB21, BB22: Bounding box C: Cluster L: Enlargement width RP1, RP2, RP3: Rectangular plane SB, SB1, SB2, SB3: Search box SC: Sub-cluster θ: Normal angle φ: Rotation angle

Claims

1. A three-dimensional model generation device that uses a processor to execute a process for generating a three-dimensional model based on point cloud data acquired by performing a three-dimensional reconstruction process on a target location, wherein the processor: sets coordinate axes along the walls of the target location for the point cloud of the target location; executes a conormal clustering process to extract, from the point cloud of the target location, a set of points whose normal direction coincides with a specified direction based on the coordinate axes within a specified tolerance range as a conormal cluster; and executes a rectangularization process to convert the conormal cluster into a rectangular plane, thereby generating the three-dimensional model that represents the target location as a set of a plurality of the rectangular planes.

2. The three-dimensional model generating device according to claim 1, characterized in that the processor estimates the directions of the four walls of the restoration location based on the distribution of normal directions for the entire point cloud of the restoration location, which is rectangular in a planar view, and sets the coordinate axes based on the estimation results.

3. The three-dimensional model generating device according to claim 2, characterized in that the processor: obtains, as a normal angle, the radial angle when the normal direction of each point constituting the point cloud is expressed in cylindrical coordinates; multiplies the normal angle of each point by four to obtain a quadrupled value of each point; estimates an optimal value of the quadrupled value of each point using an estimation method that excludes the influence of outliers; and multiplies the optimal value of the quadrupled value by one-fourth to obtain a rotation angle that defines the direction of the coordinate axes.

4. The three-dimensional model generating device of claim 1, characterized in that the processor performs an individual clustering process prior to the conormal clustering process, in which planes of any direction are extracted as individual clusters from the point cloud of the target location, and performs the rectangularization process on the individual clusters and the conormal clusters.

5. The three-dimensional model generating device according to claim 1, wherein the processor divides the conormal cluster into a plurality of sub-clusters corresponding to the shape of the conormal cluster, and converts the conormal cluster and the sub-clusters into the rectangular plane.

6. A three-dimensional model generation method in which a processor executes a process for generating a three-dimensional model based on point cloud data acquired by performing a three-dimensional reconstruction process on a target location, the three-dimensional model generation method comprising the steps of: setting coordinate axes along the walls of the target location for the point cloud of the target location; executing a conormal clustering process for extracting, from the point cloud of the target location, a set of points whose normal direction coincides with a specified direction based on the coordinate axes within a specified tolerance range as a conormal cluster; and executing a rectangularization process for converting the conormal cluster into a rectangular plane; and generating the three-dimensional model in which the target location is represented by a set of a plurality of the rectangular planes.

7. A three-dimensional model generation program in which a processor executes a process for generating a three-dimensional model based on point cloud data obtained by performing a three-dimensional reconstruction process on a target location, the three-dimensional model generation program comprising: setting coordinate axes along the walls of the target location for the point cloud of the target location; executing a conormal clustering process to extract, from the point cloud of the target location, a set of points whose normal direction coincides with a specified direction based on the coordinate axes within a specified tolerance range as a conormal cluster; and executing a rectangularization process to convert the conormal cluster into a rectangular plane, thereby generating the three-dimensional model in which the target location is represented by a set of a plurality of the rectangular planes.

8. A three-dimensional model generation device that uses a processor to execute a process for generating a three-dimensional model based on point cloud data acquired by performing a three-dimensional reconstruction process on a target location, wherein the processor: extracts a plane from the point cloud of the target location; divides the plane into a plurality of circumscribing rectangles so as to correspond to the shape of the plane; converts the point cloud included in the circumscribing rectangle into a rectangular plane corresponding to the circumscribing rectangle; and generates the three-dimensional model that represents the target location as a collection of a plurality of the rectangular planes.

9. The three-dimensional model generating device of claim 8, wherein the processor: sets coordinate axes along the walls of the target location for the point cloud of the target location; extracts a set of points from the point cloud of the target location whose set direction based on the coordinate axes and normal directions match within a predetermined tolerance range as a conormal cluster; divides the conormal cluster into a plurality of sub-clusters as the circumscribing rectangle; and converts the conormal cluster and the sub-clusters into rectangular planes.

10. The three-dimensional model generating device according to claim 8, characterized in that the processor sets two circumscribing rectangles so as to divide the plane in two, searches in the direction of the coordinate axes for a division position at which the amount of area reduction of the circumscribing rectangle due to the division is maximized, and when the division position is found, divides the plane in two at the division position and sets two circumscribing rectangles.

11. The three-dimensional model generating device according to claim 10, wherein the processor does not divide the plane if the amount of area reduction is less than a predetermined threshold value.

12. The three-dimensional model generating device according to claim 8, characterized in that the processor: sets two circumscribing rectangles to divide the plane in two, and attempts a bisection process to search in the direction of the coordinate axis for a division position that maximizes the amount of area reduction of the circumscribing rectangle due to the division; when the division position is found, divides the plane in two at that division position to set one circumscribing rectangle; and then attempts the bisection process to search for a division position in the same search direction for the remaining point cloud excluding the range of the set circumscribing rectangle.

13. The three-dimensional model generating device according to claim 8, characterized in that the processor: sets a search box of a predetermined width for the plane, sets two circumscribing rectangles to divide the cloud of points within the search box in two, and attempts a bisection process in which the division position in the direction of the coordinate axis will maximize the amount of area reduction of the circumscribing rectangle due to the division; if the division position is not found, gradually expands the width of the search box by a predetermined expansion width and attempts the bisection process again to search for a division position using the cloud of points within the search box; if the division position is found, divides the cloud of points within the search box in two to set one circumscribing rectangle; then attempts the bisection process on the remaining cloud of points excluding the range of the circumscribing rectangle that has already been set; and repeats the attempts of the bisection process until the entire plane is completed.

14. The three-dimensional model generating device according to claim 8, characterized in that the processor executes a first stage of division processing for searching for a division position in the direction of a first coordinate axis on the plane, and a second stage of division processing for searching for a division position in the direction of a second coordinate axis.

15. The three-dimensional model generating device according to claim 9, characterized in that the processor performs an axis rotation process to rotate the coordinate axes relative to the conormal cluster in accordance with the inclination of an oriented bounding box set for the conormal cluster before dividing the conormal cluster into the sub-clusters.

16. The three-dimensional model generating device according to claim 15, characterized in that the processor determines the necessity of the axis rotation processing by comparing the ratio of the orthogonal projection area onto the horizontal plane of the oriented bounding box and the axis-parallel bounding box set for the conormal cluster with a predetermined threshold value.

17. A three-dimensional model generation method in which a processor executes a process for generating a three-dimensional model based on point cloud data acquired by performing a three-dimensional reconstruction process on a target location, the three-dimensional model generation method comprising the steps of: extracting a plane from the point cloud of the target location; dividing the plane into a plurality of circumscribing rectangles so as to correspond to the shape of the plane; converting the point cloud contained in the circumscribing rectangle into a rectangular plane corresponding to the circumscribing rectangle; and generating the three-dimensional model in which the target location is represented by a collection of a plurality of the rectangular planes.

18. A three-dimensional model generation program in which a processor executes a process for generating a three-dimensional model based on point cloud data acquired by performing a three-dimensional reconstruction process on a target location, the three-dimensional model generation program comprising: extracting a plane from the point cloud of the target location; dividing the plane into a plurality of circumscribing rectangles so as to correspond to the shape of the plane; converting the point cloud included in the circumscribing rectangle into a rectangular plane corresponding to the circumscribing rectangle; and generating the three-dimensional model in which the target location is represented by a collection of a plurality of the rectangular planes.

19. A drawing creation display device that uses a processor to execute a process for creating and displaying a layout drawing from a three-dimensional model generated based on point cloud data obtained by performing a three-dimensional restoration process on a target location, wherein the processor: sets coordinate axes for the point cloud of the target location; extracts from the point cloud of the target location a set of points whose set direction based on the coordinate axes and whose normal direction match within a predetermined tolerance range as a conormal cluster; converts the conormal cluster into a rectangular plane; generates the three-dimensional model that represents the target location as a set of multiple rectangular planes; generates a two-dimensional layout drawing of the target location based on the three-dimensional model; and displays the two-dimensional layout drawing on a display.

20. The drawing creation and display device according to claim 19, characterized in that the processor generates the two-dimensional layout drawing by orthogonally projecting the three-dimensional model onto a horizontal plane after removing a floor and a ceiling from the three-dimensional model.

21. The drawing creation and display device according to claim 19, wherein the processor extracts a representative plane from the three-dimensional model, and inserts dimension lines relating to the representative plane into the two-dimensional layout drawing.

22. The drawing creation display device described in claim 19, characterized in that the processor displays on the display a layout drawing that visualizes the three-dimensional model composed of the rectangular planes, and in response to a user's operation of selecting two planes on the layout drawing, measures the inter-face distance between the two selected planes based on the three-dimensional model, and displays the inter-face distance on the display.

23. The drawing creation display device described in claim 19, characterized in that the processor, in response to a user's operation of selecting two planes on the layout drawing, measures the inter-surface distance between the two selected planes based on the three-dimensional model, calculates the parallelism of the two selected planes based on the point cloud data, and displays the parallelism on the display in addition to the inter-surface distance.

24. A drawing creation and display method in which a processor executes a process of creating and displaying a layout drawing from a three-dimensional model generated based on point cloud data obtained by performing a three-dimensional restoration process on a target location, the method comprising the steps of: setting coordinate axes for the point cloud of the target location; extracting a set of points from the point cloud of the target location whose set direction based on the coordinate axes and normal directions match within a predetermined tolerance range as a conormal cluster; converting the conormal cluster into a rectangular plane; generating the three-dimensional model in which the target location is represented by a set of multiple rectangular planes; generating a two-dimensional layout drawing of the target location based on the three-dimensional model; and displaying the two-dimensional layout drawing on a display.

25. A drawing creation and display program in which a processor executes a process of creating and displaying a layout drawing from a three-dimensional model generated based on point cloud data obtained by performing a three-dimensional restoration process on a target location, the drawing creation and display program comprising the steps of: setting coordinate axes for the point cloud of the target location; extracting a set of points from the point cloud of the target location whose set direction based on the coordinate axes and normal directions coincide within a predetermined tolerance range as a conormal cluster; converting the conormal cluster into a rectangular plane; generating the three-dimensional model that represents the target location as a set of multiple rectangular planes; generating a two-dimensional layout drawing of the target location based on the three-dimensional model; and displaying the two-dimensional layout drawing on a display.

Images