3D model generation device, 3D model generation method, and 3D model generation program

Abstract

To provide a 3D model generation device capable of generating a 3D model capable of easily measuring inter-face distances and combining multiple 3D models as well as reducing the processing load of a processor while creating clear and easy-to-read drawings by generating a moderately abstract 3D model.SOLUTION: With respect to a point cloud of an object point, a coordinate axis along the wall surface of the object point is set, and from the object point cloud, a set of points whose normal direction coincides with the set direction based on the coordinate axis within a predetermined tolerance is extracted as a conormal cluster. The conormal cluster is divided into multiple sub-clusters (bounding rectangles) that correspond to the shape of the conormal cluster. The conormal cluster and sub-cluster are converted into a rectangular plane and a 3D model is generated, in which an object point is represented as a set of multiple rectangular planes.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The present invention relates to a 3D model generation device, a 3D model generation method, and a 3D model generation program that use a processor to perform a process of generating a 3D model based on point cloud data acquired by performing a 3D reconstruction process on a target location. [Background technology]

[0002] Conventionally, a 3D restoration technique is known that generates point cloud data as 3D spatial information about a target location based on a photographed image of the target location. Furthermore, a technique is known that generates a 3D model of the target location from the point cloud data of the target location (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2022-142994 Summary of the Invention [Problem to be solved by the invention]

[0004] According to conventional technology, even users who are not familiar with algorithms for generating 3D models from point cloud data can easily and quickly generate 3D models suitable for radio wave propagation simulations.

[0005] However, 3D models generated from point cloud data can be used for a variety of purposes, not just radio wave propagation simulations. In particular, measurements such as the distance between two opposing walls can be performed based on the 3D model of a target location. Furthermore, multiple 3D models can be combined to generate a larger 3D model. Therefore, there is a demand for technology that can easily measure the distance between walls of a target location and combine multiple 3D models, but conventional technology has not addressed this need.

[0006] Furthermore, when generating a 3D model, it is desirable that the object to be restored be moderately abstracted. This allows for the creation of a clear and easy-to-read drawing when the 3D model is converted into a drawing, and also reduces the processing load on the processor when generating the 3D model.

[0007] Therefore, the main object of the present invention is to provide a 3D model generation device, a 3D model generation method, and a 3D model generation program that can generate 3D models that allow for easy measurement of inter-surface distances and merging of multiple 3D models. Another object of the present invention is to generate an appropriately abstract 3D model, thereby enabling the creation of neat, easy-to-read drawings and reducing the processing load on the processor. [Means for solving the problem]

[0008] The 3D model generation device of the present invention is a 3D model generation device that uses a processor to execute a process of generating a 3D model based on point cloud data acquired by performing 3D reconstruction processing of a target location, and the processor sets coordinate axes along the wall surfaces of the target location for the point cloud of the target location, and selects points from the point cloud of the target location where the set direction based on the coordinate axes and the normal direction match within a predetermined tolerance range. Detect the planes that form the set and create a conormal cluster based on the planes. and extracting the conormal clusters, Multiple Bounding Rectangles The conormal clusters and the sub-clusters are converted into rectangular planes, and the three-dimensional model is generated, in which the target location is represented by a set of a plurality of the rectangular planes.

[0009] Furthermore, the three-dimensional model generation method of the present invention is a three-dimensional model generation method in which a processor executes a process of generating a three-dimensional model based on point cloud data acquired by performing three-dimensional reconstruction processing of a target location, and the process includes setting coordinate axes along the wall surfaces of the target location for the point cloud of the target location, and selecting points from the point cloud of the target location where the set direction based on the coordinate axes and the normal direction coincide within a predetermined tolerance range. Detect the planes that form the set and create a conormal cluster based on the planes.and extracting the conormal clusters, Multiple Bounding Rectangles The conormal clusters and the sub-clusters are converted into rectangular planes, and the three-dimensional model is generated, in which the target location is represented by a set of a plurality of the rectangular planes.

[0010] The three-dimensional model generation program of the present invention is a three-dimensional model generation program that causes a processor to execute a process of generating a three-dimensional model based on point cloud data acquired by performing three-dimensional reconstruction processing of a target location, and that sets coordinate axes along the wall surfaces of the target location for the point cloud of the target location, and selects points from the point cloud of the target location where the set direction based on the coordinate axes and the normal direction match within a predetermined tolerance range. Detect the planes that form the set and create a conormal cluster based on the planes. and extracting the conormal clusters, Multiple Bounding Rectangles The conormal clusters and the sub-clusters are converted into rectangular planes, and the three-dimensional model is generated, in which the target location is represented by a set of a plurality of the rectangular planes. [Effects of the Invention]

[0011] According to the present invention, a plane (a set of points distributed on a plane at a predetermined density) representing an object to be restored placed at a target location is expressed in a 3D model by a plurality of rectangular planes corresponding to the shape of the object to be restored, so that a moderately abstract 3D model can be generated. Furthermore, since a moderately abstract 3D model is generated, the processing load on the processor can be reduced. [Brief explanation of the drawings]

[0012] [Figure 1] Overall configuration diagram of a 3D model generation system according to this embodiment [Figure 2] Block diagram showing the general configuration of a user terminal and a server [Figure 3] Flow diagram showing the steps of the 3D model generation process performed on the server [Figure 4] Flow diagram showing the preprocessing steps performed on the server [Figure 5]FIG. 10 is an explanatory diagram showing the state of the point cloud at the restoration location before and after the coordinate axis setting process performed by the server. [Figure 6] FIG. 10 is an explanatory diagram showing the status of the first and second coordinate axis setting processing performed by the server. [Figure 7] FIG. 10 is an explanatory diagram showing the procedure for setting a second coordinate axis performed by the server; [Figure 8] FIG. 10 is an explanatory diagram showing the procedure for setting a second coordinate axis performed by the server; [Figure 9] Flow diagram showing the clustering process performed by the server [Figure 10] An explanatory diagram showing the status of conormal clustering processing performed on the server. [Figure 11] An explanatory diagram showing an overview of the conormal clustering process performed on the server [Figure 12] An explanatory diagram showing an overview of the individual clustering processing performed by the server [Figure 13] An explanatory diagram showing the status of the axis rotation process performed by the server. [Figure 14] An explanatory diagram showing the status of sub-clustering processing performed on the server [Figure 15] An explanatory diagram showing the status of sub-clustering processing performed on the server [Figure 16] An explanatory diagram showing the status of sub-clustering processing performed on the server [Figure 17] An explanatory diagram showing the status of sub-clustering processing performed on the server [Figure 18] An explanatory diagram showing the sub-clustering process performed on the server [Figure 19] An explanatory diagram showing the sub-clustering process performed on the server [Figure 20] An explanatory diagram showing the sub-clustering process performed on the server [Figure 21] An explanatory diagram showing the sub-clustering process performed on the server [Figure 22] An explanatory diagram showing the sub-clustering process performed on the server [Figure 23]An explanatory diagram showing the sub-clustering process performed on the server [Figure 24] An explanatory diagram showing the sub-clustering process performed on the server [Figure 25] An explanatory diagram showing the sub-clustering process performed on the server [Figure 26] Flow diagram showing the procedure for rectangularization processing performed on the server [Figure 27] An explanatory diagram showing the state of rectangularization processing performed on the server. [Figure 28] An explanatory diagram showing a 3D model generated by rectangularization processing performed on the server. [Figure 29] An explanatory diagram showing a 3D model generated by rectangularization processing performed on the server. [Figure 30] An explanatory diagram showing an overview of the alignment process performed by the server [Figure 31] An explanatory diagram showing an overview of the snap processing between the first and second groups performed on the server. [Figure 32] An explanatory diagram showing an overview of the intra-group snap processing performed on the server [Figure 33] An explanatory diagram showing the status of floor plan generation processing performed on the server. [Figure 34] An explanatory diagram showing the state of dimension line insertion processing performed on the server [Figure 35] FIG. 10 is an explanatory diagram showing a viewing screen displayed on a user terminal. DETAILED DESCRIPTION OF THE INVENTION

[0013] The first invention made to solve the above problems is a 3D model generation device that uses a processor to execute processing to generate a 3D model based on point cloud data acquired by performing 3D reconstruction processing on a target location, wherein the processor sets coordinate axes along the wall surfaces of the target location for the point cloud of the target location, and selects points from the point cloud of the target location whose set direction based on the coordinate axes and normal directions coincide within a predetermined tolerance range. Detect the planes that form the set and create a conormal cluster based on the planes. and extracting the conormal clusters, Multiple Bounding RectanglesThe conormal clusters and the sub-clusters are converted into rectangular planes, and the three-dimensional model is generated, in which the target location is represented by a set of a plurality of the rectangular planes.

[0014] This allows the creation of a moderately abstract 3D model, since a plane (a set of points distributed on a plane at a predetermined density) representing the object to be restored placed at the target location is expressed in the 3D model as multiple rectangular planes corresponding to the shape of the object to be restored. Furthermore, since a moderately abstract 3D model is created, the processing load on the processor can be reduced.

[0016] This method ensures the parallelism of parallel planes at the reconstruction location, resulting in a 3D model. This makes it easy to measure the distance between surfaces. It also makes it easy to combine multiple 3D models. Furthermore, compared to when only conormal clusters are converted into rectangular planes, it is possible to generate a 3D model that faithfully represents the shape of the object to be reconstructed.

[0017] Also, Second The invention is configured such that the processor sets two circumscribing rectangles to divide the plane in two, searches in the direction of the coordinate axes for a division position that maximizes the amount of area reduction of the circumscribing rectangles due to the division, and when the division position is found, divides the plane in two at that division position and sets the two circumscribing rectangles.

[0018] This allows the plane representing the object to be restored to be appropriately divided into two and converted into a rectangular plane, making it possible to generate a moderately abstract 3D model.

[0019] Also, Third In the invention, the processor is configured not to divide the plane when the amount of area reduction is less than a predetermined threshold.

[0020] According to this, by appropriately adjusting the threshold for the amount of area reduction, it is possible to avoid randomly subdividing planes and generate a moderately abstract 3D model.

[0021] Also, Fourth The invention is configured such that the processor sets two circumscribing rectangles to divide the plane in two, and attempts a bisection process to search in the direction of the coordinate axis for a division position that maximizes the amount of area reduction of the circumscribing rectangle due to the division; when the division position is found, the processor divides the plane in two at that division position to set one circumscribing rectangle; and then attempts the bisection process to search for a division position in the same search direction for the remaining point cloud excluding the range of the set circumscribing rectangle.

[0022] In this case, since the bisection is performed multiple times, it is possible to represent an object to be restored with a more complex shape in a three-dimensional model compared to when the bisection is performed only once.

[0023] Also, Fifth In the invention, the processor sets a search box of a predetermined width for the plane, sets two circumscribing rectangles to divide the point cloud within the search box into two, and attempts a bisection process to search for a division position in the direction of the coordinate axis where the amount of area reduction of the circumscribing rectangle due to the division is greatest; if the division position is not found, the width of the search box is gradually increased by a predetermined expansion width, and the bisection process is attempted again to search for a division position for the point cloud within the search box; if the division position is found, the point cloud within the search box is divided into two to set one circumscribing rectangle, and then the bisection process is attempted for the remaining point cloud excluding the range of the circumscribing rectangle that has already been set, and the attempts to perform the bisection process are repeated until the entire plane is completed.

[0024] This allows even planes with complex shapes to be divided appropriately, making it possible to generate 3D models that represent complex planes. In this case, by appropriately adjusting the expansion width when gradually expanding the search box width, it is possible to avoid randomly subdividing the plane, and to generate a 3D model with an appropriate level of abstraction.

[0025] Also, Sixth The invention is configured such that the processor executes a first stage division process for searching for division positions in the direction of a first coordinate axis on the plane, and a second stage division process for searching for division positions in the direction of a second coordinate axis.

[0026] This makes it possible to represent an object to be restored with a more complex shape in a three-dimensional model, compared to when searching for division positions in only one direction.

[0027] Also, Seventh The invention is configured such that, before dividing the conormal cluster into the sub-clusters, the processor performs an axis rotation process to rotate the coordinate axes relative to the conormal cluster according to the inclination of an oriented bounding box set for the conormal cluster.

[0028] This makes it possible to prevent the shape of the object to be restored, which is represented by a rectangular plane in the 3D model, from becoming unnatural. For example, in the case of a long and thin object that extends in a direction that is significantly different from the direction of the coordinate axes, it is possible to prevent the object to be restored from being represented in a stepped shape in the 3D model, which would be significantly different from the actual shape.

[0029] Also, 8th The invention is configured such that the processor compares the ratio of the orthogonal projection areas of the oriented bounding box and the axis-parallel bounding box set for the conormal cluster onto a horizontal plane with a predetermined threshold to determine the necessity of the axis rotation processing.

[0030] This makes it possible to avoid the generation of a 3D model that is complicated and difficult to see due to the axis rotation process being performed indiscriminately.

[0031] A ninth aspect of the present invention is a three-dimensional model generation method for generating a three-dimensional model based on point cloud data acquired by performing three-dimensional reconstruction processing of a target location, by a processor, the method comprising: setting coordinate axes along the wall surfaces of the target location for the point cloud of the target location; and selecting points from the point cloud of the target location whose set directions based on the coordinate axes and normal directions coincide within a predetermined tolerance range. Detect the planes that form the set and create a conormal cluster based on the planes. and extracting the conormal clusters, Multiple Bounding Rectangles The conormal clusters and the sub-clusters are converted into rectangular planes, and the three-dimensional model is generated, in which the target location is represented by a set of a plurality of the rectangular planes.

[0032] According to this, as in the first invention, a plane (a set of points distributed on a plane at a predetermined density) representing the object to be restored placed at the target location is expressed in the 3D model by a plurality of rectangular planes corresponding to the shape of the object to be restored, so that a moderately abstract 3D model can be generated. Furthermore, since a moderately abstract 3D model is generated, the processing load on the processor can be reduced.

[0033] A tenth aspect of the present invention is a three-dimensional model generation program for causing a processor to execute a process of generating a three-dimensional model based on point cloud data acquired by performing three-dimensional reconstruction processing of a target location, the process including: setting coordinate axes along a wall surface of the target location for the point cloud of the target location; and selecting points from the point cloud of the target location where the set direction based on the coordinate axes and the normal direction coincide within a predetermined tolerance range. Detect the planes that form the set and create a conormal cluster based on the planes. and extracting the conormal clusters, Multiple Bounding Rectangles The conormal clusters and the sub-clusters are converted into rectangular planes, and the three-dimensional model is generated, in which the target location is represented by a set of a plurality of the rectangular planes.

[0034] According to this, as in the first invention, a plane (a set of points distributed on a plane at a predetermined density) representing the object to be restored placed at the target location is expressed in the 3D model by a plurality of rectangular planes corresponding to the shape of the object to be restored, so that a moderately abstract 3D model can be generated. Furthermore, since a moderately abstract 3D model is generated, the processing load on the processor can be reduced.

[0035] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0036] FIG. 1 is a diagram showing the overall configuration of a three-dimensional model generation system according to this embodiment.

[0037] This system includes a user terminal 1 and a server 2 (a three-dimensional model generating device and a drawing creating and displaying device). The user terminal 1 and the server 2 are connected via a network such as a LAN or the Internet.

[0038] The user terminal 1 includes a device main body 11 and a camera 12. The user terminal 1 can be configured as a tablet terminal or a notebook PC.

[0039] Camera 12 is a visible camera, i.e., a monocular camera that detects visible light to capture an image of a subject, and outputs the captured image, for example, an RGB color image. For the 3D reconstruction performed by server 2, user terminal 1 may be provided with sensors such as a depth camera that measures the distance to the subject and an IMU (Inertial Measurement Unit) that detects three-dimensional angular velocity and acceleration, in addition to camera 12 (visible camera).

[0040] The user (worker) walks through the target location while holding the device main body 11 of the user terminal 1. At this time, the target location is photographed by the camera 12 of the user terminal 1, and photographed images of each point in the target location are acquired sequentially. Note that, although an example has been shown in which the user walking through the target location while holding the device main body 11 acquires photographed images, it is also possible to acquire photographed images of the target location using a self-propelled robot or the like equipped with the user terminal 1.

[0041] The server 2 is configured as a PC. Based on the captured images received from the user terminal 1, the server 2 generates point cloud data as a 3D reconstruction result, which is 3D spatial information about the target location. The server 2 also generates a 3D model of the target location based on the point cloud data of the target location. The server 2 also generates a floor plan (2D layout drawing) of the target location based on the 3D model of the target location. The server 2 also displays the generated point cloud data, 3D model, and floor plan of the target location on the user terminal 1.

[0042] Next, a description will be given of the general configuration of the user terminal 1 and the server 2. Fig. 2 is a block diagram showing the general configuration of the user terminal 1 and the server 2. Fig. 3 is a flow diagram showing the procedure of the three-dimensional model generation process.

[0043] In addition to the camera 12, the user terminal 1 includes a display 13 (display unit), an input device 14, a storage unit 15, a processor 16 (CPU), and a communication unit 17.

[0044] The display 13 displays various operation screens related to shooting operations, editing operations, etc., a setting screen for setting various processing conditions, and a viewing screen for checking the generated 3D reconstruction results (point cloud data) and 3D models. The input device 14 is used by the user to perform input operations. When the user terminal 1 is configured as a tablet terminal, a touch panel display is provided in which a touch panel as the input device 14 and a display panel as the display 13 are integrated.

[0045] The storage unit 15 stores programs executed by the processor 16. The storage unit 15 also stores images captured by the camera 12. The captured images are transmitted to the server 2 at appropriate times.

[0046] The processor 16 executes the programs stored in the storage unit 15 to perform various processes related to the photographing by the camera 12, the screen display by the display 13, the acquisition of operation information by the input device 14, and the like.

[0047] The communication unit 17 communicates with the server 2. Specifically, the communication unit 17 transmits to the server 2 images captured by the camera 12 and operation information acquired by a user operating the input device 14. The communication unit 17 also receives display information of various screens transmitted from the server 2.

[0048] The server 2 includes a communication unit 21, a storage unit 22, and a processor 23.

[0049] The communication unit 21 communicates with the user terminal 1. Specifically, the communication unit 21 receives captured images and operation information transmitted from the user terminal 1. The communication unit 21 also transmits to the user terminal 1 display information for various screens to be displayed on the user terminal 1.

[0050] The storage unit 22 stores programs and the like executed by the processor 23. The storage unit 22 also stores point cloud data generated by the processor 23, data of three-dimensional models and two-dimensional models, floor plans (layout drawings), and the like.

[0051] The processor 23 performs various processes by executing programs stored in the storage unit 22. In this embodiment, the processor 23 performs 3D restoration processing, 3D model generation processing, drawing generation processing, drawing display processing, and the like.

[0052] In the 3D reconstruction process, the processor 23 generates point cloud data as 3D spatial information about the target location based on the captured image received from the user terminal 1. The 3D reconstruction process can be realized, for example, by determining the camera position and orientation at the time of capture in a world coordinate system using the SLAM (Simultaneous Localization and Mapping) method, and superimposing depth information obtained from a depth camera on the world coordinate system based on this camera position and orientation.

[0053] In the 3D model generation process, the processor 23 generates a 3D model of the target location based on the point cloud data generated in the 3D reconstruction process. In the 3D model generation process, the point cloud of the target location is sequentially subjected to preprocessing (ST101), clustering process (ST102), and rectangularization process (ST103), to generate the 3D model (see FIG. 3).

[0054] In the pre-processing, the processor 23 performs processes such as estimating the normals of each point that makes up the point cloud, extracting the floor surface, setting coordinate axes, and extracting flat surfaces on the point cloud of the target location generated by the 3D reconstruction process.

[0055] In the clustering process, the processor 23 detects a plane (a set of points distributed in a plane at a predetermined density) from the point cloud of the target location, and extracts the set of points that make up the plane as a cluster. In the clustering process, the processor 23 also divides the cluster into multiple sub-clusters.

[0056] In the rectangularization process, the processor 23 converts the clusters and sub-clusters acquired in the clustering process into rectangular planes, and generates a three-dimensional model that represents the target location as a set of multiple rectangular planes.

[0057] The drawing generation process includes a plan view generation process and a dimension line insertion process.

[0058] In the floor plan generation process, processor 23 generates a floor plan (2D layout drawing) of the restoration location based on the 3D model generated in the 3D model generation process. Specifically, the floor and ceiling parts are removed from the 3D model, and the 3D model is orthogonally projected onto the XY plane (horizontal plane), that is, orthogonally projected in the Z direction, to generate the floor plan.

[0059] In the dimension line insertion process, processor 23 extracts a representative plane from the 3D model and inserts dimension lines related to the representative plane into the plan view. For example, a wall surface of the restoration location is extracted as a representative plane, and a dimension line representing the inter-surface distance between two opposing wall surfaces is inserted into the plan view.

[0060] The drawing display process includes a plan view display process and a measurement display process.

[0061] In the floor plan display process, the processor 23 causes the display 13 of the user terminal 1 to display the floor plan generated in the drawing generation process.

[0062] In the measurement display process, the processor 23 executes a measurement process related to the specified measurement object in response to a user operation to specify the measurement object, and displays the measurement results on the display 13 of the user terminal 1. Specifically, the inter-plane distance between two planes is measured based on a three-dimensional model, and the inter-plane distance is displayed. In addition, the parallelism of the two planes is calculated based on the point cloud data, and the parallelism of the two planes is displayed.

[0063] Next, a description will be given of the pre-processing (ST101 in FIG. 3) performed by the server 2. FIG. 4 is a flow diagram showing the procedure of the pre-processing.

[0064] First, the server 2 performs normal estimation processing (ST201). In the normal estimation processing, the normals of the points constituting the point cloud of the restoration location are estimated.

[0065] Next, floor surface estimation processing is performed (ST202). In floor surface estimation processing, a horizontal plane that will become the floor surface is extracted from the point cloud at the restoration location. Specifically, points where the dot product of the normal vector at each point that makes up the point cloud and the Z-direction reference vector [0,0,1] exceeds a predetermined threshold (e.g., 0.9) are extracted as horizontal surfaces. In addition, a histogram of the z coordinates of each point that makes up the point cloud is obtained, and the first and second peaks correspond to the floor and ceiling surfaces, and the one with the smaller z coordinate is recognized as the floor surface.

[0066] Next, a coordinate axis setting process is performed (ST203). In the coordinate axis setting process, coordinate axes (X-axis and Y-axis) are set for the point cloud of the restoration location in directions along the wall surface of the target location.

[0067] Next, flat surface extraction processing is performed (ST204). In flat surface extraction processing, local features are calculated for the point cloud at the restoration location, and flat surfaces are extracted based on the local features. Principal component analysis is used to calculate the local features. First, a covariance matrix is ​​calculated from neighboring points of each point, and the eigenvalues ​​of the covariance matrix are calculated. The eigenvalues ​​are sorted in descending order, and if λ1, λ2, and λ3 are used in that order, points where (λ2-λ3) / λ1>0.3 are extracted. Subsequent processing, such as clustering, is performed on the extracted flat surfaces.

[0068] In addition, preprocessing also removes noise (isolated points) from the point cloud at the restoration location and downsamples the point cloud. In downsampling, the voxel size is set to 0.01 (1 cm), for example. The entire point cloud is also translated and rotated so that the floor is at z=0, the center of the point cloud is the center, and the maximum vertical plane is in the X-axis direction.

[0069] Next, the coordinate axis setting process performed by the server 2 will be described. Fig. 5 is an explanatory diagram showing the state of the point cloud at the restoration location before and after the execution of the coordinate axis setting process. Fig. 6 is an explanatory diagram showing the state of the first and second coordinate axis setting processes. Figs. 7 and 8 are explanatory diagrams showing the gist of the second coordinate axis setting process.

[0070] In the coordinate axis setting process, coordinate axes (X-axis, Y-axis, and Z-axis) of a three-dimensional Cartesian coordinate system are set for the point cloud data of the restoration location acquired by the three-dimensional restoration process. In this embodiment, the coordinate axes (X-axis and Y-axis) are set for the point cloud of the restoration location in a direction along the wall surface of the target location. Note that the vertical direction of the point cloud can be set based on the detection data during imaging by the IMU provided in the user terminal 1, so the Z-axis is known. In other words, the direction perpendicular to the floor surface acquired in the floor surface estimation process is set as the Z-axis.

[0071] In the example shown in Figure 5, the restoration location is a room with four walls constructed to form a rectangle in a plan view. Specifically, the restoration location is a conference room, with desks, chairs, etc. arranged inside the room. Note that the example shown in Figure 5 shows the point cloud generated for the restoration location as viewed from directly above, but the point cloud that constitutes the ceiling has been deleted from the point cloud of the restoration location so that the interior state of the room can be seen.

[0072] In the example shown in Fig. 5(A), the coordinate axes are provisionally set for the point cloud of the restoration location in a state inclined with respect to the wall surfaces of the restoration location. In this state, subsequent processes such as clustering cannot be performed appropriately. Therefore, in this embodiment, as shown in Fig. 5(B), the coordinate axes (X-axis and Y-axis) are set so as to follow the wall surfaces of the restoration location. In other words, the coordinate axes (X-axis and Y-axis) are set in two directions perpendicular to the four wall surfaces that form a rectangle in plan view.

[0073] The coordinate axis setting process includes a first coordinate axis setting process and a second coordinate axis setting process, and either the first coordinate axis setting process or the second coordinate axis setting process is executed. That is, it is determined whether the first coordinate axis setting process is appropriate, and if it is determined that the first coordinate axis setting process is appropriate, the first coordinate axis setting process is executed, and if it is determined that the first coordinate axis setting process is inappropriate, the second coordinate axis setting process is executed.

[0074] In the first coordinate axis setting process, the plane with the largest area detected from the point cloud of the restoration location is used as the reference plane, and the coordinate axes are set along this reference plane. At this time, a vertical plane is extracted from the point cloud of the target location. Specifically, points are extracted where the dot product of the normal of each point constituting the point cloud and the vertical normal [0,0,1] based on the coordinate axis is less than a threshold value (e.g., 0.5).

[0075] In the second coordinate axis setting process, the directions of the four wall surfaces (vertical surfaces) of the restoration location are estimated based on the distribution of normal directions across the entire point cloud of the restoration location, which forms a rectangle in plan view, and the coordinate axes are set based on the estimation results. In a typical indoor space, the four wall surfaces are constructed to form a rectangle in plan view. Furthermore, many objects, such as desks and shelves, placed indoors are arranged parallel to the walls. Therefore, the directions of the four wall surfaces of the restoration location can be estimated based on the distribution of normal directions across the entire point cloud of the restoration location.

[0076] In addition, in determining whether the first coordinate axis setting process is appropriate, it is determined whether the plane with the largest area detected from the point cloud of the restoration location is appropriate as a reference plane. For example, it is determined whether the plane with the largest area is sufficiently large compared to the overall size of the target location. Here, if it is determined that the first coordinate axis setting process is appropriate, the first coordinate axis setting process is executed, and if it is determined that the first coordinate axis setting process is inappropriate, the second coordinate axis setting process is executed.

[0077] In the example shown in Figure 6(A), the plane with the largest area is determined to be appropriate as the reference plane, and the first coordinate axis setting process is determined to be appropriate, so that the first coordinate axis setting process is executed, thereby appropriately setting the coordinate axes (X-axis and Y-axis) for the point cloud of the restoration location.

[0078] 6(B), the plane with the largest area is inappropriate as the reference plane, and when the first coordinate axis setting process is performed, the coordinate axes (X-axis and Y-axis) are inappropriately set for the point cloud of the restoration location. That is, in the first coordinate axis setting process, the influence of the distortion of the reference plane spreads to the entire image, and there is a significant error between the directions of the coordinate axes (X-axis and Y-axis) and the directions of the wall surfaces of the restoration location.

[0079] In contrast, in the second coordinate axis setting process, the coordinate axes can be set appropriately for the entire wall surface, as shown in Fig. 6(C). In this case, the error due to distortion occurring in the point cloud of the restored location is dispersed, and the coordinate axes are set so that the error between the directions of the coordinate axes (X-axis and Y-axis) and the directions of the wall surface of the restored location is minimized.

[0080] In the second coordinate axis setting process, the directions of the four wall surfaces (vertical surfaces) of the restoration location are estimated based on the distribution of normal directions for the entire point cloud of the restoration location, which is rectangular in plan view, and the coordinate axes are set based on the estimation results. Note that in the second coordinate axis setting process, since horizontal coordinate axes (X-axis and Y-axis) are set, only points whose normal directions are approximately horizontal, i.e., points representing vertical surfaces, are extracted as processing targets from among the points that make up the point cloud of the target location.

[0081] Specifically, as shown in Figure 7(A), if the normal vectors of each point constituting the point cloud at the restoration location are positioned so that their starting points are located at the origin of the coordinate axis, the end points of the normal vectors of each point, i.e., the points representing the normal directions of each point, will be lined up on a circumference (sphere), and the distribution of the normal directions of each point can be grasped.

[0082] The example shown in Figure 7(B) represents the distribution of the normal directions of each point constituting the point cloud of a restoration location where four wall surfaces are arranged to form a rectangle in a plan view. Here, the normal direction of each point constituting the point cloud is expressed in cylindrical coordinates (r, θ, z), and the normal angle θ (azimuth angle of the normal vector) is expressed by the radial angle of the cylindrical coordinates. In addition, the points representing the normal directions of each point constituting the point cloud are concentrated in four locations, spaced 90 degrees apart, corresponding to the directions of the four wall surfaces.

[0083] Next, as shown in Figure 8, the normal angle θ of each point on the cylindrical coordinate system is multiplied by four to obtain a quadrupled value, 4θ. Here, the four concentrated distribution points (see Figure 7(B)) where points representing the normal directions of each point constituting the point cloud are gathered are aggregated into one concentrated distribution point.

[0084] Next, the optimal value of the quadruple value 4θ is estimated. Here, the points representing the normal directions of each point constituting the point cloud contain many outliers. Therefore, an estimation method that removes the influence of outliers is applied to the quadruple value 4θ of each point to estimate the optimal value of the quadruple value 4θ. One possible estimation method is to remove outliers using, for example, the RANSAC (Random Sample Consensus) method, find the center of gravity in cylindrical coordinates, and obtain the radial angle of the center of gravity.

[0085] Once the optimal value of the quadruple value 4θ is found in this way, the rotation angle φ that defines the direction of the coordinate axes is found by multiplying that optimal value by 1 / 4. The rotation angle φ thus found represents the inclination of the typical normal angle relative to the provisionally set horizontal coordinate axes (X-axis and Y-axis), i.e., the inclination of the four wall surfaces of the restored location. Therefore, by rotating the provisionally set horizontal coordinate axes (X-axis and Y-axis) by the rotation angle φ, the correct coordinate axes are set to align with the four wall surfaces of the restored location. The rotation angle φ can be found within the range of -45 degrees to 45 degrees.

[0086] Furthermore, by determining the rotation angle φ, the directions of the two horizontal coordinate axes (X-axis and Y-axis) are determined, but the direction of the X-axis and Y-axis is indeterminate. Therefore, the X-axis and Y-axis should be set so that the point cloud of the restoration location becomes horizontally elongated after rotation. At this time, a bounding box (circumscribed rectangle) should be set for the point cloud of the restoration location, and it should be determined whether the point cloud of the restoration location is horizontally elongated or vertically elongated. Here, if the point cloud of the restoration location is vertically elongated, the X-axis and Y-axis are set after rotating it by an additional 90 degrees from the rotation angle φ.

[0087] In this way, in the second coordinate axis setting process, coordinate axes are set based on the distribution of normal directions for the entire point cloud of the restoration location, so errors in the point cloud are dispersed compared to the first coordinate axis setting process, which sets coordinate axes based on a specific plane, making it possible to properly set coordinate axes that follow the wall surfaces of the target location. Furthermore, because coordinate axes are set with high precision for the point cloud, clusters can be properly extracted in the clustering process, which is performed based on the coordinate axes.

[0088] However, depending on the restoration location, there may be no plane with an area suitable for the reference plane. In this case, it is difficult to set the coordinate axes in the first coordinate axis setting process, which sets the coordinate axes using the plane with the largest area as the reference plane. For example, in the case of a restoration location where many pieces of furniture such as shelves and desks are arranged in a large room (e.g., a warehouse, office, or store), if the images mainly show the furniture such as shelves and desks, the generated point cloud of the restoration location will contain almost no points representing the walls of the room. In this case, the point cloud of the restoration location does not contain a vertical plane with an area suitable for the reference plane.

[0089] On the other hand, furniture such as shelves and desks placed in a room are generally arranged parallel to the walls of the room, and the normal direction of each point constituting the point cloud representing the furniture such as shelves and desks is often perpendicular to or parallel to the walls of the room. Therefore, even if the point cloud of the restoration location contains almost no points representing the walls of the room, the second coordinate axis setting process can appropriately set horizontal coordinate axes (X-axis and Y-axis) that are assumed to be along the walls of the target location. Furthermore, even if there is distortion in the point cloud representing the furniture such as shelves and desks, the distortion can be appropriately absorbed, allowing the coordinate axes to be appropriately set.

[0090] Next, the clustering process (ST102 in FIG. 3) performed by the server 2 will be described. FIG. 9 is a flow diagram showing the procedure of the clustering process. FIG. 10 is an explanatory diagram showing the status of the conormal clustering process. FIG. 11 is an explanatory diagram showing an overview of the conormal clustering process.

[0091] In the clustering process, as shown in Fig. 9(A), first, individual clustering is performed (ST301), and then conormal clustering is performed (ST302).

[0092] In the individual clustering process, a plane (a set of points distributed in a plane at a specified density) in an arbitrary direction is detected from the point cloud of the target location, and the set of points that make up that plane is extracted as an individual cluster. In the individual clustering process, the plane in an arbitrary direction is detected using an estimation method that eliminates the influence of outliers, such as RANSAC.

[0093] In the conormal clustering process, a group of planes (a set of points distributed in a planar shape at a predetermined density) in a predetermined direction based on the coordinate axes is detected from the point cloud of the target location (the predetermined direction will be described later). In other words, a set of points whose predetermined direction and the point normal direction of the point cloud match within a predetermined tolerance range is detected as a group of planes. Furthermore, each plane is extracted from the group of planes as a conormal cluster. For example, a total of 21 predetermined directions are set in increments of 22.5 degrees.

[0094] The predetermined direction can be expressed as the components of a normal vector, for example, as follows: [1.,0.,0.],[0.,1.,0.],[0.7071,0.7071,0.],[-0.7071,0.7071,0.],[0.9239,0.3827,0.],[-0.3827,0.9239,0.],[0.3827,0.9239,0.],[-0.9239,0.3827,0.],[0.9239,0.,0.3827],[0.,0.9239,0.3827],[-0.9239, 21 possible settings are possible: [0.,0.3827],[0.,-0.9239,0.3827],[0.7071,0.,0.7071],[0.,0.7071,0.7071],[-0.7071,0.,0.7071],[0.,-0.7071,0.7071],[0.3827,0.,0.9239],[0.,0.3827,0.9239],[-0.3827,0.,0.9239],[0.,-0.3827,0.9239],[0.,0.,1.].

[0095] In the conormal clustering process (ST302 in FIG. 9A), each process is executed in the procedure shown in FIG. 9B.

[0096] First, a conormal plane extraction process is performed (ST401). In the conormal plane extraction process, a plane (conormal plane) is extracted from the point cloud of the restoration location, which is a set of points whose normal direction coincides with a predetermined direction based on the coordinate axes within an allowable range. Specifically, for each point constituting the point cloud of the restoration location, a point where the dot product of the point normal and the set normal exceeds a threshold value (e.g., 0.985) is extracted as a conormal plane. Here, the point normal refers to the normal of each point constituting the point cloud. Furthermore, the set normal refers to a normal (one of 21 possible directions) in a predetermined direction based on the coordinate axes. In other words, a group of planes whose normal is the set normal are extracted as conormal planes in the predetermined direction.

[0097] Next, density-based clustering processing is performed (ST402). In the density-based clustering processing, the conormal planes extracted in the conormal plane extraction processing are grouped using a density-based clustering (DBSCAN: Density-based spatial clustering of applications with noise) technique, and conormal clusters are generated for each predetermined direction.

[0098] Next, a minimum cluster discarding process is performed (ST403). In the minimum cluster discarding process, among the conormal clusters obtained by the density-based clustering process, conormal clusters whose constituent points are less than a predetermined threshold are discarded. The threshold is set to, for example, 200 points. In this case, if voxelization is performed in units of 1 cm in the preprocessing, for example, clusters whose area is less than 200 cm are generally discarded.

[0099] Next, it is determined whether or not axis rotation processing is necessary (ST404). If it is determined that axis rotation processing is necessary (Yes in ST404), processing to rotate the coordinate axes of the target cluster is performed (axis rotation processing) (ST405).

[0100] Next, sub-clustering processing is performed (ST406). In the sub-clustering processing, the conormal clusters obtained in the density-based clustering processing (ST402) that have been subjected to the minimum cluster discarding processing (ST403) are divided into sub-clusters. If an axial rotation processing has been performed, after the sub-clustering processing is completed, the clusters are rotated in the opposite direction to the initial axial rotation.

[0101] Next, a cluster group generation process is performed (ST407). In the cluster group generation process, the conormal clusters (including those decomposed into sub-clusters) are grouped for each predetermined direction to generate conormal cluster groups. The conormal cluster groups are generated for each predetermined direction (set normal) based on the coordinate axis.

[0102] In the conormal clustering process, each time a conormal cluster is detected, the points included in that conormal cluster are excluded from the point cloud to be processed thereafter. At this time, the points included in the conormal cluster, i.e., the points included in the bounding box when the conormal cluster was detected, specifically, the axis-aligned bounding box (AABB) or the oriented bounding box (OBB), are excluded. This prevents the same points from being included in different conormal clusters. Furthermore, since the points included in the conormal cluster are sequentially excluded from the point cloud of the target location as conormal clusters are detected, the order of conormal cluster detection can be controlled to prioritize the detected plane directions. For example, if priority is given to the detection of vertical planes, the detection of conormal clusters for vertical planes may be started first, followed by the detection of conormal clusters for horizontal planes.

[0103] 9(B) shows the sub-clustering process in the conormal clustering process, but the sub-clustering process is also performed in the individual clustering process. Specifically, the sub-clustering process is performed on points included in the oriented bounding boxes detected as individual clusters in the individual clustering process.

[0104] The example shown in Figure 10 is a case where conormal clustering processing is performed on the point cloud of the restored location shown in Figure 5(B). In the conormal clustering processing, the point cloud of the restored location is decomposed into conormal planes, and the conormal planes are grouped and extracted as conormal clusters. The conormal clusters are further grouped by predetermined direction (set normal). Note that in Figure 10, the conormal clusters are drawn in different colors (densities).

[0105] The example shown in Figure 10(A) is a cluster whose normal direction is the X direction [1,0,0], i.e., a group of clusters having conormal planes parallel to one wall surface. The example shown in Figure 10(B) is a cluster whose normal direction is the Y direction [0,1,0], i.e., a group of clusters having conormal planes parallel to the other wall surface. The example shown in Figure 10(C) is a cluster whose normal direction is the Z direction [0,0,1], i.e., a group of clusters having conormal planes parallel to the floor surface.

[0106] In the example shown in FIG. 11(A), the object to be restored is a cylinder with two planes joined in an L-shape and arranged inside it. The example shown in FIG. 11(B) is a group of clusters whose normal direction extracted from the point cloud of the target location shown in FIG. 11(A) is a predetermined direction [0,1,0]. In this case, there are three clusters belonging to the group. The example shown in FIG. 11(C) is a group of clusters whose normal direction extracted from the point cloud of the target location is a predetermined direction [-1,1,0]. In this case, there are two clusters belonging to the group. The example shown in FIG. 11(D) is a group of clusters whose normal direction extracted from the point cloud of the target location is a predetermined direction [0,0,1]. In this case, there is one cluster belonging to the group.

[0107] Next, a description will be given of the individual clustering process performed by the server 2. Fig. 12 is an explanatory diagram showing the status of the individual clustering process.

[0108] In the example shown in FIG. 12(A), partitions and a table are arranged in a room. One partition 51 is arranged in a different orientation from the other partitions and tables. The coordinate axes (X-axis and Y-axis) are set to follow the orientations of the partitions and tables other than partition 51. In this case, since partition 51 deviates from the set orientation based on the coordinate axes, when conormal clustering processing is performed, partition 51 is not properly represented in the 3D model after rectangularization processing, as shown in FIG. 12(B). In other words, there is no continuity in the rectangular plane representing partition 51, and partition 51 is decomposed into rectangular planes 52 in two predetermined directions (for example, 22.5 degrees and 45 degrees).

[0109] Therefore, in this embodiment, an individual clustering process is performed before the conormal clustering process, in which planes in any direction are individually extracted from the point cloud of the target location. As a result, in the 3D model after the rectangularization process, rectangular planes 53 representing partitions 51 are arranged on a single plane, improving the state to be closer to the actual shape, as shown in Figure 12(C).

[0110] In the individual clustering process, a plane of arbitrary direction is extracted as an individual cluster from the point cloud of the target location. At this time, an oriented bounding box is set for the point cloud of the restored location, and the points included in the oriented bounding box are extracted as an individual cluster. In addition, the individual clustering process uses an estimation method such as RANSAC to eliminate the influence of outliers.

[0111] In addition, the individual clustering process uses a planar patch detection technique. In planar patch detection, planar patches are detected based on the coplanarity angle. The coplanarity angle (coplanarity deg) defines the angle between the vector drawn from the center of the plane to a point cloud constituent point and the normal vector. A coplanarity angle of 90 degrees is the strictest, and in this case, only points that are completely on the plane are allowed. On the other hand, if the coplanarity angle is reduced, points that are outside the plane are also allowed. The coplanarity angle is set to, for example, 85 degrees. This plane detection technique is described in the literature "A. Araujo and M. Oliveira, A robust statistics approach for plane detection in unorganized point clouds, Pattern Recognition, 2020."

[0112] Here, individual clustering can detect planes in any direction by setting oriented bounding boxes, but in this case, a probabilistic search is performed using the RANSAC method. Therefore, individual clustering can detect large planes from the point cloud of the restored location, but may not be able to detect small planes. On the other hand, conormal clustering cannot detect planes in directions that deviate significantly from the specified direction based on the coordinate axes, but it can also detect small planes.

[0113] In this way, because individual clustering processing can extract planes in any direction, it compensates for the drawback of conormal clustering processing, which can only extract planes in a specified direction. On the other hand, while individual clustering processing cannot detect small planes, conormal clustering processing can detect even small planes as long as they are in a specified direction, so the drawback of individual clustering processing is compensated for by conormal clustering processing. As a result, by combining both conormal clustering processing and individual clustering processing, it is possible to appropriately extract the required clusters.

[0114] Furthermore, in this embodiment, the individual clustering process is performed before the conormal clustering process, and first, a large plane in an arbitrary direction is detected by the individual clustering process. Next, the conormal clustering process is performed. At this time, the points constituting the individual clusters acquired by the individual clustering process are excluded from the point cloud of the target location, and then the conormal clustering process is performed. This prevents the points constituting the individual clusters detected by the individual clustering process from being re-detected by the conormal clustering process. This makes it possible to avoid the generation of incorrect clusters caused by forcibly decomposing a plane in a direction deviating from a predetermined direction based on the coordinate axes into a plane in the predetermined direction.

[0115] Furthermore, although conormal clustering may detect many small planes, they are limited to planes in a specific direction based on the coordinate axes, so the 3D model obtained by the rectangularization process is generated in an orderly manner without being overly complicated.

[0116] In the individual clustering process, planes of any direction can be detected by setting a directional bounding box for the point cloud. However, the normal direction obtained from the directional bounding box is based on the random number values ​​provided by RANSAC and is therefore of low accuracy. Therefore, it is advisable to redo the plane estimation for each point included in the directional bounding box to obtain the correct normal direction.

[0117] In the coordinate axis setting process, the coordinate axes (X-axis and Y-axis) are set so that they are parallel to the walls of a room that is rectangular in plan view. However, in the case of a room that is not rectangular in plan view, some of the walls may not be parallel to the coordinate axes. In this case, even the conormal clustering process may not be able to extract some of the walls. On the other hand, the individual clustering process can detect planes in any direction that deviates from the predetermined direction based on the coordinate axes, and therefore can be used to detect the walls of a room that is not rectangular in plan view.

[0118] In addition, if there is an object, such as partition 51 in the example shown in Figure 12, that is arranged in a direction that is significantly different from the wall of the room and many other objects that are arranged parallel to that wall, such an object will be an outlier in the second coordinate axis setting process based on the distribution of normal directions, and will not affect the setting of the coordinate axis.

[0119] Next, a description will be given of the axis rotation process performed by the server 2. Fig. 13 is an explanatory diagram showing the state of the axis rotation process.

[0120] In the example shown in FIG. 13(A), a long, thin handrail 61 is placed diagonally along a wall surface. The handrail 61 extends in a direction significantly deviating from the direction of the coordinate axes. On the other hand, in the conormal clustering process, one cluster is set in the point cloud representing the handrail, and this cluster is then divided into multiple sub-clusters in the sub-clustering process. At this time, a bounding box (circumscribing rectangle) with sides aligned with the coordinate axis directions is set, and the cluster set for the entire handrail 61 is decomposed into bounding boxes aligned with the coordinate axes. For this reason, when the rectangularization process is performed after the clustering process, the handrail 61 is represented in a stepped shape by multiple rectangular planes 62, as shown in FIG. 13(B), which is significantly different from the actual shape.

[0121] Therefore, in this embodiment, before starting the sub-clustering process, an axis rotation process is performed to rotate the coordinate axes relative to the target cluster. As a result, as shown in Fig. 13(C), multiple rectangular planes 63 representing the handrail 61 are aligned straight in the direction in which the handrail 61 extends, improving the state to be closer to the actual shape.

[0122] The necessity of axial rotation processing is also determined, and if axial rotation processing is necessary, axial rotation processing is performed. The necessity of axial rotation processing is determined based on the fitting rate of an oriented bounding box set in an arbitrary direction relative to the point cloud of the cluster. In other words, axial rotation processing is performed if the fitting rate of the oriented bounding box is high. Furthermore, when axial rotation processing is performed, the orientation of the sides of the rectangular plane corresponding to the axially rotated cluster in the 3D model after rectangularization processing differs from that of other rectangular planes, which may make the 3D model appear cluttered and unsightly. Therefore, it is recommended to adjust the threshold for the fitting rate of the oriented bounding box to prevent excessive axial rotation processing. This allows the 3D model to be generated in an orderly and easy-to-view state.

[0123] In the axis rotation process, the angle by which the coordinate axes are rotated relative to the cluster is set based on the tilt of the axial direction of the oriented bounding box set for the point cloud of the cluster.

[0124] Specifically, first, the cluster's point cloud is rotated so that the normal points upward. Next, an axis-aligned bounding box and an oriented bounding box are set for the cluster's point cloud. Next, the orthogonal projection areas of the axis-aligned bounding box and the oriented bounding box onto the XY plane (horizontal plane) are calculated. Next, if the ratio of the orthogonal projection areas of the axis-aligned bounding box and the oriented bounding box exceeds a predetermined threshold (e.g., 1.5), it is determined that axial rotation is necessary. If it is determined that axial rotation is necessary, the coordinate axes are rotated relative to the cluster according to the inclination of the oriented bounding box. Once the sub-clustering process is completed, the cluster's point cloud is subjected to axial rotation in the opposite direction to the initial rotation.

[0125] Next, we will explain the sub-clustering process performed by server 2. Figures 14, 15, 16, and 17 are explanatory diagrams showing the status of the sub-clustering process. Note that in Figures 14, 15, 16, and 17, the sub-clusters are drawn in different colors (densities).

[0126] In the sub-clustering process, the conormal clusters obtained in the density-based clustering process and subjected to the smallest cluster discarding process are divided into a plurality of sub-clusters.

[0127] The example shown in Figure 14 is a cluster whose normal direction is the X direction [1,0,0], that is, a group of conormal clusters having a conormal plane parallel to one wall surface. When the subclustering process is performed on the conormal cluster shown in Figure 14(A), the conormal cluster is divided into subclusters as shown in Figure 14(B).

[0128] The example shown in Figure 15 is a group of clusters whose normal direction is the Y direction [0,1,0], that is, clusters having a conormal plane parallel to the other wall surface. When the sub-clustering process is performed on the conormal cluster shown in Figure 15(A), the conormal cluster is divided into sub-clusters as shown in Figure 15(B).

[0129] The example shown in Figure 16 is a group of clusters whose normal direction is the Z direction [0,0,1], that is, clusters having conormal planes parallel to the floor surface. When the sub-clustering process is performed on the conormal cluster shown in Figure 16(A), the conormal cluster is divided into sub-clusters as shown in Figure 16(B).

[0130] The example shown in Figure 17 is a group of clusters whose normal direction is the Z direction [0,0,1], similar to the example shown in Figure 16, but is drawn as viewed from a different direction than the example shown in Figure 16.

[0131] Next, the procedure for the sub-clustering process will be explained. Figures 18, 19, 20, 21, 22, 23, 24, and 25 are explanatory diagrams showing the procedure for the sub-clustering process. In the examples shown in Figures 18 to 25, it is assumed that the clusters are on the XY plane.

[0132] Clusters (conormal clusters and individual clusters) are planar and irregular in shape, and often have uneven density and holes. Therefore, in this embodiment, a cluster is divided into multiple sub-clusters that correspond to the shape of the cluster. This makes it possible to generate a 3D model that faithfully represents the shape of the cluster to a certain extent.

[0133] For example, if a cluster is L-, T-, or U-shaped, simply setting a bounding box (circumscribing rectangle) for the cluster results in the L-, T-, or U-shaped shape being represented as a simple rectangle, resulting in a significantly insufficient ability to represent the shape. On the other hand, if the cluster is too faithful to its shape and a rectangular model with a sawtooth-like outline is obtained, this is undesirable for the original purpose, and it is desirable to abstract it appropriately. Therefore, in this embodiment, one of the following modes is executed: 2-division mode, N-division mode, or 2-stage division mode, as shown below.

[0134] First, the two-division mode will be described. As shown in Fig. 18, in the two-division mode, two bounding boxes BB1 and BB2 (circumscribing rectangles) are set for cluster C so as to divide cluster C into two. In addition, in the two-division mode, a division position that maximizes the amount of area reduction due to division is searched for. Specifically, a division position that maximizes the difference (S0-S1-S2) between the area S0 of the bounding box BB0 corresponding to the entire point cloud of the cluster before division and the sum of the areas S1 and S2 of the first and second bounding boxes BB1 and BB2 corresponding to the point cloud after division is searched for. In the example shown in Fig. 18, the division position is searched for in the X direction.

[0135] The example shown in Figure 18(A) is a case where cluster C is L-shaped. In this case, the area reduction amount (S0-S1-S2) due to division is the largest and most optimal at division position b shown in Figure 18(B-2) compared to division position a shown in Figure 18(B-1) and division position c shown in Figure 18(B-3). Therefore, cluster C is divided into two sub-clusters SC at division position b. In this case, the L-shape of cluster C is expressed by two rectangular planes set corresponding to the two bounding boxes BB1 and BB21 in the rectangularization process.

[0136] Here, even if the area reduction amount by division is small, if a cluster is divided into sub-clusters, the 3D model that is finally generated becomes complex and difficult to see. Therefore, if the area reduction amount by division (S0-S1-S2) is less than a predetermined threshold, division is not performed. The threshold is, for example, 0.25 m 2 is set to

[0137] In this way, in the two-division mode, two bounding boxes (circumscribing rectangles) are set to divide the cluster in half along the coordinate axis, and the division position along the coordinate axis is searched for to maximize the area reduction of the bounding boxes due to the division. This divides the cluster so that the total area of ​​the bounding boxes is reduced, making it possible to generate a 3D model that faithfully represents the shape of the cluster to a certain extent.

[0138] 18, the division positions are searched for in the X direction, but the division positions may also be searched for in the Y direction. Alternatively, the division positions may be searched for in both the X and Y directions, and the division position that results in the largest amount of area reduction may be adopted.

[0139] Next, an example will be described in which an appropriate division position cannot be set by a single search in one direction (X direction) in the two-division mode. The example shown in Fig. 19(A) is a case in which cluster C is T-shaped.

[0140] In this case, the area of ​​the bounding box is reduced in the case of division position a shown in Fig. 19(B-1) and division position c shown in Fig. 19(B-3). However, in the case of division position a shown in Fig. 19(B-1), division position b shown in Fig. 19(B-2), and division position c shown in Fig. 19(B-3), the T-shape of the cluster cannot be represented, so these are not optimal.

[0141] Next, we will explain an example in which an appropriate division position can be set by searching twice in one direction (X direction) in the two-division mode. The example shown in Figure 20(A) is the case where cluster C is T-shaped, similar to the example shown in Figure 19(A).

[0142] In this case, first, as shown in Figure 20(B), the point cloud of cluster C is divided a first time, and bounding boxes BB1 and BB2 are set. Next, as shown in Figures 20(C-1) and (C-2), the point cloud of bounding box BB2 is divided a second time, and bounding boxes BB21 and BB22 are set. As a result, as shown in Figure 20(D), three bounding boxes BB1, BB21, and BB22 are set for the point cloud of cluster C, and the point cloud of cluster C is divided into three sub-clusters SC. In this case, the T-shape of cluster C is represented by three rectangular planes set corresponding to the three bounding boxes BB1, BB21, and BB22 in the rectangularization process.

[0143] In this way, by repeating the search and division in one direction twice, a T-shaped cluster is expressed. In the example shown in Fig. 20, the search in the X direction is performed twice and the division is performed twice. On the other hand, the second search may be performed in a direction (Y direction) different from the first search. Also, conversely, the first search may be performed in the Y direction and the second search in the X direction.

[0144] Next, an example will be described in which an appropriate division position cannot be set in the two-division mode. The example shown in Fig. 21(A) is a case in which cluster C is U-shaped.

[0145] In this case, the areas of the bounding boxes BB1 and BB2 are not reduced and the U-shape of cluster C cannot be expressed in any of the cases of division position a shown in Fig. 21(B-1), division position b shown in Fig. 21(B-2), and division position c shown in Fig. 21(B-3), which are not optimal. Moreover, the results are the same even if the search direction is changed.

[0146] Next, the N-division mode will be described. As shown in Fig. 22, in the N-division mode, a search box SB (such as SB1 in the figure) is set for cluster C, and a division position that allows for division into two is searched for within the search box SB. If no division position is found within the search box SB, the width of the search box SB (search length) is gradually widened to L, 2L, 3L, etc. Furthermore, if a division position that allows for division into two is found within the search box SB, a bounding box BB (such as BB1 in the figure) is set so that the point cloud is divided at that division position. Next, a search for a division position using the search box SB is performed for the remaining point clouds of cluster C, and this process is repeated throughout cluster C.

[0147] The example shown in FIG. 22(A) is a case where cluster C is U-shaped. In this case, first, as shown in FIG. 22(B-1), a search box SB1 having a width of search length L is set, and it is determined whether or not there is a division position by dividing into two, and it is determined here that there is no division position. Next, as shown in FIG. 22(B-2), a search box SB2 having a width of search length 2L is set, and it is determined whether or not there is a division position by dividing into two, and it is also determined here that there is no division position. Next, as shown in FIG. 22(B-3), a search box SB3 having a width of search length 3L is set, and it is determined whether or not there is a division position by dividing into two, and it is determined here that there is a division position. Therefore, as shown in FIG. 22(C), the point cloud included in search box SB3 is divided into two. As a result, a bounding box BB1 is set.

[0148] Next, as shown in FIG. 23(A), a division position is searched for in the remaining portion after removing the bounding box BB1 from cluster C. First, as shown in FIG. 23(B-1), a search box SB1 having a width of search length L is set, and it is determined whether or not there is a division position by dividing the cluster into two. Here, it is determined that there is no division position. Next, as shown in FIG. 23(B-2), a search box SB2 having a width of search length 2L is set, and it is determined whether or not there is a division position by dividing the cluster into two. Here, it is also determined that there is no division position. Next, as shown in FIG. 23(B-3), a search box SB3 having a width of search length 3L is set, and it is determined whether or not there is a division position by dividing the cluster into two. Here, it is determined that there is a division position. Therefore, as shown in FIG. 23(C), the point cloud included in search box SB3 is divided into two. As a result, a bounding box BB21 is set.

[0149] Next, as shown in FIG. 24(A), a division position is searched for in the remaining portion after removing bounding boxes BB1 and BB21 from cluster C. First, as shown in FIG. 24(B-1), a search box SB1 having a width of search length L is set, and it is determined whether or not there is a division position resulting from division into two, and it is determined that there is no division position. Next, as shown in FIG. 24(B-2), a search box SB2 having a width of search length 2L is set, and it is determined whether or not there is a division position resulting from division into two, and it is also determined that there is no division position. Next, as shown in FIG. 24(B-3), a search box SB3 having a width of search length 3L is set, and it is determined whether or not there is a division position resulting from division into two, and it is also determined that there is no division position. At this time, since there is no point cloud beyond search box SB3 in the X direction, it is determined that the search has ended. Therefore, as shown in FIG. 24(C), a bounding box BB22 is set for the remaining portion after removing bounding boxes BB1 and BB21 from cluster C. As a result, three bounding boxes BB1, BB21, and BB22 are set for cluster C, and cluster C is divided into three sub-clusters SC. In this case, the U-shape of cluster C is expressed by three rectangular planes set in the rectangularization process corresponding to the three bounding boxes BB1, BB21, and BB22.

[0150] In this way, in the N-division mode, the width (search length) of the search box SB (such as SB1 in the figure) is gradually expanded by the expansion width L, and a two-division process is attempted; that is, when a division position is found within the search box SB, the two-division process is performed at that division position and one bounding box BB (such as BB1 in the figure) is set, and then the same process is repeated for the remaining point clouds, and this process is repeated until a bounding box BB is set for the entire point cloud of cluster C.

[0151] In addition, in the N-division mode, a search box SB is set when searching for a division position, and the division position is searched for while gradually widening the width (search length) of the search box SB to L, 2L, and 3L. Therefore, only one division position can be set for each set search box SB. As a result, it may not be possible to express shapes smaller than the width of the search box SB, but this prevents the cluster from being randomly subdivided, and allows the shape of the cluster to be expressed at an appropriate level of abstraction. The expansion width L when gradually widening the width of the search box SB may be set appropriately depending on the restoration location, etc., but may be set to, for example, 0.5 m.

[0152] Note that the examples shown in Figures 22, 23, and 24 are for clusters that are U-shaped, but in N-division mode, even for clusters that are not U-shaped, multiple bounding boxes (sub-clusters) can be appropriately set to divide the cluster.

[0153] Next, the two-stage division mode will be described. In the two-stage division mode, the cluster is divided into two stages with different search directions. That is, the cluster is divided in a first stage along a first direction (X direction or Y direction) and in a second stage along a second direction different from the first direction.

[0154] In the example shown in FIG. 25(A), cluster C has a complex shape. In this case, as shown in FIG. 25(B-1) and FIG. 25(C-1), a single division in only one direction, either the X direction or the Y direction, cannot set an appropriate bounding box for the cluster. On the other hand, as shown in FIG. 25(B-2) and FIG. 25(C-2), when a second division is performed in a direction different from the first division, cluster C is appropriately divided into multiple sub-clusters SC. Here, in the example shown in FIG. 25(B-1) and (B-2), division is first performed in the X direction, and then in the Y direction. Also, as shown in FIG. 25(C-1) and (C-2), division is first performed in the Y direction, and then in the X direction.

[0155] Here, in the two-stage division mode, as in the two-stage division mode, a threshold value is set for the amount of area reduction when determining whether or not division is possible, and if the amount of area reduction due to division is less than the threshold value, division is not performed. Also, if the threshold value for the second stage division is set smaller than that for the first stage division, division is more likely to occur. For this reason, it is preferable to set separate threshold values ​​for determining whether or not division is possible for the first and second stages. For example, the threshold value for the first division is 0.25 m. 2 , 0.05m for the second division 2 may be set to

[0156] In this embodiment, the sub-clustering process that divides a cluster (conormal cluster or individual cluster) obtained by the conormal clustering process or the individual clustering process into multiple sub-clusters can be performed in two-division mode, N-division mode, or two-stage division mode, but these processes are not limited to the use of dividing a cluster into sub-clusters (sub-clustering).

[0157] That is, a clustering process may be performed in which a plane (a set of points distributed on a plane at a predetermined density) is extracted from the point cloud of the target location, and the plane is divided into clusters by performing processing in a two-division mode, an N-division mode, or a two-stage division mode. Such clustering processing may be used, for example, to create a drawing (such as a front view or a plan view) of a single object to be restored at the target location from the point cloud of the object. Specifically, it may be used, for example, to create a rough sketch of the exterior of a building.

[0158] Next, the rectangularization process (ST103 in FIG. 3) performed by the server 2 will be described. FIG. 26 is a flow diagram showing the procedure of the rectangularization process. Note that the process of this flow is performed in groups of conormal clusters and individual clusters generated in the clustering process. FIG. 27 is an explanatory diagram showing the status of the rectangularization process. FIGS. 28 and 29 are explanatory diagrams showing the three-dimensional model (rectangle model) generated in the rectangularization process.

[0159] In the rectangularization process, first, the clusters obtained in the clustering process (see ST102 in FIG. 3) are converted into rectangular planes (initial rectangularization process) (ST501). Here, the process is executed for individual clusters and their sub-clusters, and conormal clusters and their sub-clusters.

[0160] In the initial rectangularization process, the cluster is first rotated around the origin so that the normal points upwards to [0,0,1]. This causes the rotated cluster to extend horizontally, simplifying subsequent processing. Next, the bounding box and centroid of the cluster are obtained, and the z coordinate of the bounding box is replaced with the z coordinate of the centroid. This results in a horizontal rectangular plane. Next, the rectangular plane is rotated in the opposite direction to the initial rotation.

[0161] In the example shown in FIG. 27, similar to the example shown in FIG. 11, the object to be restored is a cylinder in which two planes are arranged in an L-shape. In the initial rectangularization process, as shown in FIG. 27(A), a cluster extracted from the point cloud of the object to be restored is converted into a rectangular plane. Next, as shown below, a process is performed to adjust the position and shape of the rectangular plane. In the example shown in FIG. 27(A), an end of one rectangular plane protrudes from the other rectangular plane. In this case, by cutting off the protruding portion of one rectangular plane, the shape of the rectangular plane is adjusted so that two adjacent rectangular planes share a side, as shown in FIG. 27(B).

[0162] 26, as a process for adjusting the position and shape of rectangular planes, an alignment process is first performed (ST502). In the alignment process, rectangular planes belonging to the same group are targeted, and a process for aligning multiple rectangular planes that should be located on one plane on the same plane is performed.

[0163] Next, a first inter-group snapping process is performed (ST503). In the first inter-group snapping process, two rectangular planes in different groups are targeted, and if there is a gap between the two rectangular planes, the gap is filled to adjust the shapes of the rectangular planes so that the two rectangular planes share a side.

[0164] Next, a second inter-group snapping process is performed (ST504). In the second inter-group snapping process, two rectangular planes in different groups are targeted, and if there is a protrusion between the two rectangular planes, the protruding portion is removed to adjust the shape of the rectangular planes so that the two rectangular planes share a side.

[0165] Next, intra-group snap processing is performed (ST505). In intra-group snap processing, rectangular planes belonging to the same group are targeted, and the shapes of the rectangular planes are adjusted by filling in gaps between two rectangular planes and cutting off portions of one rectangular plane that extend beyond the other rectangular plane.

[0166] Note that the group in the alignment process, the first and second inter-group snapping processes, and the intra-group snapping process represents a group of rectangular models (rectangular group) when the cluster is converted into a rectangular plane by the initial rectangularization process, and corresponds to a group of clusters (conormal clusters and individual clusters).

[0167] When the sub-clustering process is not performed, the rectangularization process is performed on the individual clusters and conormal clusters acquired by the clustering process, as shown in Fig. 28. On the other hand, when the sub-clustering process is performed, the rectangularization process is performed on the individual clusters and their sub-clusters, and the conormal clusters and their sub-clusters, as shown in Fig. 29.

[0168] As shown in Fig. 28, when the sub-clustering process is not performed, the fidelity of the objects at the restored location is low, and for example, the U-shaped desk placed in the center of the room is not sufficiently represented. On the other hand, as shown in Fig. 29, when the sub-clustering process is performed, the fidelity of the objects at the restored location is high, and for example, the U-shaped desk placed in the center of the room is sufficiently represented.

[0169] Next, a description will be given of the alignment processing performed by the server 2. Fig. 30 is an explanatory diagram showing the state of the alignment processing.

[0170] The alignment process involves aligning adjacent rectangular planes that belong to the same group (conormal rectangle group). The rectangular planes of each cluster obtained by the initial rectangle process may not be located on the same plane, even though they actually belong to a single plane (for example, a wall, floor, or ceiling). If this is left unchecked, they will not be displayed as a single line when drawn on a floor plan (2D layout diagram). This is why the alignment process is performed.

[0171] When the alignment process is performed, for example, an unnatural state where there is a step between rectangular planes that make up indoor wall surfaces, i.e., rectangular planes that should be located on the same plane, is eliminated. In the example shown in Figure 30(A), there is a step between rectangular planes RP1 and RP2. When the alignment process is performed, as shown in Figure 30(B), the step between rectangular planes RP1 and RP2 disappears, and the rectangular planes RP1 and RP2 are aligned on the same plane.

[0172] In the alignment process, first, the rectangular planes are rotated so that their normals point upward. Next, the proximity between rectangular planes is determined, and adjacent planes are grouped together. Next, the z coordinates of the rectangular planes in each group are weighted averaged, and the z coordinate of the rectangular plane is replaced with this average value. At this time, the weighting is set according to the number of points that make up the original cluster. Next, the rectangular planes are rotated in the opposite direction to the initial rotation. This results in rectangular planes belonging to the same group being aligned on the same plane.

[0173] Next, a description will be given of the snap processing between the first and second groups performed by the server 2. Fig. 31 is an explanatory diagram showing the status of the snap processing between the first and second groups.

[0174] In the first inter-group snapping process, when two rectangular planes in different groups (conormal rectangle groups) are targeted and there is a gap between the two rectangular planes, the shape of the rectangular planes is adjusted so that the two rectangular planes share an edge. Specifically, one edge (nearby edge) of the two rectangular planes is joined to the other face (nearby face) to eliminate the gap between the two rectangular planes.

[0175] When the first inter-group snapping process is performed, for example, an unnatural state in which there are gaps between rectangular planes representing each surface (such as the top surface or side surfaces) of a box-like object is eliminated. In the example shown in Fig. 31(A-1), there are gaps between the rectangular planes RP1, RP2, and RP3. When the first inter-group snapping process is performed, as shown in Fig. 31(A-2), the gaps between the rectangular planes RP1, RP2, and RP3 disappear, and the rectangular planes RP1, RP2, and RP3 are reshaped so that they share a side.

[0176] In the first inter-group snapping process, first, the rectangular planes are rotated so that their normals point upward. Next, a proximity determination is made between the rectangular planes. Here, if the rectangular planes intersect, that is, if one of the rectangular planes protrudes from the other, this process is skipped and the process proceeds to the next, second inter-group snapping process. If there is a gap between the rectangular planes, the side (nearby side) of one rectangular plane is extended to join with the face (nearby face) of the other rectangular plane. At this time, the near face is extended as necessary. Next, the rectangular planes are rotated in the opposite direction to the initial rotation.

[0177] In the second inter-group snapping process, when two rectangular planes in different groups (conormal rectangle groups) are targeted and one of the two rectangular planes protrudes from the other, a process is performed to adjust the shape of the rectangular plane so that the two rectangular planes share an edge. Specifically, a process is performed to cut off the protruding portion. In this case, if the protruding amount is large and the protruding edge is not close to the face of the other side, the second inter-group snapping process is not performed.

[0178] When the second inter-group snapping process is performed, for example, an unnatural state in which one of the rectangular planes representing each face (such as the top or side) of a box-like object protrudes from the other is eliminated. In the example shown in Fig. 31(B-1), the rectangular plane RP1 protrudes from the rectangular plane RP2. When the second inter-group snapping process is performed, as shown in Fig. 31(B-2), the protruding portion of the rectangular plane RP1 is removed, and the shape is adjusted so that the rectangular plane RP1 and the rectangular plane RP2 share a side.

[0179] In the second inter-group snapping process, first, the rectangular planes are rotated so that their normals point upward. Next, a proximity determination is made between the rectangular planes. Here, if the rectangular planes do not intersect, that is, if one of the rectangular planes does not protrude beyond the other, this process is skipped. Next, the protruding portion of the edge (neighboring edge) of one of the rectangular planes is cut off. Next, the rectangular planes are rotated in the opposite direction to the initial rotation.

[0180] Next, a description will be given of intra-group snap processing performed by the server 2. Fig. 32 is an explanatory diagram showing the status of intra-group snap processing.

[0181] In the intra-group snapping process, two adjacent rectangular planes that belong to the same group (conormal rectangular group) are targeted, and the shapes of the rectangular planes are adjusted.

[0182] In the example shown in Fig. 32(A-1), of two rectangular planes RP1 and RP2 belonging to the same group, one side of the rectangular plane RP1 slightly protrudes into the other rectangular plane RP2. In this case, as shown in Fig. 32(A-2), a process is performed to remove the protruding portion of one rectangular plane RP1, thereby correcting the two rectangular planes RP1 and RP2 so that the sides of the two rectangular planes RP1 and RP2 are in contact with each other.

[0183] In the example shown in Fig. 32(B-1), among the rectangular planes RP1, RP2, and RP3 that belong to the same group, there is a small gap between the rectangular plane RP1 and the rectangular planes RP2 and RP3. In this case, as shown in Fig. 32(B-2), a process is performed to extend one of the rectangular planes RP1 so as to eliminate the gap between the rectangular planes, thereby correcting the three rectangular planes RP1, RP2, and RP3 so that their sides are in contact with each other.

[0184] Next, a description will be given of the floor plan generation processing carried out by the server 2. Fig. 33 is an explanatory diagram showing the status of the floor plan generation processing.

[0185] In the server 2, the floor and ceiling parts, i.e., the rectangular planes representing the floor and ceiling, are deleted from the 3D model (rectangular model), and then the 3D model is orthogonally projected onto the XY plane (horizontal plane) to generate a floor plan (2D layout drawing) (floor plan generation process). The floor and ceiling parts in the 3D model can be identified based on the z coordinate.

[0186] As shown in Figure 33(A), the 3D model is composed of multiple rectangular planes. Note that in the 3D model shown in Figure 33(A), the rectangular planes that make up the ceiling have been removed so that the interior state of the room where the image is restored can be seen. As shown in Figure 33(B), the floor plan depicts the walls of the room where the image is restored, as well as furniture such as desks, chairs, and shelves that are placed in the room.

[0187] In this embodiment, the majority of objects in the restoration location in the 3D model are represented by rectangular planes aligned along horizontal coordinate axes (X-axis and Y-axis), and the shapes of the objects to be restored are represented fairly faithfully, making it possible to obtain a reasonably abstract, orderly, and easy-to-read plan view (2D layout view).

[0188] Next, a description will be given of the dimension line insertion process performed by the server 2. Fig. 34 is an explanatory diagram showing the state of the dimension line insertion process.

[0189] The server 2 extracts a representative plane from the three-dimensional model and inserts dimension lines related to the representative plane into the floor plan (dimension line insertion process). At this time, in the three-dimensional model (rectangular model), for example, two planes that are spaced apart are extracted as representative planes. For example, in the case of a room that is rectangular in plan view, the walls of the room are extracted. Furthermore, dimension lines representing the inter-plane distance between two opposing planes are inserted into the floor plan as dimension lines related to the representative planes. In the example shown in FIG. 34, dimension lines representing the inter-plane distance between two pairs of wall surfaces that are opposing each other in the directions of the coordinate axes (X-axis and Y-axis) in a room that is rectangular in plan view are inserted into the floor plan.

[0190] In this case, a combination of a predetermined number of planes may be selected in descending order of inter-plane distance. Alternatively, a combination of planes whose area is equal to or greater than a predetermined threshold may be selected. The threshold may be set based on the overall size of the restored location. This makes it possible to avoid the plan view becoming cluttered due to the insertion of numerous dimension lines into the plan view.

[0191] Next, a description will be given of the drawing display process performed by the server 2. Fig. 35 is an explanatory diagram showing the viewing screens 101 and 201 displayed on the user terminal 1.

[0192] In user terminal 1, viewing screens 101 and 201 are displayed on display 13 under the control of server 2. Fig. 35(A) shows viewing screen 101 in the plan view display mode, and Fig. 35(B) shows viewing screen 201 in the measurement display mode.

[0193] As shown in Fig. 35(A), the viewing screen 101 in the plan view display mode displays a plan view 111 (two-dimensional layout drawing) generated by the drawing generation process. In the example shown in Fig. 35(A), the plan view 111 with dimension lines inserted is displayed. Note that a CAD mode may be provided so that editing operations can be performed on the plan view 111, such as erasing unnecessary lines and adding necessary lines.

[0194] A visualization of a three-dimensional model is displayed on the viewing screen 201 in the measurement display mode. In the example shown in Fig. 35(B), a three-dimensional layout drawing 211 is displayed, which visualizes a three-dimensional model (rectangular model) made up of multiple rectangular planes as viewed from a predetermined viewpoint.

[0195] Furthermore, on the viewing screen 201, measurement results related to the specified measurement object are displayed in response to a user operation of specifying the measurement object on the three-dimensional layout drawing 211. In the example shown in Fig. 35(B), when the user specifies two wall surfaces to be measured by a predetermined operation (for example, a drag operation), a display box 212 for the inter-surface distance and a display box 213 for the parallelism are displayed.

[0196] 35(B), a three-dimensional layout drawing 211 is displayed as a layout drawing that visualizes a three-dimensional model, but a two-dimensional layout drawing (floor plan) may also be displayed. Furthermore, on the viewing screen 201, when a user selects a plane (e.g., a wall surface of a room) to be measured, the selected plane may be highlighted on the layout drawing. For example, the drawing color of the selected plane on the layout drawing may change.

[0197] Furthermore, rectangular planes may be drawn in different colors according to their normal directions in the three-dimensional layout diagram 211. For example, rectangular planes whose normal direction is the X direction may be drawn in red, rectangular planes whose normal direction is the Y direction may be drawn in green, and among rectangular planes whose normal direction is the Z direction, rectangular planes representing the floor may be drawn in gray, and rectangular planes representing surfaces other than the floor may be drawn in blue.

[0198] In the server 2, measurement and display processing for the inter-surface distance and measurement and display processing for the parallelism are performed in response to user operations on the user terminal 1, and a display box 212 for the inter-surface distance and a display box 213 for the parallelism are displayed on the viewing screen 201 of the user terminal 1.

[0199] In the measurement and display process for the inter-surface distance, in response to the user selecting two planes on the three-dimensional layout diagram 211, the inter-surface distance between the two selected planes is measured based on the three-dimensional model, and the inter-surface distance is displayed in the display box 212 on the viewing screen 201.

[0200] In this case, the user simply designates one point on each of two planes (e.g., the walls of a room) to be used for measuring the inter-plane distance. The user simply selects one point on each plane as appropriate, and the direction connecting the two points designated by the user may deviate from the normal to the two planes. In this embodiment, the parallelism of two planes arranged parallel to each other at the restored location is ensured in the 3D model by conormal clustering processing. Therefore, once the two planes to be measured are selected by the user, the inter-plane distance between the two planes, i.e., the separation distance in the normal direction between the two planes, can be easily measured.

[0201] In this embodiment, the distance between two planes is measured, but the distance between one plane and a line parallel to that plane may also be measured. For example, the distance between a wall of a room and the edge (straight line) of a piece of furniture, such as a desk, placed parallel to that wall may also be measured.

[0202] Furthermore, in the parallelism measurement and display process, when the user selects two planes on the 3D layout drawing, the parallelism of the two selected planes is calculated based on the point cloud data, and the parallelism of the two planes is displayed in a display box 213 on the viewing screen 201. In actual construction, there are cases where two planes that should be parallel are not constructed parallel, and in such cases, the user can easily check the parallelism of the two planes by selecting the two planes to be measured.

[0203] In this embodiment, when the user selects points on two parallel planes, the inter-plane distance between the two planes is measured and displayed on the viewing screen 201. However, when the user selects two points, the distance between the two points, i.e., the length of the straight line connecting the two points, may also be measured and displayed on the viewing screen 201.

[0204] In this embodiment, a 3D model is obtained in which the parallelism of parallel planes at the reconstruction location is ensured. This makes it easy to measure the distance between surfaces and to combine 3D models. As a specific example, consider a case in which partially obtained 3D models of a square building are combined to construct a 3D model of the entire building. In the 3D model generated by the 3D model generation device according to the present invention, the coordinate axes are set in accordance with the wall surfaces of the building, and the constituent planes are limited to a predetermined direction. Therefore, in combining, only translation and three types of rotation (90-degree, 180-degree, and -90-degree rotations about the Z axis) need to be considered, making it easier to combine compared to models in which the orientation of the constituent planes is not limited.

[0205] The present embodiment is expected to be widely used as a fundamental technology for creating digital twins of real spaces. For example, the technology disclosed herein can be used to generate 3D models of factories, warehouses, stores, etc., and for a variety of purposes, such as facility and inventory management, renovations, layout change planning, and acquiring spatial data for simulations.

[0206] As described above, the embodiments have been described as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited to these, and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made. Furthermore, it is also possible to combine the components described in the above embodiments to create new embodiments. [Industrial Applicability]

[0207] The 3D model generation device, 3D model generation method, and 3D model generation program according to the present invention can generate 3D models that allow for easy restoration of inter-surface distances and the combination of multiple 3D models, and by generating moderately abstract 3D models, have the effect of creating neat, easy-to-read drawings and reducing the processing load on the processor. They are useful as 3D model generation devices, 3D model generation methods, and 3D model generation programs that use a processor to perform 3D restoration processing of a target location and generate a 3D model based on point cloud data obtained. [Explanation of symbols]

[0208] 1: User terminal 2: Server 12: Camera 13: Display 14: Input device 16: Processor 51: Partition 52,53: Rectangular plane 61: Handrail 62,63: ​​Rectangular plane 101: Viewing screen 111: Floor plan (2D layout drawing) 201: Viewing screen 211: 3D layout diagram 212: Display box for face-to-face distance 213: Parallelism display box BB,BB0,BB1,BB2,BB21,BB22: Bounding boxes C: Cluster L: Expansion width RP1, RP2, RP3: Rectangular plane SB, SB1, SB2, SB3: Search box SC: Sub-cluster θ: Normal angle φ: Rotation angle

Claims

1. A three-dimensional model generation device that uses a processor to execute a process of generating a three-dimensional model based on point cloud data acquired by performing a three-dimensional reconstruction process on a target location, The processor: Set a coordinate axis along the wall of the target location for the point cloud of the target location, A plane is detected from the point cloud of the target location, which is a set of points whose normal direction coincides with a set direction based on the coordinate axis within a predetermined tolerance range, and a conormal cluster is extracted based on the plane; Dividing the conormal cluster into a plurality of sub-clusters as circumscribing rectangles; Transforming the conormal cluster and the sub-clusters into a rectangular plane; A three-dimensional model generating device that generates the three-dimensional model in which a target location is expressed by a set of a plurality of the rectangular planes.

2. The processor: setting two circumscribing rectangles so as to divide the plane into two, and searching for a division position in the direction of the coordinate axis where the amount of area reduction of the circumscribing rectangles due to the division is the largest; 2. The three-dimensional model generating device according to claim 1, wherein when the dividing position is found, the plane is divided into two at the dividing position to set two of the circumscribing rectangles.

3. The processor:

3. The three-dimensional model generating device according to claim 2, wherein when the amount of area reduction is less than a predetermined threshold, the plane is not divided.

4. The processor: setting two circumscribing rectangles so as to divide the plane into two, and performing a bisection process to search for a division position in the direction of a coordinate axis where the amount of area reduction of the circumscribing rectangles due to the division is maximized; When the division position is found, the plane is divided into two at the division position to set one of the circumscribing rectangles; The three-dimensional model generating device according to claim 1, further comprising: a step of performing the bisection process for searching for a division position in the same search direction on the remaining point cloud excluding the range of the set circumscribing rectangle;

5. The processor: A search box of a predetermined width is set on the plane, and two circumscribing rectangles are set so as to divide the point cloud within the search box into two, and a bisection process is attempted in which a division position in the direction of the coordinate axis is searched for that maximizes the area reduction of the circumscribing rectangle due to the division; If the division position cannot be found, the width of the search box is gradually increased by a predetermined expansion width, and the bisection process is attempted again to search for a division position among the point cloud within the search box; If the dividing position is found, the point cloud within the search box is divided into two to set one of the circumscribing rectangles; Next, the bisection process is attempted on the remaining point cloud excluding the range of the set circumscribing rectangle, 2. The three-dimensional model generating device according to claim 1, wherein the trial of the bisection process is repeated until the entire plane is completed.

6. The processor:

2. The three-dimensional model generating device according to claim 1, further comprising: a first stage division process for searching for division positions in the direction of a first coordinate axis on the plane; and a second stage division process for searching for division positions in the direction of a second coordinate axis on the plane.

7. The processor: Before dividing the conormal cluster into the sub-clusters, The three-dimensional model generating device according to claim 2, characterized in that an axis rotation process is performed to rotate the coordinate axes relative to the conormal cluster in accordance with the inclination of an oriented bounding box set for the conormal cluster.

8. The processor: The three-dimensional model generating device according to claim 7, characterized in that the necessity of the axis rotation processing is determined by comparing the ratio of the orthogonal projection areas of the oriented bounding box and the axis-parallel bounding box set for the conormal cluster onto a horizontal plane with a predetermined threshold.

9. A three-dimensional model generation method for generating a three-dimensional model based on point cloud data acquired by performing three-dimensional reconstruction processing on a target location, the method comprising: Set a coordinate axis along the wall of the target location for the point cloud of the target location, A plane is detected from the point cloud of the target location, which is a set of points whose normal direction coincides with a set direction based on the coordinate axis within a predetermined tolerance range, and a conormal cluster is extracted based on the plane; Dividing the conormal cluster into a plurality of sub-clusters as circumscribing rectangles; Transforming the conormal cluster and the sub-clusters into a rectangular plane; A three-dimensional model generation method, characterized in that the three-dimensional model is generated by expressing a target location as a set of a plurality of the rectangular planes.

10. A three-dimensional model generation program that causes a processor to execute a process of generating a three-dimensional model based on point cloud data acquired by performing a three-dimensional reconstruction process on a target location, Set a coordinate axis along the wall of the target location for the point cloud of the target location, A plane is detected from the point cloud of the target location, which is a set of points whose normal direction coincides with a set direction based on the coordinate axis within a predetermined tolerance range, and a conormal cluster is extracted based on the plane; Dividing the conormal cluster into a plurality of sub-clusters as circumscribing rectangles; Transforming the conormal cluster and the sub-clusters into a rectangular plane; A three-dimensional model generation program, characterized by generating the three-dimensional model in which a target location is represented by a set of a plurality of the rectangular planes.

Images