3D model generation device, 3D model generation method, and 3D model generation program
The 3D model generation device and method address the challenge of reproducing complex surfaces and occlusions by segmenting and determining component types in point clouds, resulting in accurate 3D models of building interiors.
Patent Information
- Application Number
- JP2025078987
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2025-05-09
- Publication Date
- 2026-11-19
AI Technical Summary
Conventional 3D model generation techniques struggle to accurately reproduce complex surfaces and missing components in building interiors due to occlusions, leading to incomplete or inaccurate 3D models.
A 3D model generation device and method that performs segmentation to extract flat surfaces from a point cloud, determines whether segments represent external or internal components, and generates a 3D model based on these segments, using preprocessing and optimization processes to enhance accuracy.
The method effectively reproduces complex surfaces and missing components in the 3D model, even with defects in the point cloud, ensuring accurate representation of building components.
Smart Images

Figure 2026182615000001_ABST
Abstract
Description
[Technical Field]
[0001] The present invention relates to a 3D model generation apparatus, a 3D model generation method, and a 3D model generation program that generate a 3D model of the components of a target space based on a restored point cloud obtained by a 3D restoration process of the target space within a building. [Background technology]
[0002] Conventionally, a 3D reconstruction technique is known that generates point cloud data (reconstructed point cloud) as 3D spatial information about a target space based on images taken of the target space within a building. Furthermore, a technique is known that generates a 3D model of the target space from the point cloud data of the target space.
[0003] In particular, a technique has been proposed for hybrid 3D reconstruction of indoor scenes that integrates object recognition. According to this technique, a point cloud is divided into internal and external components, and each is reconstructed using different methods. Specifically, the external components are reconstructed using plane estimation to represent the room shape, while the internal components are reconstructed by recognizing objects using a random forest and fitting a CAD model. A lightweight 3D reconstruction model is then generated by merging the reconstructed internal and external components (see Non-Patent Literature 1).
[0004] Furthermore, there is a known technique that reads a 3D point cloud acquired from the interior of a building, divides it into clusters according to the distance between the points, sequentially polygonizes the floor, ceiling, and wall surfaces based on each cluster, separates the 3D point clouds of structures other than the floor, ceiling, and wall surfaces based on their positional relationship with the generated polygons, and generates a 3D model of structures other than the building using primitives prepared in advance for structures from the separated 3D point clouds (see Patent Document 1). [Prior art documents] [Non-patent literature]
[0005] [Non-Patent Document 1] Mingfan Li, 3 others, "Hybrid 3D Reconstruction of Indoor Scenes Integrating Object Recognition," [online], February 8, 2024, MDPI, [Searched March 4, 2020], Internet<https: / / www.mdpi.com / 2072-4292 / 16 / 4 / 638> [Patent Documents]
[0006] [Patent Document 1] Japanese Patent Publication No. 2019-168976 [Overview of the Initiative] [Problems that the invention aims to solve]
[0007] According to conventional techniques, a lightweight or more accurate 3D model of the building can be generated by separating internal and external components from point cloud data within the building and performing a reconstruction process.
[0008] However, rooms may contain beams, columns, and steps in addition to floors, ceilings, and walls. In such cases, conventional technology has the problem of not being able to accurately reproduce the complex surfaces of the room's components in a 3D model. Furthermore, problems such as missing points due to occlusion can occur in the reconstructed point cloud of the target space, resulting in missing components of the room being reproduced in the 3D model.
[0009] Therefore, the main objective of the present invention is to provide a 3D model generation device, a 3D model generation method, and a 3D model generation program that can generate a 3D model in which the components of the target space are appropriately reproduced, even when the components of the target space are composed of complex surfaces and when there are defects in the restored point cloud of the target space. [Means for solving the problem]
[0010] The present invention provides a 3D model generation apparatus that uses a processor to generate a 3D model of the components of a target space based on a restored point cloud obtained by a 3D restoration process of the target space within a building. The processor performs a segmentation process to extract a plurality of segments representing flat surfaces from the restored point cloud, performs an automatic segment determination process to determine whether the segments extracted in the segmentation process are modeling targets representing external components or non-modeling targets representing internal components, and performs a modeling process to generate the 3D model representing the external components based on the segments determined to be modeling targets in the automatic segment determination process.
[0011] Furthermore, the present invention provides a 3D model generation method in which a processor performs a process to generate a 3D model relating to the components of a target space based on a restored point cloud obtained by a 3D restoration process relating to a target space within a building, wherein the method includes a segmentation process to extract multiple segments representing flat surfaces from the restored point cloud, an automatic segment determination process to determine whether the segments extracted in the segmentation process are modeling targets representing external components or non-modeling targets representing internal components, and a modeling process to generate the 3D model representing the external components based on the segments determined to be modeling targets in the automatic segment determination process.
[0012] Further, the three-dimensional model generation program of the present invention is a three-dimensional model generation program that causes a processor to execute processing for generating a three-dimensional model related to the constituent elements of a target space based on a restored point cloud obtained by three-dimensional restoration processing related to the target space inside a building, the program being configured to execute segmentation processing for extracting a plurality of segments representing flat surfaces from the restored point cloud, execute automatic segment determination processing for determining whether each segment extracted by the segmentation processing is a modeling target representing an external constituent element or a non-modeling target representing an internal constituent element, and execute modeling processing for generating the three-dimensional model representing the external constituent element based on the segments determined to be the modeling target in the automatic segment determination processing. [Effects of the Invention]
[0013] According to the present invention, segments representing flat surfaces are extracted from the restored point cloud, it is determined whether each segment is a modeling target representing an external constituent element, and a three-dimensional model representing the external constituent element is generated using the segments representing the external constituent element. Thereby, even when the constituent elements of the target space are configured with complex surfaces and a defect occurs in the restored point cloud of the target space, it is possible to generate a three-dimensional model in which the constituent elements of the target space are appropriately reproduced. [Brief Description of the Drawings]
[0014] [Figure 1] Overall configuration diagram of the three-dimensional model generation system according to the present embodiment [Figure 2] Explanatory diagram showing the restored point cloud of a room (target space) and the restored point cloud in a segmented state [Figure 3] Explanatory diagram showing segments representing external constituent elements of a room (external segments) and segments representing internal constituent elements of the room (internal segments) [Figure 4] Explanatory diagram showing a three-dimensional model representing external constituent elements of a room [Figure 5] Block diagram showing the schematic configuration of a user terminal [Figure 6] Block diagram showing the schematic configuration of a server [Figure 7] Flowchart showing the procedure of 3D model generation processing performed by the server [Figure 8] Flowchart showing the procedure of preprocessing performed by the server [Figure 9] Flowchart showing the procedure of segmentation processing performed by the server [Figure 10] Explanatory diagram showing the detection status of line segments [Figure 11] Flowchart showing the procedure of automatic inside-outside determination processing performed by the server [Figure 12] Explanatory diagram showing the status of floating surface determination processing [Figure 13] Explanatory diagram showing the maximum inter-surface distance [Figure 14] Explanatory diagram showing the outline of bounding box determination processing [Figure 15] Flowchart showing the procedure of modeling processing performed by the server [Figure 16] Explanatory diagram showing the generation status of initial surfaces and candidate surfaces [Figure 17] Explanatory diagram showing the outline of coverage score and distance penalty [Figure 18] Explanatory diagram showing the procedure for calculating the same-region segmentation cost [Figure 19] Explanatory diagram showing the procedure for calculating the same-region segmentation cost [Figure 20] Explanatory diagram showing the procedure for calculating the same-region segmentation cost [Figure 21] Explanatory diagram showing the basic screen displayed on the user terminal [Figure 22] Explanatory diagram showing the restored point cloud display screen displayed on the user terminal [Figure 23] Explanatory diagram showing the segment display screen displayed on the user terminal [Figure 24] Explanatory diagram showing a state where the result of the automatic inside-outside determination processing is reflected on the segment display screen [Figure 25] Explanatory diagram showing the 3D model display screen in a state where the interpolation region displayed on the user terminal is displayed [Figure 26]This is an explanatory diagram showing the 3D model display screen with the error region displayed on the user's terminal. [Figure 27] Diagram illustrating the settings screen displayed on the user's terminal. [Modes for carrying out the invention]
[0015] The first invention made to solve the aforementioned problems is a 3D model generation device that uses a processor to perform a process of generating a 3D model relating to the components of a target space based on a restored point cloud obtained by a 3D restoration process relating to the target space within a building, wherein the processor performs a segmentation process to extract a plurality of segments representing flat surfaces from the restored point cloud, performs an automatic segment determination process to determine whether the segments extracted in the segmentation process are modeling targets representing external components or non-modeling targets representing internal components, and performs a modeling process to generate the 3D model representing the external components based on the segments determined to be modeling targets in the automatic segment determination process.
[0016] According to this method, segments representing flat surfaces are extracted from the reconstructed point cloud, it is determined whether or not each segment represents an external structure to be modeled, and a 3D model representing the external structure is generated using the segments representing the external structure. As a result, even if the structures in the target space are composed of complex surfaces, or if there are gaps in the reconstructed point cloud of the target space, a 3D model that appropriately reproduces the structures in the target space can be generated.
[0017] Furthermore, the second invention provides that, in the modeling process, the processor generates candidate faces that are candidates for constituent faces of the 3D model based on the segment to be modeled, calculates an index value for each candidate face regarding its suitability as a constituent face of the 3D model based on the distribution of points on that candidate face, selects a candidate face that is suitable as a constituent face of the 3D model based on that index value, and generates the 3D model relating to the external components to be modeled based on the selected candidate face.
[0018] According to this method, it is possible to generate a highly reproducible 3D model of the external components of the target space.
[0019] Furthermore, the third invention is configured such that, before executing the segmentation process, the processor performs noise reduction, downsampling, and coordinate axis setting processes on the restored point cloud as preprocessing.
[0020] This allows for improved accuracy in subsequent processing. In addition, normal estimation, flatness estimation, and floor surface estimation may also be performed.
[0021] Furthermore, the fourth invention is configured such that the processor, in the segmentation process, performs a point cloud division process that divides the restored point cloud into multiple segments by a region expansion method that expands the region by sequentially adding neighboring points, and also performs a linear segment detection process that detects linear segments that are not to be modeled from among the segments obtained in the point cloud division process.
[0022] According to this method, segments representing flat surfaces can be appropriately extracted from the restored point cloud.
[0023] Furthermore, the fifth invention is configured such that, in the automatic segment determination process, the processor groups adjacent segments into one group, determines that the segment included in the group with the maximum number of points is a modeling target representing an external component, and determines that the segment not included in the group with the maximum number of points is an external component that forms a floating surface not in contact with an external component and is not a modeling target.
[0024] According to this, floating surfaces that are not in contact with external components can be considered as forming internal components, allowing for the appropriate determination of segments representing internal components of a building.
[0025] Furthermore, the sixth invention provides that, in the automatic segment determination process, the processor groups segments whose normal direction is the direction of each coordinate axis of the three-dimensional orthogonal coordinate system for each coordinate axis, determines the maximum distance between segments in each coordinate axis group as the maximum inter-face distance for each coordinate axis, sets a three-dimensional bounding box for each segment with sides parallel to each coordinate axis, and determines that a segment is a modeling target representing an external structure if the ratio of the length of the longest side to the maximum inter-face distance with respect to the coordinate axis parallel to the longest side of the three-dimensional bounding box exceeds a predetermined threshold.
[0026] According to this, segments that are sufficiently long relative to the overall dimensions of the target space can be appropriately identified as segments representing external components of the room.
[0027] Furthermore, the seventh invention is configured such that, in the modeling process, the processor calculates a point cloud distribution index value as the index value, which represents the proportion of the range in which point clouds are distributed on the candidate surface, and performs an optimization process to select the candidate surface that is appropriate as a constituent surface of the 3D model based on the point cloud distribution index value.
[0028] According to this, it is possible to select appropriate candidate surfaces as constituent surfaces of a 3D model.
[0029] Furthermore, the eighth invention is configured such that, in the modeling process, the processor calculates a distance index value as the index value, which represents the degree of inappropriateness regarding the distance of the candidate surface from its original segment, and performs an optimization process to select a candidate surface that is appropriate as a constituent surface of the 3D model based on the distance index value.
[0030] According to this, it is possible to select appropriate candidate surfaces as constituent surfaces of a 3D model.
[0031] Furthermore, the ninth invention is configured such that the processor calculates a region division index value as the index value, which represents the degree of inappropriateness regarding the combination of two candidate faces that cause the division of the segment that will become the candidate face, and performs an optimization process to select a candidate face that is appropriate as a constituent face of the 3D model based on the region division index value.
[0032] According to this, it is possible to select appropriate candidate surfaces as constituent surfaces of a 3D model.
[0033] Furthermore, the tenth invention is configured such that the processor presents the determination result of the automatic segment determination process to the user and executes a manual segment setting process that modifies the determination result of the automatic segment determination process in response to the user's operation of selecting a segment and specifying whether or not it is a modeling target.
[0034] According to this, if there is an error in the result of the automatic segment determination process, the user can easily correct that error.
[0035] Furthermore, the 11th invention is a 3D model generation method in which a processor performs a process to generate a 3D model relating to the components of a target space based on a restored point cloud obtained by a 3D restoration process relating to the target space within a building, wherein the method is configured to perform a segmentation process to extract a plurality of segments representing flat surfaces from the restored point cloud, perform an automatic segment determination process to determine whether the segments extracted in the segmentation process are modeling targets representing external components or non-modeling targets representing internal components, and perform a modeling process to generate the 3D model representing the external components based on the segments determined to be modeling targets in the automatic segment determination process.
[0036] According to this, similar to the first invention, even if the components of the target space are composed of complex surfaces, and even if there are gaps in the restored point cloud of the target space, it is possible to generate a 3D model in which the components of the target space are appropriately reproduced.
[0037] Furthermore, the twelfth invention is a 3D model generation program that uses a processor to generate a 3D model of the components of a target space based on a restored point cloud obtained by a 3D restoration process of the target space within a building, wherein the program performs a segmentation process to extract multiple segments representing flat surfaces from the restored point cloud, performs an automatic segment determination process to determine whether the segments extracted in the segmentation process are modeling targets representing external components or non-modeling targets representing internal components, and performs a modeling process to generate the 3D model representing the external components based on the segments determined to be modeling targets in the automatic segment determination process.
[0038] According to this, similar to the first invention, even if the components of the target space are composed of complex surfaces, and even if there are gaps in the restored point cloud of the target space, it is possible to generate a 3D model in which the components of the target space are appropriately reproduced.
[0039] Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[0040] Figure 1 is an overall diagram of the 3D model generation system according to this embodiment.
[0041] This system performs a 3D reconstruction process on a target space within a building to obtain a reconstructed point cloud of the target space, and then generates a 3D model of the components of the target space based on that reconstructed point cloud. In this embodiment, a 3D model is generated for a room (living room) where furniture is placed, but the target space is not limited to a room and may be a warehouse, workshop, etc.
[0042] This system consists of a user terminal 1 (display device) and a server 2 (3D model generation device). User terminal 1 and server 2 are connected via a network such as a LAN or the internet.
[0043] User terminal 1 comprises a main unit 11 and a camera 12. User terminal 1 can be a tablet or a notebook PC.
[0044] Camera 12 is a visible light camera, that is, a monocular camera that detects visible light and photographs the subject, and outputs a captured image, for example, an RGB color image. In addition to camera 12 (visible light camera), user terminal 1 may also be equipped with sensors such as a depth camera for measuring the distance to the subject and an IMU (Inertial Measurement Unit) for detecting three-dimensional angular velocity and acceleration, for the 3D reconstruction performed on server 2.
[0045] The user (worker) moves through the target space (room) while holding the main unit 11 of the user terminal 1. At this time, the target space is photographed by the camera 12 on the user terminal 1, and images of each point in the target space are acquired sequentially. Here, an example is shown in which images are acquired by a user walking through the target space while holding the main unit 11, but images of the target space may also be acquired using a self-propelled robot equipped with the user terminal 1 or the like.
[0046] Server 2 is composed of a PC. Based on the captured images received from user terminal 1, Server 2 performs 3D reconstruction processing on the target space and generates a reconstructed point cloud of the target space. Server 2 also generates a 3D model of the target space based on the reconstructed point cloud of the target space.
[0047] Next, we will explain the overview of the 3D model generation process performed on Server 2. Figure 2 is an explanatory diagram showing the restored point cloud of the room (target space) and the restored point cloud of the segment division state. Figure 3 is an explanatory diagram showing the segment representing the external components of the room (external segment) and the segment representing the internal components of the room (internal segment). Figure 4 is an explanatory diagram showing the 3D model (room model) representing the external components of the room. In the examples shown in Figures 3, 4, and 5, the viewpoint is set to the outside of the room, and the interior of the room is made visible by making the wall and ceiling surfaces on the side closer to the viewpoint transparent.
[0048] On server 2, as shown in Figure 2(A), a reconstructed point cloud of the room (target space) is generated by a 3D reconstruction process based on captured images of the room. In the example shown in Figure 2(A), the external components of the room include a floor, ceiling, and walls. Furthermore, there is a beam-like step at the boundary between the ceiling and the wall, and a columnar protrusion at the boundary between the two walls, resulting in the external components of the room being composed of complex surfaces. The internal components of the room include a sofa, desk, chair, and shelves.
[0049] Here, a 3D model of only the external components of a room, that is, a 3D model of a room without any furniture or other objects inside, can be used, for example, to conduct on-site inspections of buildings without visiting the site, or to simulate room layout changes or renovations. For this reason, in this embodiment, segments (sub-point clouds) representing the flat surfaces of the external components are extracted from the reconstructed point cloud of the room, and a 3D model representing the external components is generated based on these segments.
[0050] Specifically, as shown in Figure 2(B), when segmentation processing is performed on server 2, multiple segments (sub-point clouds) representing flat surfaces are extracted from the reconstructed point cloud of the room. Each extracted segment has a different position and orientation. In this embodiment, the segmentation processing extracts flat surfaces as segments using the region expansion method, but small flat areas, such as the legs of a desk or chair, are not extracted as segments representing flat surfaces. Figure 2(B) shows an example where the tabletop of the desk and the seat and back of the chair are extracted as segments.
[0051] Next, as shown in Figures 3(A) and (B), Server 2 performs a process (internal / external sorting) to divide the extracted segments into external segments representing external components of the room and internal segments representing internal components of the room. As shown in Figure 3(A), the external segment group includes segments representing external components of the room such as floors, ceilings, walls, beams, columns, and steps. As shown in Figure 3(B), the internal segment group includes segments representing internal components of the room such as furniture like sofas, desks (tabletops), chairs (seats and backs), and shelves. The internal segment group also includes linear segments that appear at the boundary between two surfaces of external components.
[0052] In this embodiment, it is determined whether the segments extracted from the restored point cloud correspond to external or internal components of a room, and based on the determination result, the segments are divided into external segments representing external components and internal segments representing internal components (automatic internal / external determination process).
[0053] Furthermore, in this embodiment, segments extracted from the restored point cloud are presented to the user, and in response to the user's operation (internal / external sorting operation) specifying whether they belong to internal or external components, the selected segments are sorted into external segments representing external components and internal segments representing internal components (manual internal / external setting process).
[0054] Next, as shown in Figure 4, Server 2 generates a 3D model representing the external components based on the external segments representing the external components. In this 3D model, the internal components of the room, i.e., furniture such as sofas, desks, chairs, and shelves placed in the room, are removed. In addition, as external components of the room, complex surfaces such as beams, walls, and steps are appropriately reproduced, as well as the floor, ceiling, and walls.
[0055] In this embodiment, an internal / external sorting process is performed to generate a 3D model representing the external components of the room, i.e., a 3D model in which the internal components of the room (such as furniture) have been removed. However, an internal / external sorting process may not be performed, and a 3D model in which the internal components of the room remain in place may be generated. Such a 3D model in which the internal components of the room remain in place can be used for simulations to analyze the physical characteristics of the room, such as simulations to analyze acoustic characteristics or air conditioning characteristics.
[0056] Furthermore, in this embodiment, segments representing the external components of a room are set as the modeling target, and a 3D model representing the external components of a room is generated. However, segments representing components required by the user may be set as the modeling target, and a 3D model representing the components required by the user may be generated.
[0057] Furthermore, in this embodiment, segments representing flat surfaces are extracted from the reconstructed point cloud of the room as a partial point cloud that forms the basis of the constituent surfaces of the 3D model. However, in addition to segments representing flat surfaces, segments representing basic shapes such as cylindrical surfaces may also be extracted, and the constituent surfaces of the 3D model may be formed based on the segments representing these basic shapes.
[0058] Furthermore, when the goal is to generate a 3D model representing the external components, which are room models, it is desirable to restrict the segmentation process of the restored point cloud by using area conditions of the constituent surfaces to prevent the extraction of fine segments.
[0059] Next, we will describe the general configuration of user terminal 1. Figure 5 is a block diagram showing the general configuration of user terminal 1.
[0060] In addition to the camera 12, the user terminal 1 includes a display 13 (display unit), an input device 14, a storage unit 15, a processor 16 (CPU), and a communication unit 17.
[0061] The display 13 displays various operation screens related to shooting and editing operations, setting screens for setting various processing conditions, and viewing screens for checking the generated restored point cloud and 3D model. The input device 14 is used by the user for input operations. If the user terminal 1 is a tablet terminal, a touch panel display is provided in which the touch panel as the input device 14 and the display panel as the display 13 are integrated.
[0062] The memory unit 15 stores programs executed by the processor 16, etc. The memory unit 15 also stores images captured by the camera 12. The captured images are transmitted to the server 2 at appropriate intervals.
[0063] The processor 16 executes programs stored in the memory unit 15 to perform various processes related to taking pictures with the camera 12, displaying images on the display 13, and acquiring operation information from the input device 14.
[0064] The communication unit 17 communicates with the server 2. Specifically, the communication unit 17 transmits images captured by the camera 12 and operation information acquired by the user through operation of the input device 14 to the server 2. The communication unit 17 also receives various screen display information transmitted from the server 2.
[0065] Next, we will describe the general configuration of Server 2. Figure 6 is a block diagram showing the general configuration of Server 2.
[0066] Server 2 comprises a communication unit 21, a storage unit 22, and a processor 23.
[0067] The communication unit 21 communicates with the user terminal 1. Specifically, the communication unit 21 receives captured images and operation information transmitted from the user terminal 1. The communication unit 21 also transmits display information for various screens to be displayed on the user terminal 1 to the user terminal 1.
[0068] The memory unit 22 stores programs executed by the processor 23, etc. The memory unit 22 also stores data such as the restored point cloud, segments, and 3D model of the target space generated by the processor 23.
[0069] The processor 23 performs various processes by executing programs stored in the memory unit 22. In this embodiment, the processor 23 performs 3D reconstruction processing, 3D model generation processing, focus region detection processing, and display processing.
[0070] In the 3D reconstruction process, the processor 23 generates a reconstructed point cloud of the target space based on the captured image received from the user terminal 1. The 3D reconstruction process can be realized, for example, by estimating the position and orientation of the camera 12 at the time of capture in the world coordinate system using the SLAM (Simultaneous Localization and Mapping) method, and then superimposing depth information obtained from a depth camera or the like onto the world coordinate system based on this position and orientation estimation information.
[0071] In the 3D model generation process, processor 23 generates a 3D model of the room based on the reconstructed point cloud of the room generated in the 3D reconstruction process. The 3D model generation process includes preprocessing, segmentation, internal / external sorting, and modeling.
[0072] In the preprocessing stage, processor 23 performs processes such as noise reduction, downsampling, and setting of coordinate axes on the reconstructed point cloud generated by the 3D reconstruction process.
[0073] In the segmentation process, the processor 23 extracts multiple segments (sub-point clouds) representing flat surfaces from the pre-processed restored point cloud.
[0074] In the internal / external sorting process, the processor 23 sorts each segment extracted in the segmentation process into an external segment representing the external components of the room and an internal segment representing the internal components of the room. The internal / external sorting process includes an automatic internal / external determination process (automatic segment determination process) and a manual internal / external setting process (manual segment setting process).
[0075] In the automatic internal / external determination process, the processor 23 determines whether the extracted segment corresponds to an external or internal component of the room, and then divides the segment into an internal segment representing an internal component and an external segment representing an external component.
[0076] In the manual internal / external configuration process, the processor 23 distributes the selected segments into internal segments representing internal components and external segments representing external components, in response to a user operation (internal / external distribution operation) specifying whether the extracted segments belong to internal or external components.
[0077] In the modeling process, the processor 23 generates a 3D model (room model) representing the external components of the room based on the external segments obtained in the internal / external sorting process (automatic internal / external determination process, manual internal / external setting process). Alternatively, in the modeling process, a 3D model (room model) with furniture and other items placed inside the room may be generated based on all the segments extracted in the segmentation process.
[0078] In the region of interest detection process, the processor 23 detects regions where there is a significant discrepancy between the 3D model generated in the 3D model generation process and the restored point cloud, and identifies these regions of interest. The region of interest detection process includes interpolation region detection, interpolation reason determination, and error region detection.
[0079] In the interpolation region detection process, the processor 23 compares the restored point cloud with the 3D model and detects as an interpolation region a region in which a surface of the 3D model is formed in a manner that interpolates the missing points in the restored point cloud.
[0080] In this embodiment, when determining the constituent surfaces of a 3D model from segments during the modeling process, the surfaces represented by the segments are first extended to the entire range of the reconstructed point cloud, and then surfaces are selected based on the distribution of the point cloud. As a result, in the 3D model, surfaces are formed with interpolation applied to areas in the reconstructed point cloud where there are no points.
[0081] In the interpolation reason determination process, the processor 23 determines the interpolation reason for the interpolation region detected in the interpolation region detection process, based on the restored point cloud generated in the 3D reconstruction process and the position and orientation estimation information obtained by the SLAM method in the 3D reconstruction process.
[0082] Specifically, it is determined whether the interpolation region is included in the field of view (shooting range) of camera 12. It is also determined whether occlusion occurs in the interpolation region due to other objects. If the interpolation region is included in the field of view of camera 12 and occlusion does not occur due to other objects, it is determined to be interpolation of a point cloud area with no points due to an opening. If the interpolation region is included in the field of view of camera 12 and occlusion occurs due to other objects, it is determined to be interpolation of a point cloud gap due to occlusion. If the interpolation region is not included in the field of view of camera 12, it is determined to be interpolation of a point cloud gap due to a missed shot. Note that point clouds are not formed in openings such as doorways and windows. Also, point clouds are not formed when the floor or walls are obscured by furniture such as sofas or shelves.
[0083] In the error region detection process, the processor 23 compares the restored point cloud with the 3D model and detects regions where a large error occurs in the 3D model relative to the restored point cloud as error regions. Specifically, the error of the 3D model relative to the restored point cloud (the difference between the restored point cloud and the 3D model) is calculated (error calculation process), and regions where this error is greater than a predetermined threshold are detected as error regions. At this time, for example, corresponding points are found in the restored point cloud and the 3D model (mesh model), and the error is calculated as a percentage based on the distance between these corresponding points.
[0084] Furthermore, the interpolation region may be detected overlapping with the error region. That is, since the interpolation region is a region where a surface of the 3D model is formed in a manner that interpolates the missing parts of the restored point cloud, the error of the 3D model relative to the restored point cloud also becomes large in the interpolation region. For this reason, a region in which a large error occurs in the 3D model relative to the restored point cloud, and which does not correspond to the interpolation region, may be detected as the error region.
[0085] In the display process, the processor 23 generates a screen for instructions, confirmations, and settings related to the 3D reconstruction process and the 3D model generation process, and displays this screen on the display 13 of the user terminal 1.
[0086] The display process includes the restoration point cloud display process, the segment display process, the 3D model display process, and the focus region display process.
[0087] In the restored point cloud display process, the restored point cloud generated in the 3D restoration process is converted into a 2D image based on the specified viewpoint and displayed on the screen of user terminal 1. At this time, the display mode of the restored point cloud changes according to the user's 3D operations, that is, operations that specify the position and angle of the viewpoint in 3D space.
[0088] In the segment display process, the restored point cloud, which reflects the extraction results from the segmentation process, i.e., the restored point cloud in the segmented state, is converted into a 2D image based on the specified viewpoint and displayed on the screen of user terminal 1. At this time, multiple segments included in the restored point cloud are displayed in different colors according to a predetermined color coding rule. Furthermore, the display manner of the restored point cloud in the segmented state changes in response to the user's 3D operation, i.e., the operation of specifying the position and angle of the viewpoint in 3D space.
[0089] In the 3D model display process, the 3D model generated in the 3D model generation process is converted into a 2D image based on a specified viewpoint and displayed on the screen of user terminal 1. At this time, the display mode of the 3D model changes according to the user's 3D operations, that is, operations that specify the position and angle of the viewpoint in 3D space.
[0090] The process for displaying the area of interest includes the interpolation area display process, the interpolation reason display process, and the error area display process.
[0091] In the interpolation region display process, the processor 23 displays the interpolation region detected in the interpolation region detection process on the 3D model (room model) displayed on the screen of the user terminal 1 in a way that allows it to be distinguished from other regions. Specifically, the interpolation region is highlighted in a different color from other regions. Furthermore, the interpolation region can be displayed in a different color depending on the reason for interpolation. In addition, the interpolation region can be displayed in a color specified by the user.
[0092] In the interpolation reason display process, the processor 23 displays the interpolation reasons determined in the interpolation reason determination process on the 3D model (room model) displayed on the screen of the user terminal 1. Specifically, the interpolation reasons for the selected interpolation area are displayed in response to the user's operation (mouse over) to select an interpolation area.
[0093] In the error region display process, the processor 23 displays the error regions detected in the error region detection process on the 3D model (room model) displayed on the screen of the user terminal 1. Specifically, the error regions are highlighted in a different color from other regions. Furthermore, the error regions can be displayed in a color specified by the user.
[0094] Next, we will explain the 3D model generation process performed on Server 2. Figure 7 is a flowchart showing the steps of the 3D model generation process.
[0095] In the 3D model generation process, the processor 23 first performs preprocessing (ST101). In the preprocessing, the reconstructed point cloud generated in the 3D reconstruction process is subjected to processes such as noise reduction, downsampling, and setting of coordinate axes.
[0096] Next, processor 23 performs segmentation (ST102). In segmentation, multiple segments (sub-point clouds) representing flat surfaces are extracted from the pre-processed restored point cloud.
[0097] Next, when the user instructs the system to perform automatic interior / exterior determination (Yes in ST103), the processor 23 executes the automatic interior / exterior determination process (ST104). In the automatic interior / exterior determination process, it is determined whether the segments extracted in the segmentation process belong to the exterior or interior components of the room, and the segments are then divided into internal segments representing interior components and external segments representing exterior components.
[0098] Next, when the user performs an internal / external sorting operation (Yes in ST105), the processor 23 executes a manual internal / external setting process (ST106). In the manual internal / external setting process, the selected segment is sorted into an internal segment representing an internal component and an external segment representing an external component, according to the user's operation (internal / external sorting operation) specifying whether the segment is an internal component or an external component.
[0099] Next, processor 23 performs the modeling process (ST107). In the modeling process, a 3D model (room model) representing the external components of the room is generated based on the external segments obtained in the internal / external sorting process (automatic internal / external determination process, manual internal / external setting process).
[0100] Here, depending on whether or not an operation to instruct automatic inside / outside determination is performed (Yes / No in ST103) and whether or not an inside / outside sorting operation is performed (Yes / No in ST105), there are four patterns: one in which the automatic inside / outside determination process and the manual inside / outside setting process are performed sequentially; one in which only the automatic inside / outside determination process is performed; one in which only the manual inside / outside setting process is performed; and one in which neither the automatic inside / outside determination process nor the manual inside / outside setting process is performed. Furthermore, in the pattern in which the automatic inside / outside determination process and the manual inside / outside setting process are performed sequentially, the manual inside / outside setting process is performed to correct the result of the automatic inside / outside determination process.
[0101] Next, we will explain the preprocessing performed on server 2 (ST101 in Figure 7). Figure 8 is a flowchart showing the preprocessing procedure.
[0102] The preprocessing steps include noise reduction (ST201), downsampling (ST202), normal estimation (ST203), flatness estimation (ST204), floor surface estimation (ST205), and coordinate axis setting (ST206), performed sequentially.
[0103] In the noise reduction process (ST201), processor 23 removes noise (isolated points) contained in the restored point cloud.
[0104] In the downsampling process (ST202), the processor 23 performs downsampling of the point cloud using a voxel grid filter. The size of the voxel grid filter (voxel size) is set to, for example, 0.01m.
[0105] In the normal estimation process (ST203), processor 23 estimates the normal vector (point normal) of each point included in the restored point cloud.
[0106] In the flatness estimation process (ST204), processor 23 estimates the flatness of each point included in the restored point set. At this time, assuming that the eigenvalues of the covariance matrix of neighboring points are e0>e1>e2 in descending order, the flatness for each point is calculated as (e1-e2) / e0.
[0107] In the floor surface estimation process (ST205), the processor 23 estimates the horizontal plane that will be the floor of the room from the restored point cloud. Specifically, first, points where the dot product of the point normal estimated in the normal estimation process and the unit vector [0,0,1] in the z-axis direction exceeds a predetermined threshold (e.g., 0.9) are extracted as the horizontal plane (sub-point cloud). Next, a histogram of the z-coordinates of each point in the point cloud is obtained, and the first and second most important peaks are detected as corresponding to the floor and ceiling of the room, and the one with the smaller z-coordinate value is determined as the floor height. Note that the vertical direction of the point cloud can be set based on the detection data from the IMU installed in the user terminal 1, so the z-axis is known.
[0108] In the coordinate axis setting process (ST206), the processor 23 sets the x and y axes of the 3D Cartesian coordinate system (x, y, and z axes) for the restored point cloud. The z axis is perpendicular to the floor surface obtained in the floor surface estimation process, and the x and y axes are aligned with the walls (vertical planes) of the room. At this time, the RANSAC (Random Sample Consensus) method is used to eliminate the influence of outliers regarding the direction of the reference maximum vertical plane, and estimation considering errors is performed. In addition, translation and rotation are performed on the entire restored point cloud so that the floor surface becomes z=0, the center of the restored point cloud becomes the origin, and the maximum vertical plane becomes the x-axis direction.
[0109] Next, we will explain the segmentation process performed on server 2 (ST102 in Figure 7). Figure 9 is a flowchart showing the procedure for the segmentation process. Figure 10 is an explanatory diagram showing the detection status of line segments.
[0110] In the segmentation process, multiple segments (sub-point clouds) representing flat surfaces are extracted from the reconstructed point cloud.
[0111] As shown in Figure 9, the segmentation process sequentially performs point cloud division using the region expansion method (ST301) and linear segment detection (ST302).
[0112] In the point cloud division process using the domain expansion method (ST301), the reconstructed point cloud is divided into multiple segments (sub-point clouds) representing flat surfaces using the domain expansion method. In the domain expansion method, the process of searching for neighboring points starts from the seed point, and the found neighboring points are added to the region of the segment. In this embodiment, when setting the initial point and the seed point for domain expansion, flatness is used as the selection criterion instead of curvature. This makes it easier to select flat and highly isotropic points, and prevents corners such as the boundary between the ceiling and the wall from being selected as the starting point. The search range for neighboring points may also be set to, for example, 0.05m. Furthermore, as a condition for adding neighboring points to the region of the segment, the angle between the point normals of the initial point and the neighboring points may be set to be less than a threshold (for example, 30°).
[0113] Specifically, each surface of the external components of a room (floor, ceiling, walls, beams, columns, steps, etc.) and each surface of the internal components of a room (furniture such as sofas, desks, chairs, and shelves) are divided into separate segments (see Figure 2(B)). At this time, multiple segments with different positions and orientations are extracted. On the other hand, point clouds that do not form flat surfaces, i.e., point clouds representing surfaces or edges with high curvature (for example, the legs of a desk or chair), are not extracted as segments.
[0114] In the linear segment detection process (ST302), the processor 23 detects linear segments that appear at the boundary between two surfaces, etc. (see Figure 10). The detected linear segments are excluded from the external segments to be modeled.
[0115] Specifically, a rectangular prism (3D bounding box) with the minimum volume surrounding the segment is defined, and a segment is determined to be linear if the length of the middle side L1 among the three side lengths L0, L1, and L2 (L0≧L1≧L2) in that rectangular prism is less than a threshold (e.g., 0.05m). Additionally, a segment is determined to be linear if the ratio L2 / L1 (the ratio of the shortest side length L2 to the middle side length L1) is less than a threshold (e.g., 0.5).
[0116] Next, we will explain the automatic inside / outside determination process performed on server 2 (ST104 in Figure 7). Figure 11 is a flowchart showing the procedure for the automatic inside / outside determination process. Figure 12 is an explanatory diagram showing the status of the floating surface determination process. Figure 13 is an explanatory diagram showing the maximum inter-surface distance. Figure 14 is an explanatory diagram showing an overview of the bounding box determination process.
[0117] In the automatic interior / exterior determination process, each segment extracted in the segmentation process is determined to be either an exterior or interior component of the room, and each segment is then classified into an exterior segment representing the exterior components of the room and an interior segment representing the interior components of the room.
[0118] As shown in Figure 11, the automatic inside / outside determination process sequentially performs the following steps: floating surface determination process (ST401), segment integration process (ST402), maximum inter-surface distance acquisition process (ST403), and bounding box determination process (ST404).
[0119] In the floating surface determination process (ST401), the processor 23 determines whether each segment extracted in the segmentation process corresponds to a floating surface that is not in contact with the ceiling, walls, or floor. Segments that correspond to floating surfaces are excluded from the external segments to be modeled and are treated as internal segments.
[0120] Here, since the ceiling, walls, and floor, as external components, are in contact with each other, if we group adjacent segments together, the external segments representing external components can be grouped into one group. On the other hand, segments not included in this group become floating surfaces that are not in contact with the ceiling, walls, and floor, and these floating surfaces are determined to be internal segments. Furthermore, since the ceiling, walls, and floor have relatively large areas, the total number of points included in the group of segments representing external components will be large. Conversely, since the segments corresponding to floating surfaces have relatively small areas, the total number of points included in the group of segments corresponding to floating surfaces will be small.
[0121] Therefore, in this embodiment, it is assumed that the group with the highest score is composed of segments representing external components (ceiling, walls, and floor), and segments not included in the group with the highest score are determined to be segments corresponding to floating surfaces.
[0122] Specifically, first, the minimum distance between points in each segment is used as the inter-segment distance, and the inter-segment distance is calculated for all combinations of segments. Next, combinations of segments whose inter-segment distance is within a threshold (e.g., 0.05m) are determined to belong to the same group. Then, the scores of each segment in each group are added together to calculate the group score. Finally, segments that are not included in the group with the highest score are determined to be the floating surfaces.
[0123] In the example shown in Figure 12, during the floating surface determination process, segments representing external components such as floors, ceilings, walls, beam-like steps, columnar protrusions, sofas, and shelves are grouped together. On the other hand, segments representing some internal components such as desks (tabletops) and chairs (seats and backs) form a separate group from the segments representing external components. Here, the group of segments representing external components is identified as such based on the number of points in the point cloud, as it contains a large number of points.
[0124] Segments corresponding to floating surfaces are excluded from subsequent segment integration processing, maximum inter-surface distance acquisition processing, and bounding box determination processing.
[0125] In segment integration processing (ST402), the processor 23 integrates segments that are assumed to constitute the same plane.
[0126] Specifically, for each combination of segment A and segment B which has more points than segment A, the plane equation for the plane represented by segment B is determined, and the distance from each point in segment A to the plane represented by segment B is calculated. Next, points where the calculated distance is less than or equal to a predetermined threshold (e.g., 0.075 m) are counted. Then, if such points constitute a predetermined proportion (e.g., 75%) or more, the two segments A and B are determined to constitute the same plane, and the two segments A and B are merged. This process is performed for all combinations of segments.
[0127] In the maximum inter-face distance acquisition process (ST403), the processor 23 calculates the maximum distance between segments in each of the x, y, and z axes as the maximum inter-face distance. The maximum inter-face distance is used as an indicator of the size of the room.
[0128] In the example shown in Figure 13, the entrance door is open and the outside of the room is visible, causing some segments to protrude from the room. Similarly, when the outside of the room is visible through a transparent windowpane, some segments also protrude from the room. In such cases, if a 3D bounding box is set to enclose all segments, the size of that 3D bounding box will differ significantly from the actual size of the room, making it an unsuitable indicator of room size.
[0129] On the other hand, the maximum inter-surface distance represents the distance between opposing surfaces of the room's external components (floor, ceiling, walls, etc.), and serves as an indicator of the overall size of the room, even if some segments extend beyond the room's boundaries.
[0130] Specifically, first, the segments are grouped according to which axis direction (x, y, or z) the normal direction of each segment is closest to. Specifically, first, the plane equation for the plane represented by each segment is determined. Next, the dot product of the normal direction of the plane represented by each segment and the x, y, and z axes is calculated, and the axis that maximizes the absolute value of this dot product becomes the group to which that segment belongs. Next, for each group of x, y, and z axes, the centroid of all segments included in that group is calculated.
[0131] Next, the distance between the furthest centroids of the segments included in each group of the x, y, and z axes is determined as the maximum inter-plane distance for each of the x, y, and z axes. For example, in the case of the z-axis group, the segment with the maximum z-coordinate value and the segment with the minimum z-coordinate value are found, and the difference in the z-coordinate values of these two segments is the maximum inter-plane distance for the z axis. The maximum inter-plane distance can be determined similarly for the x-axis group and the y-axis group.
[0132] The x-axis group includes segments close to the vertical plane, where the segment with the maximum x-coordinate value and the segment with the minimum x-coordinate value represent two opposing walls, and the maximum distance between these walls in the x-axis direction represents the depth of the room. Similarly, the y-axis group includes segments close to the vertical plane, where the segment with the maximum y-coordinate value and the segment with the minimum y-coordinate value represent two opposing walls, and the maximum distance between these walls in the y-axis direction represents the width of the room. The z-axis group includes segments close to the horizontal plane, where the segment with the maximum z-coordinate value represents the ceiling, the segment with the minimum z-coordinate value represents the floor, and the maximum distance between these walls in the z-axis direction represents the ceiling height of the room.
[0133] In the bounding box determination process (ST404), the processor 23 determines whether a segment is an external segment representing an external component of the room or an internal segment representing an internal component of the room, based on the size of the segment relative to the overall size of the room. External components (floors, ceilings, walls, beams, columns, etc.) are assumed to have sufficient length in one of the x, y, or z axes relative to the overall size of the room. Therefore, segments that are sufficiently long relative to the dimensions of the room are determined to be external segments representing external components. On the other hand, segments that are not sufficiently long relative to the dimensions of the room, such as sofas and shelves, are not considered external segments and are determined to be internal segments representing internal components.
[0134] Specifically, as shown in Figure 14, a 3D bounding box is defined for each segment, with edges parallel to the x, y, and z axes. Next, the size of the 3D bounding box for each segment is compared with the maximum inter-face distances in each of the x, y, and z axes to determine whether the segment is an external or internal segment. At this time, the length L of the longest side of the 3D bounding box is determined. max The ratio (L) of the maximum inter-plane distance D along the axis parallel to its longest side max If the ratio (L) exceeds a predetermined threshold (e.g., 0.5), the segment is determined to be an external segment. max If / D) is below the threshold, the segment is determined to be an internal segment. In the example shown in Figure 14, the target segment is long in the y-axis direction, and the longest side of the 3D bounding box is parallel to the y-axis.
[0135] Next, we will explain the modeling process performed on Server 2 (ST107 in Figure 7). Figure 15 is a flowchart showing the steps of the modeling process. Figure 16 is an explanatory diagram showing the generation status of the initial surface and candidate surfaces. Figure 17 is an explanatory diagram showing an overview of the coverage score and distance penalty. Figures 18, 19, and 20 are explanatory diagrams showing how to calculate the cost of partitioning the same region.
[0136] In the modeling process, a 3D model (polygon mesh model) of the room is generated based on the segments to be modeled (external segments) that represent external components obtained through the internal / external sorting process (automatic internal / external determination process, manual internal / external setting process).
[0137] As shown in Figure 15(A), the modeling process sequentially performs candidate surface generation (ST501), candidate surface index calculation (ST502), and optimization (ST503).
[0138] In the candidate surface generation process (ST501), the processor 23 generates candidate surfaces that will be candidates for constituent surfaces of the 3D model, based on the segments to be modeled (external segments) obtained in the internal / external sorting process.
[0139] In the candidate surface index calculation process (ST502), the processor 23 calculates index values (coverage score, distance penalty, same-domain partitioning cost) for each candidate surface, which serve as indicators for deciding whether to adopt a candidate surface obtained in the candidate surface generation process, i.e., when adopting a candidate surface as a constituent surface of the 3D model. The index values represent the degree of appropriateness of the candidate surface as a constituent surface of the 3D model, depending on the degree of point cloud allocation (distribution of point clouds), etc.
[0140] In the optimization process (ST503), the processor 23 selects appropriate candidate faces from among the candidate faces obtained in the candidate face generation process, based on the index values obtained in the candidate face index calculation process, to be used as constituent faces of the 3D model. In the optimization process, the optimization solver is used to solve the question of whether or not to adopt each candidate face, that is, whether or not to adopt each candidate face as a constituent face of the 3D model, as an optimization problem, specifically a 0-1 integer programming problem, and the adoption status of the candidate faces is determined by the solution vectors 0 and 1.
[0141] As shown in Figure 15(B), in the candidate surface generation process (ST501 in Figure 15(A)), the processor 23 first sets a 3D bounding box that surrounds the entire restored point cloud (ST601). The 3D bounding box is circumscribing the entire restored point cloud, and its three sides are parallel to the x, y, and z axes, respectively.
[0142] Next, the processor 23 estimates the initial faces that will become the candidate faces based on the external segments to be modeled, which represent the external components (ST602). Specifically, first, the plane equations for the planes represented by each segment are obtained using the RANSAC method. Next, the intersection points (intersection lines) where the bounding box surrounding the entire restored point cloud intersects with the planes represented by each segment are obtained. Then, the initial faces of the polygons are created by connecting the intersection points.
[0143] Next, the processor 23 divides the initial faces at the points where the initial faces intersect (intersection lines) for all combinations of initial faces to find candidate faces (face division process) (ST603). In the example shown in Figure 16, initial faces are generated that include the faces of external components such as the floor, ceiling, walls, beam-like steps, and columnar protrusions, and a large number of candidate faces are generated by dividing these initial faces at the points where they intersect. Note that some candidate faces include faces of external components, while others do not.
[0144] In this candidate surface generation process, initial surfaces are generated by extending the planes represented by the segments to the extent that they intersect with the 3D bounding box that surrounds the entire restored point cloud. Candidate surfaces are then generated by dividing these initial surfaces at the points where they intersect.
[0145] As shown in Figure 15(C), in the candidate surface index calculation process (ST502 in Figure 15(A)), the processor 23 first calculates a coverage score (point cloud distribution index value) for each candidate surface (ST701).
[0146] Specifically, as shown below, the coverage score Score(f) is calculated as the ratio of the area of the alpha shape formed by the point cloud on the candidate surface f to the area of the candidate surface f. The alpha shape is a polygonal demarcation of the area where points exist that are included in the segment (sub-point cloud) that forms the basis of the initial surface before the candidate surface is subdivided and generated (see Figure 17(A)).
[0147]
number
[0148] Here, the coverage score is an index that represents the degree of appropriateness of a candidate surface as a constituent surface of a 3D model, depending on whether or not a sufficient number of points included in the segment are present on the candidate surface. If a sufficient number of points included in the segment are present on the candidate surface and the candidate surface is appropriate as a constituent surface of the 3D model, the coverage score will be positive, and the value of the coverage score will increase according to the degree of appropriateness. On the other hand, if a sufficient number of points included in the segment are not present on the candidate surface and the candidate surface is inappropriate as a constituent surface of the 3D model, the coverage score will be negative, and the value of the coverage score will decrease according to the degree of inappropriateness.
[0149] Next, processor 23 calculates a distance penalty (distance index value) for each candidate face (ST702).
[0150] Specifically, as shown below, for each candidate face f, the point closest to each vertex of the candidate face is determined from among the points included in the segment that forms the basis of the initial face before the candidate face is subdivided and generated. The average distance between that vertex and the points included in the segment is then calculated as the distance penalty Penalty(f).
[0151]
number
[0152] In the example shown in Figure 17(B), the point closest to each vertex v1, v2, v3, v4 of the candidate face is determined from among the points in the segment that forms the basis of the initial face before the candidate face is subdivided and generated. The average of the distances d1, d2, d3, d4 between these vertices v1, v2, v3, v4 and the points in the segment is calculated as the distance penalty Penalty(f).
[0153] Here, the distance penalty is an indicator of the degree of unsuitability of a candidate face as a constituent surface of a 3D model. If a candidate face is far from the segment that forms the basis of the initial face before it is subdivided and generated, it is assumed that the candidate face is unsuitable as a constituent surface of the 3D model, and therefore the distance penalty value will be large.
[0154] Next, the processor 23 calculates the same-domain partitioning cost (domain partitioning index value) for any combination of two adjacent candidate faces (ST703). The same-domain partitioning cost is an index that represents the degree of inappropriateness in the partitioning of one face's domain when both candidate faces are adopted as constituent faces of the 3D model.
[0155] Specifically, as shown below, for each combination of two candidate faces f1 and f2, scores RScore(f1) and RScore(f2) are calculated, and based on these scores RScore(f1) and RScore(f2), the same region partitioning cost PairCost(f1,f2) is calculated. Furthermore, for each combination of a candidate face f(f1 or f2) and a coplanar candidate face f', a point cloud ratio r represents the proportion of points located on candidate faces f and f' among the points included in the initial segment (sub-point cloud) that forms the basis of the initial faces before the partitioning generation of candidate faces f and f'. f ,r f ' is calculated, and its point cloud ratio r f ,r f Based on this, the score RScore(f) is calculated.
[0156]
number
[0157] Here, in the candidate plane generation process (ST501 in FIG. 15(A)), as shown in FIG. 18, the initial planes are divided by one intersection line where two initial planes intersect, so that four candidate planes are generated in a manner in contact with the intersection line. Then, in the optimization process (ST503 in FIG. 15(A)), candidate planes to be adopted as constituent planes of the three-dimensional model are selected. At this time, there are a case where two candidate planes are adopted from the four candidate planes, and a case where none of the four candidate planes is adopted.
[0158] As shown in FIG. 19(A), when considering a combination of two candidate planes f1 and f2 that are in contact with an intersection line, if the original segment of the initial plane before the two candidate planes f1 and f2 are respectively divided and generated exists so as to straddle the two candidate planes f1 and f2, it is desirable that the two candidate planes f1 and f2 are collectively selected as constituent planes of the three-dimensional model. Therefore, in the present embodiment, in such a case, the same region division cost is set so as to give a reward to the combination of the two candidate planes f1 and f2.
[0159] On the other hand, unlike the example shown in FIG. 19(A), although the original segment of the initial plane exists so as to straddle two candidate planes, it is not desirable that another set of two candidate planes f1 and f2 is selected as the constituent planes of the three-dimensional model. Therefore, in the present embodiment, in such a case, the same region division cost is set so as to give a penalty to the combination of the two candidate planes f1 and f2.
[0160] Specifically, first, as shown in FIG. 19(B), the point cloud ratio r representing the ratio of points located on the candidate plane f among the points included in the segment (partial point group) that is the source of the initial plane before the candidate plane f is divided and generated f is calculated. This point cloud ratio r fAs shown in Figure 19(C), this is determined for each candidate surface f and its coplanar candidate surface f'. Next, the smaller of the point cloud ratios rf and rf' for each candidate surface f and f' is compared with a predetermined threshold, and a reward or penalty is given for the combination of two candidate surfaces f and f'.
[0161] Furthermore, when calculating the cost of partitioning the same region for a combination of two candidate surfaces f1 and f2, the process is divided based on whether or not the two candidate surfaces f1 and f2 are coplanar.
[0162] The example shown in Figure 20(A) is when the two candidate surfaces f1 and f2 are coplanar. In this case, the point cloud ratio rf (=0.4) of candidate surface f (=f1) is used to determine the point cloud ratio r of candidate surface f' (=f2). f '(=0.2) is small, and this r f Since the value exceeds the threshold (0.05), a reward is given. Note that the threshold is not limited to 0.05.
[0163] On the other hand, the example shown in Figure 20(B-1) is the case where the two candidate faces f1 and f2 are not coplanar. In this case, as shown in Figures 20(B-2) and (B-3), we assume two candidate faces f' that are coplanar to each of the two candidate faces f1 and f2, and consider whether there is a penalty for these two possible combinations of candidate faces f(f1,f2),f'.
[0164] In the example shown in Figure 20(B-2), the point cloud ratio r of candidate surface f' which is coplanar with candidate surface f (=f2) is shown. f From (=0.4), the point cloud ratio r of candidate surface f is f (=0.2) is small, and this r f Since (=0.2) is greater than or equal to the threshold (0.05), a penalty is imposed. On the other hand, in the example shown in Figure 20 (B-3), the point cloud ratio r of the candidate surface f (=f1) f (=0.2) The point cloud ratio r of candidate surface f' that is coplanar with candidate surface f is f '(=0.01) is small, and this r f Since '(=0.01) is below the threshold (0.05), no penalty is imposed.
[0165] As shown in Figure 15(D), in the optimization process (ST503 in Figure 15(A)), the processor 23 first sets the decision variable (ST801). Specifically, the decision variable x is set to 1 when each candidate face is adopted and 0 when it is not adopted, as shown below. f This will be set.
[0166]
number
[0167] Next, processor 23 sets the objective function (ST802). Specifically, as shown below, the objective function E is set using the index values for each candidate surface obtained in the candidate surface index calculation process, namely the coverage score (f), distance penalty (f), and identical region partitioning cost (f1, f2).
[0168]
number
[0169] Next, processor 23 sets the constraints for the integer programming problem (ST803). The constraints are set so that the 3D model is watertight and generated as a manifold. Specifically, the constraints are set so that the candidate faces f adjacent to each edge are either 0 or 2, as shown below.
[0170]
number
[0171] Next, processor 23 runs a general-purpose optimization solver for solving 0-1 integer programming problems based on the objective function and constraints to select appropriate candidate faces as constituent faces of the 3D model (ST804). At this time, a solution vector is obtained that minimizes the objective function based on the index values (coverage score, distance penalty, same-domain partitioning cost) for each candidate face while satisfying the constraints.
[0172] Next, the processor 23 creates a three-dimensional model of the room (polygon mesh model) (see Figure 4) based on the candidate faces selected as constituent faces of the three-dimensional model (ST805).
[0173] Next, we will explain the basic screen 31 (initial screen) displayed on user terminal 1. Figure 21 is an explanatory diagram showing the basic screen 31.
[0174] The basic screen 31 is provided with a 3D viewer section 32 and an operation panel section 33.
[0175] The 3D viewer unit 32 is provided with a mode display selection unit 34 and an image display unit 35. The mode display selection unit 34 is provided with a "point cloud" tab 37, a "segment" tab 38, and a "model" tab 39. Tabs 37, 38, and 39 are switched according to the progress of the 3D model generation process (point cloud input process, segmentation process, and modeling process), and the image displayed in the image display unit 35 (reconstructed point cloud, segment, and 3D model) changes accordingly. The user can also switch to the respective mode screen by operating tabs 37, 38, and 39.
[0176] The control panel 33 is equipped with a "Modeling" button 41 and a "Settings" button 42. Figure 21 shows the state when the "Modeling" button 41 is selected. When the user operates the "Settings" button 42, the system transitions to the settings screen 91 (see Figure 27).
[0177] Furthermore, the control panel 33 is equipped with a "File Import" button 43, a "Segmentation Execution" button 44, an automatic internal / external determination instruction unit 45, an internal / external sorting operation unit 46, a "Modeling Execution" button 47, and a "File Export" button 48.
[0178] When the user operates the "File Import" button 43, the restored point cloud file stored in the storage unit 22 is input, and the system transitions to the restored point cloud display screen 61 (see Figure 22). The restored point cloud file is generated by the 3D restoration process and stored in the storage unit 22. Alternatively, when the "File Import" button 43 is operated, a screen (not shown) may be displayed in which the user can select one of several restored point cloud files stored in the storage unit 22.
[0179] Next, we will explain the restored point cloud display screen 61 that is displayed on the user terminal 1. Figure 22 is an explanatory diagram showing the restored point cloud display screen 61.
[0180] On the restored point cloud display screen 61, the restored point cloud stored in the input file is displayed on the image display unit 35. The restored point cloud is displayed as a two-dimensional image based on the specified viewpoint on the image display unit 35. In the example shown in Figure 22, the viewpoint is set to the outside of the room, and the walls and ceiling on the side closer to the viewpoint are made transparent so that the inside of the room can be seen. However, the walls and ceiling on the side closer to the viewpoint can also be made opaque.
[0181] Furthermore, the image display unit 35 allows the user to perform three-dimensional operations on the restored point cloud. That is, the user can observe the entire restored point cloud in detail by changing the position and angle of the viewpoint relative to the restored point cloud (for example, by dragging and dropping).
[0182] On the restored point cloud display screen 61, when the user operates the "Execute Segmentation" button 44, preprocessing and segmentation processing begin. Once preprocessing and segmentation processing are complete, the screen transitions to the segment display screen 71 (see Figure 23).
[0183] Next, we will explain the segment display screen 71 that is displayed on the user terminal 1. Figure 23 is an explanatory diagram showing the segment display screen 71. Figure 24 is an explanatory diagram showing the state in the segment display screen 71 where the results of the automatic inside / outside determination process are reflected.
[0184] As shown in Figure 23, the segment display screen 71 displays the restored point cloud, that is, the restored point cloud in the segment division state, on the image display unit 35, which reflects the extraction results from the segmentation process. The image display unit 35 displays the restored point cloud in the segment division state as a two-dimensional image based on the specified viewpoint. In the example shown in Figure 23, the viewpoint is set to the outside of the room, and the interior of the room is visible by making the wall and ceiling surfaces on the side closer to the viewpoint transparent. However, the wall and ceiling surfaces on the side closer to the viewpoint can also be made opaque.
[0185] Furthermore, in the reconstructed point cloud in segmented state, each segment is displayed in a different color. That is, each point in the reconstructed point cloud is displayed colored according to the color of the segment. Specifically, each segment is displayed in a different chromatic color (a color other than gray). On the other hand, point clouds that were not extracted as segments, such as the legs of a desk or chair, are displayed in achromatic colors (gray). This allows the user to easily check the generation status of segments that form the basis of the constituent surfaces of the 3D model.
[0186] Furthermore, the image display unit 35 allows the user to perform three-dimensional operations on the reconstructed point cloud in the segmented state. That is, the user can observe the entire reconstructed point cloud in the segmented state in detail by changing the position and angle of the viewpoint relative to the reconstructed point cloud in the segmented state (for example, by dragging and dropping).
[0187] The automatic inside / outside determination instruction unit 45 is equipped with a toggle switch 51 and a slider 52.
[0188] When the user turns on the toggle switch 51, the automatic inside / outside determination process is executed. Once the automatic inside / outside determination process is complete, the restored point cloud of the segment division state, reflecting the results of the automatic inside / outside determination process, is displayed on the image display unit 35, as shown in Figure 24.
[0189] In the restored point cloud reflecting the results of the automatic internal / external determination process, external segments representing external components and internal segments representing internal components are displayed with different brightness levels. Specifically, external segments, i.e., segments targeted for modeling, are highlighted using high-brightness chromatic colors. Internal segments, i.e., segments not targeted for modeling, are suppressed using low-brightness chromatic colors. This allows users to easily check the generation status of segments representing external components that form the basis of the 3D model's surfaces.
[0190] The user can change the segment size ratio used in the automatic inside / outside determination process by operating the slider 52. In the bounding box determination process of the automatic inside / outside determination process, it is determined whether a segment is an external segment or not based on the ratio of the length of the longest side of the bounding box set for the segment to the maximum inter-face distance, and the segment size ratio is used as the threshold for this determination. When the user operates the slider 52 and changes the segment size ratio, the automatic inside / outside determination process is re-executed, and the color coding state of the restored point cloud of the segment division state displayed on the image display unit 35 changes according to the determination result.
[0191] The internal / external sorting operation unit 46 can perform internal / external sorting operations to specify whether the segments displayed on the image display unit 35 belong to internal or external components. When a user selects a segment on the restored point cloud of the segment division state displayed on the image display unit 35 and then operates the internal / external sorting operation unit 46, a manual internal / external setting process is executed for the selected segment. In the manual internal / external setting process, the selected segment is sorted into segments that represent external components of a room and are subject to modeling, and segments that represent internal components of a room and are not subject to modeling, according to the user's internal / external sorting operation.
[0192] In the image display unit 35, the user can visually determine whether there are errors in the settings of the segments to be modeled, which represent the external components of a room, and the segments not to be modeled, which represent the internal components of a room, in the restored point cloud of the segment division state that reflects the results of the automatic internal / external determination process. The user can then select the segments with errors by operating the input device 14 (for example, by clicking the mouse). At this time, the image display unit 35 highlights the selected segments. Specifically, the selected segments are highlighted by blinking (change in brightness).
[0193] The internal / external distribution operation unit 46 is equipped with an "Undo" button 55, a "Redo" button 56, an "Add" button 57, and a "Delete" button 58. When a user operates any of the "Undo" button 55, "Redo" button 56, "Add" button 57, or "Delete" button 58, the manual internal / external setting process corresponding to that operation is executed.
[0194] Specifically, if a user selects a segment that is not set to be modeled (an internal segment) and then operates the "Add" button 57, the selected segment will be added to the modeling target and changed to an external segment. Also, if a user selects a segment that is set to be modeled (an external segment) and then operates the "Delete" button 58, the selected segment will be removed from the modeling target and changed to an internal segment. Furthermore, if a user operates the "Undo" button 55, they can revert to the settings before the operation of the "Add" button 57 or the "Delete" button 58. Also, if a user operates the "Redo" button 56, they can revert to the settings before the operation of the "Undo" button 55.
[0195] Here, when multiple segments are selected by the user through user operation on the image display unit 35, if the user operates the internal / external distribution operation unit 46, the settings of the selected multiple segments are changed all at once in the manual internal / external setting process. At this time, for example, multiple segments can be selected sequentially by operating the input device 14 (e.g., keyboard and mouse shift-click operations). Also, by inputting a bounding box using the input device 14 (e.g., drag operation), multiple segments contained within that bounding box can be selected simultaneously.
[0196] Furthermore, by turning on the toggle switch 51 of the automatic indoor / outdoor determination instruction unit 45, the user can operate the indoor / outdoor distribution operation unit 46 after the automatic indoor / outdoor determination process has been executed. In this case, manual indoor / outdoor setting processing is performed on the segment in which the results of the automatic indoor / outdoor determination process have been reflected, in accordance with the user's indoor / outdoor distribution operation. This allows the user to manually correct any errors in the results of the automatic indoor / outdoor determination process.
[0197] On the other hand, the user can also operate the interior / exterior sorting operation unit 46 while the toggle switch 51 of the automatic interior / exterior determination instruction unit 45 remains in the off state. In this case, manual interior / exterior setting processing is performed on the initial segments according to the user's interior / exterior sorting operation. In the initial state, all segments are set as external segments, which are the modeling targets, i.e., external segments representing the external components of a room. The user can exclude segments that are internal components from the modeling targets by selecting segments that are set as external segments despite being internal components, and then operating the "delete" button 58.
[0198] In this embodiment, the user can visually identify segments to be modeled and segments to be excluded from modeling and manually change the settings. This improves the accuracy of the generated 3D model (room model). Furthermore, in this embodiment, the user can set segments to be modeled and segments to exclude on a segment-by-segment basis. Therefore, compared to, for example, a case where the user specifies a range using a bounding box for the restored point cloud and sets segments to be modeled and segments to exclude, the working time can be significantly reduced.
[0199] On the segment display screen 71 (see Figures 23 and 24), when the user operates the "Execute Modeling" button 47, the 3D modeling process begins. Once the 3D modeling process is complete, the screen transitions to the 3D model display screen 81 (see Figures 25 and 26).
[0200] Furthermore, if the user operates the "Execute Modeling" button 47 without operating the toggle switch 51 of the automatic interior / exterior determination instruction unit 45 or the interior / exterior sorting operation unit 46, the interior / exterior sorting process (automatic interior / exterior determination process and manual interior / exterior setting process) is not executed, and the 3D modeling process starts. In this case, since the interior / exterior sorting process is omitted, a 3D model is generated that includes both the exterior and interior components of the room.
[0201] Next, we will describe the 3D model display screen 81 displayed on the user terminal 1. Figure 25 is an explanatory diagram showing the 3D model display screen 81 with the interpolation region displayed. Figure 26 is an explanatory diagram showing the 3D model display screen 81 with the error region displayed.
[0202] On the 3D model display screen 81, the 3D model (room model) generated by the 3D modeling process is displayed on the image display unit 35. The image display unit 35 displays the 3D model as a 2D image based on the specified viewpoint. In the example shown in Figures 25 and 26, the viewpoint is set to the outside of the room, and the walls and ceiling on the side closer to the viewpoint are made transparent, allowing the interior of the room to be seen. However, the walls and ceiling on the side closer to the viewpoint can also be made opaque.
[0203] Furthermore, the image display unit 35 allows the user to perform three-dimensional operations on the three-dimensional model. That is, the user can observe the entire three-dimensional model in detail by changing the position and angle of the viewpoint relative to the three-dimensional model (for example, by dragging and dropping).
[0204] As shown in Figure 25, on the 3D model display screen 81, images 82, 83, and 84 of the interpolated regions are displayed on the 3D model in the image display unit 35. Specifically, the images 82, 83, and 84 of the interpolated regions are highlighted with a color that is more conspicuous than other parts (a highly eye-catching color). In addition, the images 82, 83, and 84 of the interpolated regions may be drawn in different ways (color, pattern, etc.) depending on the reason for interpolation. In the example shown in Figure 25, image 82 of the interpolated region relating to point cloud loss due to concealment by furniture (sofas and shelves), image 83 of the interpolated region relating to point cloud-less areas due to openings (doorways), and image 84 of the interpolated region relating to point cloud loss due to missed shooting are displayed. This allows the user to easily check the interpolation status.
[0205] Furthermore, when the user selects images 82, 83, or 84 of the interpolation region in the image display unit 35 (for example, by hovering the mouse over them), the interpolation reason for the selected interpolation region is displayed. Specifically, an interpolation reason display unit 85 (tooltip) is displayed with text indicating the interpolation reason, such as "Interpolate openings such as doors," "Interpolate defects due to occlusion," or "Interpolate defects due to missed image capture." This allows the user to easily confirm the interpolation reason. For example, the user can confirm that point cloud defects due to missed image capture have been appropriately interpolated. Also, the user can confirm that openings such as doorways have been incorrectly interpolated even though they are not point cloud defects.
[0206] As shown in Figure 26, on the 3D model display screen 81, the image display unit 35 displays an image 86 of the error region on the 3D model. Specifically, the image 86 of the error region is highlighted with a color that is more conspicuous (a highly visible color), similar to the images 82, 83, and 84 of the interpolation region. In the example shown in Figure 26, the columnar protrusion at the boundary between the two walls is not reproduced in the 3D model, and this area is highlighted as an error region. This allows the user to easily check the error status of the 3D model relative to the restored point cloud.
[0207] When the user operates the "Settings" button 42 on the 3D model display screen 81, the user is redirected to the settings screen 91 (see Figure 27).
[0208] Furthermore, when the user operates the "File Export" button 48 on the 3D model display screen 81, the generated 3D model (room model) is output in a predetermined file format. The user may choose the file format as appropriate.
[0209] In this embodiment, a 3D model (room model) representing the external components of the room is generated. However, the output file may also contain, in addition to the 3D model, segments representing the external components of the room and segments representing the internal components of the room, in an identifiable manner. This allows the segments representing the internal components to be used later for processes such as object recognition or 3D model replacement.
[0210] Next, we will explain the settings screen 91 displayed on user terminal 1. Figure 27 is an explanatory diagram showing the settings screen 91.
[0211] The settings screen 91 allows the user to configure how the 3D model (room model) is displayed on the 3D model display screen 81 (see Figures 25 and 26). The settings screen 91 includes an interpolation region display setting unit 92 and an error region display setting unit 93.
[0212] The interpolation area display setting unit 92 is provided with checkboxes 95 and display color boxes 96 for each interpolation reason item (opening, missed shot, concealment). By operating the checkboxes 95, the user can switch the display of the interpolation area on or off for each interpolation reason item. In addition, by operating the display color box 96, a color selection box (not shown) will be displayed, and by selecting a color in that color selection box, the user can specify the display color of the interpolation area for each interpolation reason item.
[0213] The error area display setting unit 93 is provided with an input box 97 and a display color box 98. In the input box 97, the user can input a threshold value as a detection condition in the error area detection process. Areas where the error is greater than the input threshold value are displayed as an error area image 86 (see Figure 26). In addition, by operating the display color box 98, a color selection box (not shown) is displayed, and the user can specify the display color of the error area by selecting a color in that color selection box.
[0214] As described above, embodiments have been explained as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited to these embodiments and can be applied to embodiments that have been modified, replaced, added, or omitted. Furthermore, it is possible to create new embodiments by combining the components described in the above embodiments. [Industrial applicability]
[0215] The 3D model generation apparatus, 3D model generation method, and 3D model generation program according to the present invention have the effect of generating a 3D model in which the components of the target space are appropriately reproduced, even when the components of the target space are composed of complex surfaces and when there are defects in the restored point cloud of the target space. They are useful as a 3D model generation apparatus, 3D model generation method, and 3D model generation program, etc., which use a processor to execute a process to generate a 3D model of the components of the target space based on the restored point cloud obtained by a 3D restoration process of the target space in a building. [Explanation of Symbols]
[0216] 1: User terminal (display device) 2: Server (3D model generation device) 23: Processor 31: Basic screen 32: 3D Viewer Section 33: Control Panel 34: Mode display selection section 35: Image display section 43, 44: Buttons 45:Automatic inside / outside judgment instruction section 46: Indoor / Outdoor Distribution Operation Unit 47, 48: Buttons 51: Toggle switch (operating part) 52: Slider (operating part) 55, 56, 57, 58: Buttons 61: Restored Point Cloud Display Screen 71: Segment display screen 81: 3D Model Display Screen 82, 83, 84: Interpolation region 85: Interpolation Reason Display Section 86:Error area 91: Settings screen 92: Interpolation Area Display Setting Section 93:Error area display setting section 95: Checkbox 96: Display Color Box 97: Input Box 98: Display Color Box
Claims
1. A 3D model generation device that uses a processor to generate a 3D model of the components of a target space based on a restored point cloud obtained by a 3D restoration process of the target space within a building, The aforementioned processor, A segmentation process is performed to extract multiple segments representing flat surfaces from the aforementioned restored point cloud. An automatic segment determination process is performed to determine whether the segment extracted in the above segmentation process is a modeling target representing an external component or an unmodeling target representing an internal component. A three-dimensional model generation apparatus characterized by performing a modeling process to generate a three-dimensional model representing an external component based on the segments determined to be modeling targets in the automatic segment determination process.
2. The aforementioned processor, In the aforementioned modeling process, The 3D model generation apparatus according to claim 1, characterized in that it generates candidate surfaces that are candidates to be constituent surfaces of the 3D model based on the segment to be modeled, calculates an index value for the appropriateness of each candidate surface as a constituent surface of the 3D model based on the distribution of point clouds on the candidate surface, selects an appropriate candidate surface as a constituent surface of the 3D model based on the index value, and generates the 3D model relating to the external components to be modeled based on the selected candidate surface.
3. The aforementioned processor, Before performing the aforementioned segmentation process, as a preprocessing step, The 3D model generation apparatus according to claim 1, characterized in that it performs noise reduction processing, downsampling processing, and coordinate axis setting processing on the restored point cloud.
4. The aforementioned processor, In the aforementioned segmentation application process, A point cloud segmentation process is performed to divide the restored point cloud into multiple segments using a region expansion method that expands the region by sequentially adding neighboring points, The 3D model generation apparatus according to claim 1, characterized in that it performs a linear segment detection process to detect linear segments that are not to be modeled from among the segments obtained by the point cloud division process.
5. The aforementioned processor, In the automatic segment determination process, Segments that are close to each other are grouped together into one group. The segment included in the group with the highest score is determined to be the modeling target representing an external component. The 3D model generation apparatus according to claim 1, characterized in that segments not included in the group with the highest score are determined to be excluded from modeling, representing internal components that form floating surfaces not in contact with external components.
6. The aforementioned processor, In the automatic segment determination process, Segments whose normal direction corresponds to the direction of each coordinate axis in a three-dimensional Cartesian coordinate system are grouped by coordinate axis. The maximum distance between segments in each group of coordinate axes is determined as the maximum inter-plane distance for each coordinate axis. For each segment, a 3D bounding box is set with edges parallel to each coordinate axis. The 3D model generation apparatus according to claim 1, characterized in that, with respect to each segment, if the ratio of the length of the longest side to the maximum inter-face distance with respect to the coordinate axis parallel to the longest side of the 3D bounding box exceeds a predetermined threshold, it is determined that the segment is a modeling target representing an external component.
7. The aforementioned processor, In the aforementioned modeling process, The three-dimensional model generation apparatus according to claim 2, characterized in that, as the index value, a point cloud distribution index value representing the proportion of the range in which point clouds are distributed on the candidate surface is calculated, and an optimization process is performed to select the candidate surface that is appropriate as a constituent surface of the three-dimensional model based on the point cloud distribution index value.
8. The aforementioned processor, In the aforementioned modeling process, The 3D model generation apparatus according to claim 2, characterized in that, as the index value, it calculates a distance index value representing the degree of inappropriateness regarding the distance of the candidate surface from its original segment, and performs an optimization process to select a candidate surface that is appropriate as a constituent surface of the 3D model based on the distance index value.
9. The aforementioned processor, The three-dimensional model generation apparatus according to claim 2, characterized in that, as the index value, it calculates a region division index value that represents the degree of inappropriateness regarding the combination of two candidate surfaces that cause the division of the segment that will become the candidate surface, and performs an optimization process to select a candidate surface that is appropriate as a constituent surface of the three-dimensional model based on the region division index value.
10. The aforementioned processor, The 3D model generation apparatus according to claim 1, characterized in that it presents the determination result of the automatic segment determination process to the user, and executes a manual segment setting process to change the determination result of the automatic segment determination process in response to the user's operation of selecting a segment and specifying whether or not it is a target for modeling.
11. A 3D model generation method in which a processor performs a process to generate a 3D model of the components of a target space based on a restored point cloud obtained by a 3D restoration process of the target space within a building, A segmentation process is performed to extract multiple segments representing flat surfaces from the aforementioned restored point cloud. An automatic segment determination process is performed to determine whether the segment extracted in the above segmentation process is a modeling target representing an external component or an unmodeling target representing an internal component. A method for generating a three-dimensional model, characterized by executing a modeling process to generate the three-dimensional model representing an external component based on the segments determined to be modeling targets in the automatic segment determination process.
12. A 3D model generation program that uses a processor to generate a 3D model of the components of a target space based on a restored point cloud obtained by a 3D reconstruction process of the target space within a building, A segmentation process is performed to extract multiple segments representing flat surfaces from the aforementioned restored point cloud. An automatic segment determination process is performed to determine whether the segment extracted in the above segmentation process is a modeling target representing an external component or an unmodeling target representing an internal component. A three-dimensional model generation program characterized by executing a modeling process to generate a three-dimensional model representing an external component based on the segments determined to be modeling targets in the automatic segment determination process.
Citation Information
Patent Citations
Three-dimensional model generation device
JP2019168976A
