Point cloud combining device and point cloud combining method

Abstract

To reduce a burden on a user when there are many pieces of point group data to be coupled by improving operability of the user in alignment when coupling two pieces of point group data acquired by three dimensional restoration processing of a target place.SOLUTION: Displaying, on a display, a canvas screen 61 including a canvas 72 on which two point cloud images 71 obtained by visualizing two pieces of point cloud data to be combined are arranged so as to be operable by a user and a button 81 for instructing execution of point cloud combination processing, and executing the point cloud combination processing for combining the two pieces of point cloud data in response to a positioning operation by the user for superimposing common regions of the two point cloud images on the canvas and an operation of the button by the user; A screen including a combined point cloud image obtained by visualizing the point cloud data generated by the point cloud combining process is displayed.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The present invention relates to a point cloud merging device and a point cloud merging method for merging two pieces of point cloud data acquired by three-dimensional reconstruction processing of a target location after mutual alignment. [Background technology]

[0002] There is a known 3D restoration technology that generates point cloud data as 3D spatial information about a target location based on photographed images of the target location. Among these 3D restoration technologies, the SLAM (Simultaneous Localization And Mapping) method has been attracting attention in recent years. With the SLAM method, point cloud data of a target location can be obtained by having a mobile object (e.g., a worker) hold a photographing device and photograph the target location while moving around the location.

[0003] On the other hand, when the target location is large, such as a factory, or when the target location is divided into complex sections, such as a house, it is not possible to capture the entire target location at once. In such cases, the user must capture the target location multiple times. Furthermore, a process is performed in which multiple point cloud data generated by capturing partial images of the target location are combined to generate combined point cloud data representing the entire target location. In this case, since the positional relationships between the multiple point cloud data cannot be accurately recognized, combining the multiple point cloud data requires alignment to align common areas where the same subject is captured in the multiple point cloud data.

[0004] A known technique for aligning multiple point cloud data is to display a point cloud image that visualizes the point cloud data on a screen, and allow the user to manipulate the point cloud image so that the common areas of the two point cloud screens overlap (see Patent Document 1). In this technique, by placing multiple markers (targets) that serve as references for alignment at target locations, the markers are displayed on the point cloud image, and the user can perform alignment operations based on the markers. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2018-147065 Summary of the Invention [Problem to be solved by the invention]

[0006] However, with conventional technology, it is necessary to place multiple markers (targets) that serve as references for alignment at the target location and measure the installation positions of the markers using surveying equipment (such as a total station), which is extremely time-consuming. Therefore, there is a need for technology that allows users to perform alignment operations without using markers as references.

[0007] Furthermore, when the target location is large, such as a factory, photography may be taken multiple times (for example, 10 times), resulting in a large number of point cloud data to be combined (for example, 10 pieces), which poses a problem of the work of combining the point cloud data being extremely time-consuming. For this reason, a technology that can reduce the burden on users when there is a large number of point cloud data to be combined is desired.

[0008] Therefore, the main object of the present invention is to provide a point cloud combining device and a point cloud combining method that can improve user operability when aligning multiple point cloud data obtained by 3D reconstruction processing of a target location, thereby reducing the burden on the user when there is a large number of point cloud data to be combined. [Means for solving the problem]

[0009] The point cloud combining device of the present invention is a point cloud combining device that uses a processor to perform a process of combining two point cloud data obtained by a 3D reconstruction process of a target location after aligning them with each other, and the processor displays on a display device a screen including a canvas on which multiple point cloud images, which visualize the multiple point cloud data to be combined, are arranged so that they can be operated by a user, and an operation unit that instructs the execution of the point cloud combining process, and executes the point cloud combining process to combine the two point cloud data in accordance with a user's alignment operation to overlap the common areas of the two point cloud images on the canvas and a user's operation on the operation unit, and displays on the display device a screen including a combined point cloud image that visualizes the combined point cloud data generated by the point cloud combining process.

[0010] Furthermore, the point cloud combining method of the present invention is a point cloud combining method that causes a processor to perform a process of combining two point cloud data acquired by a 3D reconstruction process of a target location after aligning them with each other, and is configured to display a screen on a display device that includes a canvas on which multiple point cloud images, which visualize the multiple point cloud data to be combined, are arranged so that they can be operated by a user, and an operation unit that instructs the execution of the point cloud combining process, and to execute the point cloud combining process to combine the two point cloud data in accordance with a user's operation of aligning the common areas of the two point cloud images on the canvas and a user's operation on the operation unit, and to display a screen on the display device that includes a combined point cloud image that visualizes the combined point cloud data generated by the point cloud combining process. [Effects of the Invention]

[0011] According to the present invention, multiple point cloud images that visualize multiple point cloud data to be combined are arranged on a canvas, and the user can appropriately select and align the point cloud images on the canvas. This improves user operability and reduces the burden on the user when there is a large amount of point cloud data to be combined. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is an explanatory diagram showing a situation of a photographing operation performed by a user using the photographing system according to the first embodiment; [Figure 2] FIG. 10 is an explanatory diagram illustrating an example of a point cloud combining process performed by the control device. [Figure 3] A block diagram showing the schematic configuration of an imaging device and a control device. [Figure 4] A block diagram showing an overview of the processing performed by the processor of the control device. [Figure 5] A flowchart showing the procedure of processing performed by the control device. [Figure 6] Flow diagram showing the procedure of the point cloud alignment process performed by the control device [Figure 7] FIG. 10 is a flowchart showing the procedure of the preliminary alignment process performed by the control device. [Figure 8] FIG. 10 is an explanatory diagram showing a menu screen displayed on the display of the control device. [Figure 9] FIG. 10 is an explanatory diagram showing a canvas screen displayed on the display of the control device. [Figure 10] An explanatory diagram showing the operation status of a point cloud image and a combined point cloud image on the canvas. [Figure 11] An explanatory diagram showing the operation status of a point cloud image and a combined point cloud image on the canvas. [Figure 12] An explanatory diagram showing the operation status of a point cloud image and a combined point cloud image on the canvas. [Figure 13] An explanatory diagram showing a situation where the opacity of a point cloud image is adjusted on the canvas screen. [Figure 14] An explanatory diagram showing how the canvas is enlarged or reduced on the canvas screen. [Figure 15] An explanatory diagram showing the operation status of undoing merging on the canvas screen [Figure 16] An explanatory diagram showing a situation in which a frame image is displayed around a point cloud image on the canvas screen. [Figure 17] FIG. 10 is an explanatory diagram showing a situation in which a point cloud image is set to a locked state on the canvas screen. [Figure 18] FIG. 10 is an explanatory diagram showing a connection history screen displayed on the display of the control device. [Figure 19] FIG. 10 is an explanatory diagram showing a main canvas screen and a sub-canvas screen displayed on a display of a control device according to a modified example of the first embodiment; [Figure 20] FIG. 10 is an explanatory diagram showing a canvas screen displayed on a display of a control device according to a second embodiment; DETAILED DESCRIPTION OF THE INVENTION

[0013] The first invention made to solve the above problem is a point cloud merging device that uses a processor to execute a process of merging two point cloud data acquired by a 3D reconstruction process of a target location after aligning them with each other, wherein the processor displays on a display device a screen including a canvas on which multiple point cloud images that visualize the multiple point cloud data to be merged are arranged so that the user can operate, and an operation unit that instructs the execution of the point cloud merging process, and executes the point cloud merging process to merge the two point cloud data in accordance with the user's alignment operation to overlap the common areas of the two point cloud images on the canvas and the user's operation on the operation unit, and displays on the display device a screen including a combined point cloud image that visualizes the combined point cloud data generated by the point cloud merging process.

[0014] According to this, multiple point cloud images that visualize multiple point cloud data to be merged are arranged on a canvas, and the user can appropriately select the point cloud image on the canvas and perform the alignment operation. This improves user operability and reduces the burden on the user when there is a large amount of point cloud data to be merged. Note that the point cloud merging process can be performed not only to merge two source point cloud data, but also to merge source point cloud data with merged point cloud data, or to merge two merged point cloud data.

[0015] In addition, in a second invention, the processor is configured to display on the canvas the combined point cloud image, which visualizes the combined point cloud data generated by combining two pieces of point cloud data, and combine the combined point cloud data and the point cloud data in response to a user operation to superimpose the common areas of the combined point cloud image and the point cloud image.

[0016] According to this, three or more point cloud data can be sequentially combined two by two to obtain point cloud data in which three or more point cloud data are combined. Note that in addition to combining two point cloud data or combining combined point cloud data with other point cloud data, it is also possible to combine two combined point cloud data.

[0017] In addition, in a third invention, the processor displays a screen including an operation unit that instructs restoration of the state before combination, and in response to a user's operation of the operation unit, returns the combined point cloud data to the state before combination and displays the point cloud image corresponding to the point cloud data before combination.

[0018] This allows the user to restore the point cloud data to its pre-combination state if the user makes an error in operation or if an undesirable combination result is obtained. The operation unit may be a button provided on the screen.

[0019] In addition, a fourth invention is configured such that the processor displays a screen including an operation unit for specifying the opacity of the point cloud image, and displays the point cloud image in a semi-transparent state with the specified opacity in response to a user's operation of the operation unit.

[0020] By displaying the point cloud image in a semi-transparent state, the user can easily check the alignment state of the common area when the point cloud images are superimposed. Note that the operation unit may be a slider provided on the screen.

[0021] In addition, a fifth invention is configured such that the processor displays a screen including an operation unit for instructing the enlargement or reduction of the canvas, and enlarges or reduces the canvas in response to a user's operation of the operation unit.

[0022] This allows the user to easily check the alignment of the common area when the point cloud images are superimposed by enlarging or reducing the point cloud image on the canvas. The operation unit may be a button or a slider provided on the screen.

[0023] In addition, a sixth invention is configured such that the processor enlarges or reduces the canvas by changing the distance from the viewpoint to the canvas while keeping the position and size of the point cloud image relative to the canvas constant.

[0024] This allows the point cloud image on the canvas to be scaled by scaling the canvas. Furthermore, when a large number of point cloud images are arranged on the canvas, display processing for changing the positions and angles of those point cloud images in response to user operations becomes easier. In this case, when a user performs an operation to select multiple point cloud images, the canvas may be scaled around the midpoint of the selected multiple point cloud images.

[0025] In addition, the seventh invention is configured such that the processor displays a screen including an operation unit that instructs the display of a frame image representing the outer periphery of the point cloud image, and switches between a display state in which the frame image is displayed and a non-display state in which the frame image is not displayed in response to a user's operation of the operation unit.

[0026] This allows the user to easily check the degree of overlap of common areas of the point cloud data using the frame image as a guide. The operation unit may be a button provided on the screen. The point cloud image on which the frame image is displayed may be selected in advance in response to a user operation. The point cloud image on which the frame image is displayed may be one or multiple.

[0027] In addition, an eighth invention is configured such that the processor displays a screen including an operation unit for instructing locking of the point cloud image, and sets the point cloud image to a locked state in response to operation of the operation unit by a user.

[0028] This allows the point cloud image to be locked, i.e., fixed and inoperable, preventing the user from accidentally moving the point cloud image. The operation unit may be a button provided on the screen. The point cloud image to be set to the locked state may be selected in advance in response to a user operation. The number of point cloud images to be set to the locked state at the same time may be one or more.

[0029] In addition, a ninth invention is configured such that the processor displays a screen including an operation unit that instructs the display of combination history information representing the history of combining two of the point cloud data, and displays a screen including the combination history information on the display device in response to a user's operation of the operation unit.

[0030] This allows the user to easily check the history of combining multiple point cloud data, i.e., the combinations in which the point cloud data have been combined up to now. The operation unit may be buttons provided on the screen.

[0031] In addition, in a tenth aspect of the present invention, the processor is configured to display a screen including the combination history information and evaluation information representing the degree of appropriateness of alignment using a transformation matrix estimated at the time of combination in the combined point group data generated by the point group combination processing.

[0032] This allows the user to easily check the degree of appropriateness of bonding in the bonding point group data.

[0033] In addition, an eleventh invention is a point cloud merging method that causes a processor to perform a process of merging two point cloud data obtained by a 3D reconstruction process of a target location after aligning them with each other, wherein a screen including a canvas on which multiple point cloud images visualizing the multiple point cloud data to be merged are arranged so that the user can operate, and an operation unit for instructing the execution of the point cloud merging process is displayed on a display device, and the point cloud merging process is executed to merge the two point cloud data in accordance with a user's operation of aligning the common areas of the two point cloud images on the canvas and a user's operation on the operation unit, and a screen including a combined point cloud image visualizing the combined point cloud data generated by the point cloud merging process is displayed on the display device.

[0034] As with the first invention, this improves user operability when aligning multiple point cloud data obtained by 3D reconstruction processing of a target location, thereby reducing the burden on the user when there is a large number of point cloud data to be combined.

[0035] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0036] (First embodiment) FIG. 1 is an explanatory diagram showing a situation in which a user performs a photographing operation using the photographing system according to the first embodiment.

[0037] As shown in FIG. 1, the photographing system includes a photographing device 1 and a control device 2 (point cloud combining device).

[0038] The photographing device 1 includes a sensor unit 11, a display / input panel unit 12, and a rod-shaped support member 13. The sensor unit 11 and the display / input panel unit 12 of the photographing device 1 are connected via a first cable 18. The display / input panel unit 12 of the photographing device 1 and the control device 2 are connected via a second cable 19.

[0039] The control device 2 is configured as a laptop or tablet PC that can be carried by a user (worker). In the example shown in Fig. 1, the control device 2 can be stored in a shoulder bag and carried by the user, but the manner in which the user carries the control device 2 is not limited to this.

[0040] A user performs a photographing task by walking around a target location while holding the photographing device 1 in his / her hand and causing the photographing device 1 to photograph the target location. At this time, the user can change the position (height) of the sensor unit 11 by moving the arm holding the photographing device 1.

[0041] Next, a description will be given of the point cloud combining process performed by the control device 2. Fig. 2 is an explanatory diagram showing an example of the point cloud combining process.

[0042] When the target location is large, such as a factory, or when the target location is divided into complex sections, such as a house, it is difficult to capture the entire target location at once. In this case, the user takes multiple photos of the target location. In addition, the control device 2 performs a process of combining multiple point cloud data generated by partially photographing the target location to generate combined point cloud data representing the entire target location.

[0043] In the example shown in FIG. 2, the target location is a house. The house is equipped with a living room, kitchen, bedroom, toilet, bathroom, etc. In this example, the interior of the house is photographed four times, and four point cloud data #1 to #4 are generated. The point cloud data #1 and #2 are combined to generate combined point cloud data #S1, and the point cloud data #3 and #4 are combined to generate combined point cloud data #S2. The combined point cloud data #S1 and #S2 are further combined to generate combined point cloud data #S3, which represents the entire target location. Note that the order in which multiple point cloud data are combined is not limited to this example.

[0044] Next, we will explain the general configuration of the photographing device 1 and the control device 2. Fig. 3 is a block diagram showing the general configuration of the photographing device 1 and the control device 2. Fig. 4 is a block diagram showing an overview of the processing performed by the processor 46 of the control device 2.

[0045] As shown in FIG. 3, the sensor unit 11 of the image capturing device 1 includes a visible camera 21, a depth sensor 22, an IMU (Inertial Measurement Unit) 23, and an input / output interface 24.

[0046] The visible light camera 21 (color camera) takes color photographs and outputs color photographed images.

[0047] The depth sensor 22 outputs depth information (distance image) as a detection result based on images captured by left and right infrared cameras (not shown).

[0048] The IMU 23 detects the motion state of the device itself, specifically, three-dimensional angular velocity and acceleration. Based on the detection results of the IMU 23, the position, orientation, and velocity of the sensor unit 11 can be detected.

[0049] The input / output interface 24 inputs and outputs data to and from the control device 2 via the display / input panel unit 12. Specifically, the detected data from the visible camera 21, the depth sensor 22, and the IMU 23 is transmitted through the input / output interface 24. The input / output interface 24 may be based on the USB (registered trademark) standard.

[0050] The visible camera 21, depth sensor 22, and IMU 23 may not be integrated. Alternatively, the depth sensor 22 and IMU 23 may be omitted, and only the visible camera 21 may be provided. Alternatively, the visible camera 21 may be provided together with either the depth sensor 22 or the IMU 23.

[0051] The display / input panel unit 12 of the photographing device 1 includes a touch panel display 31 , a repeater 32 , and an input / output interface 33 .

[0052] The touch panel display 31 displays, under the control of the control device 2, a screen that assists the user in taking photographs.

[0053] The repeater 32 repeats data communication between the control device 2 and the sensor unit 11. The repeater 32 may be a hub based on the USB (registered trademark) standard.

[0054] The input / output interface 33 inputs and outputs data to and from the control device 2. Specifically, it receives display information such as a screen that supports the user's photographing operation from the control device 2. The input / output interface 33 may be based on the USB (registered trademark) standard.

[0055] The control device 2 includes an input / output interface 41, a display 42 (display device), an input device 43, a memory 44, a storage device 45, and a processor 46.

[0056] The input / output interface 41 inputs and outputs data to and from the image capturing device 1. The input / output interface 41 may be based on the USB (registered trademark) standard.

[0057] The display 42 displays a screen relating to management of previously acquired photographic data, a screen relating to setting of operating conditions of the photographic device 1, a screen relating to merging of point cloud data, and the like.

[0058] The input device 43 is used by the user to perform input operations. The input device 43 may be a keyboard, a mouse, a touchpad, a touch panel, etc. If the control device 2 is configured as a tablet PC, a touch panel display is provided in which the touch panel as the input device 43 and the display panel as the display 42 are integrated.

[0059] The memory 44 stores programs executed by the processor 46 and the like.

[0060] The storage device 45 stores the shooting data (shooting information) acquired from the image capturing device 1. The shooting data includes the image captured by the visible camera 21, the detection result (distance information) of the depth sensor 22, the detection result of the IMU 23, the shooting time, etc. The storage device 45 also stores point cloud data generated by the processor 46 as a 3D reconstruction result.

[0061] The processor 46 performs various processes by executing programs stored in the memory 44. In this embodiment, the processor 46 performs a three-dimensional reconstruction process P1.

[0062] In the three-dimensional reconstruction process P1, the processor 46 generates point cloud data (environment map) as three-dimensional spatial information about the target location using the SLAM (Simultaneous Localization And Mapping) method based on images captured by the visible camera 21. In the three-dimensional reconstruction process P1, self-position estimation is performed in addition to the generation of the point cloud data, and the self-position for each time, i.e., the position of the photographing point, is acquired.

[0063] As shown in FIG. 4, the three-dimensional reconstruction process P1 includes a feature extraction process P11, a tracking process P12, a position and orientation correction process P13, and a point cloud generation process P14.

[0064] In the feature extraction process P11, the processor 46 extracts feature information (feature points, etc.) from the image (frame) captured by the visible camera 21.

[0065] In tracking process P12, processor 46 compares the currently extracted feature points with previously extracted feature points to estimate the amount of transition related to the position and orientation of the image capture device 1, and updates the position and orientation trajectory data based on the amount of transition. Note that the position and orientation trajectory data is obtained by tracking the position and orientation of the image capture device 1, and includes the tracking results of the position and orientation at the time of capturing each captured image (frame), i.e., information regarding the position and orientation of the image capture device 1 at each time of capturing.

[0066] In the position and orientation correction process P13, the processor 46 corrects the position and orientation trajectory data acquired in the tracking process P12 based on the detection data of the IMU 23. Here, the position and orientation trajectory data is corrected so as to compensate for the measurement results of locations with few features, such as walls and ceilings.

[0067] In the point cloud generation process P14, the processor 46 generates point cloud data based on the distance information from the depth sensor 22 and the position and orientation trajectory data acquired in the tracking process P12 and the position and orientation correction process P13. The point cloud generation process P14 includes a process (standard point cloud generation process) for generating regular point cloud data acquired as a 3D reconstruction result, and a process (simplified point cloud generation process) for generating simplified point cloud data that allows the user to check the shooting conditions (point cloud generation conditions). The simplified point cloud generation process must be performed in real time during shooting, but the standard point cloud generation process may be performed after shooting.

[0068] The processor 46 also performs a point cloud alignment process P2, a point cloud joining process P3, a drawing generation process P4, a display information generation process P5, and a display process P6.

[0069] In the point cloud alignment process P2, the processor 46 performs alignment to align common areas of multiple point cloud data to be combined. Specifically, as an alignment condition for aligning common areas of two point cloud data on the source side (alignment source) and the target side (alignment destination), a transformation matrix for coordinate transformation of each point of the source side point cloud data is calculated, and the transformation matrix is ​​applied to the source side point cloud data to perform coordinate transformation of the source side point cloud data.

[0070] In the point cloud combining process P3, the processor 46 combines the point cloud data that have been aligned in the point cloud alignment process P2 to generate a single combined point cloud data that represents the entire target location.

[0071] In the drawing generation process P4, the processor 46 generates a 3D model representing the entire target location based on the point cloud data representing the entire target location generated by the 3D reconstruction process P1, and generates a layout drawing representing the entire target location based on the 3D model. At this time, a 2D layout drawing (plan view) representing the entire target location is generated by projecting the 3D model representing the entire target location onto a horizontal plane. In addition, a 3D layout drawing representing the entire target location is generated by projecting the 3D model representing the entire target location based on a predetermined line of sight.

[0072] In the display information generation process P5, the processor 46 generates display information for a screen to be displayed on the display 42 of the control device 2. The display 42 displays a screen on which the user performs operations such as managing the photographed data, issuing instructions for post-photography processing (point cloud generation process P14, point cloud alignment process P2, point cloud joining process P3, and drawing generation process P4), and setting processing conditions. The processor 46 also generates display information for a screen to be displayed on the touch panel display 31 of the photographing device 1. The touch panel display 31 displays a screen to assist the user in the photographing operation.

[0073] In the display process P6, the processor 46 displays a screen on the display 42 of the control device 2 and also displays a screen on the touch panel display 31 of the image capturing device 1 based on the display information generated in the display information generation process P5.

[0074] In this embodiment, the processing related to the merging of point cloud data is performed in a control device 2 that can be carried by a user, but the processing related to the merging of point cloud data may also be performed in another device (e.g., a server device) that can communicate with the control device 2.

[0075] Next, a description will be given of the procedure of the process carried out by the control device 2. FIG.

[0076] First, the processor 46 acquires the point cloud data to be combined in response to a user's operation to specify a point cloud file in which the point cloud data to be combined is stored (ST101). The point cloud data generated by the 3D reconstruction process P1 is stored in the storage device 45.

[0077] Next, the processor 46 performs alignment to align the common areas of the multiple point cloud data to be combined (point cloud alignment process) (ST102). At this time, as an alignment condition for aligning the common areas of the two point cloud data on the source side and the target side, a transformation matrix for coordinate transformation of each point of the point cloud data on the source side is obtained, and the transformation matrix is ​​applied to the point cloud data on the source side to perform coordinate transformation of the point cloud data on the source side.

[0078] Next, the processor 46 combines the point cloud data that have been aligned in the point cloud alignment process (ST102) to generate one combined point cloud data that represents the entire target location (point cloud combination process) (ST103). At this time, in addition to combining two pieces of point cloud data, combining one combined point cloud data with another point cloud data, or combining one combined point cloud data with another combined point cloud data, is performed as necessary.

[0079] Next, a description will be given of the procedure of the point cloud alignment process performed by the control device 2. Fig. 6 is a flowchart showing the procedure of the point cloud alignment process.

[0080] The positional relationships between the multiple point cloud data generated by taking multiple photographs of the target location cannot be accurately recognized. Therefore, when combining the multiple point cloud data, the control device 2 performs a process of aligning the common areas of the multiple point cloud data (point cloud alignment process).

[0081] First, the processor 46 displays a point cloud image that visualizes each of the multiple point cloud data to be combined on the screen of the display 42, and in response to a user's operation to overlap the common parts of the two point cloud images, obtains information (preliminary alignment information) regarding the relative positional relationship between the two overlapping point cloud images (preliminary alignment process) (ST201).

[0082] Next, based on the preliminary alignment information acquired in the preliminary alignment process, processor 46 extracts point clouds contained in the common area from each of the two point cloud data on the source side and the target side, and generates two extracted point cloud data on the source side and the target side (common area extraction process) (ST202).

[0083] Next, the processor 46 performs preprocessing (ST203) on the two extracted point cloud data on the source side and the target side extracted in the common area extraction process (ST202) to properly perform the next rough alignment process (ST204).

[0084] Next, the processor 46 determines the rough alignment conditions (transformation matrix) for roughly aligning the two extracted point cloud data on the source side and the target side, and applies the rough alignment conditions to the extracted point cloud data on the source side to perform coordinate transformation of the extracted point cloud data on the source side (rough alignment process) (ST204). By performing the rough alignment process in this way prior to the fine alignment process, it is possible to avoid the problem of getting stuck in a local solution in the fine alignment process and obtain appropriate fine alignment conditions.

[0085] Next, the processor 46 uses the extracted point cloud data aligned using the rough alignment conditions (transformation matrix) acquired in the rough alignment process (ST204) to determine fine alignment conditions (transformation matrix) for precisely aligning the two point cloud data on the source side and the target side, and applies these fine alignment conditions to the point cloud data on the source side to perform coordinate transformation of the point cloud data on the source side (fine alignment process) (ST205).

[0086] Next, a description will be given of the preliminary alignment process performed by the control device 2. Fig. 7 is a flowchart showing the procedure of the preliminary alignment process.

[0087] In the preliminary alignment process (ST201 in FIG. 6), first, the processor 46 generates a point cloud image that visualizes the point cloud data in a bird's-eye view (point cloud image generation process) (ST301). Specifically, the point cloud image is generated by projecting each point of the point cloud data onto a two-dimensional horizontal plane. At this time, multiple points of the point cloud data may be compressed into one point to reduce the size of the point cloud image.

[0088] Next, the processor 46 displays a canvas screen 61 (point cloud operation screen, see FIG. 9) on the display 42 (ST302). On the canvas screen 61, the user can perform an operation to overlap the common areas of two point cloud images that visualize each of the two point cloud data.

[0089] Next, the processor 46 acquires information (preliminary alignment information) regarding the relative positional relationship between the two overlapping point cloud images based on the user's operation on the canvas screen 61 (ST303).

[0090] Next, a description will be given of the menu screen 51 displayed on the display 42 of the control device 2. FIG.

[0091] The menu screen 51 has a "Generate point cloud" button 52, a "Display point cloud" button 53, a "Combine point clouds" button 54, a "Generate drawing" button 55, a "Display drawing" button 56, and a "List of shooting data" button 57.

[0092] When the user operates the "Generate Point Cloud" button 52, the point cloud generation process P14 is started. At this time, individual point cloud data is generated based on individual photographic data generated by partially photographing the target location. When the user operates the "Display Point Cloud" button 53, the screen transitions to one that displays the point cloud data generated in the point cloud generation process P14, and a point cloud image that visualizes the individual point cloud data is displayed.

[0093] Furthermore, when the user operates the "Point Cloud Combination" button 54, the point cloud alignment process P2 is initiated, and the screen transitions to a screen where the user can perform operations related to the combination of individual point cloud data, specifically the canvas screen 61 (see FIG. 9). Note that when an operation is performed on the canvas screen 61 to instruct the execution of the point cloud combination process P3, the point cloud combination process is initiated, and as a processing result, a combined point cloud image that visualizes the combined point cloud data that represents the entire target location is displayed.

[0094] Furthermore, when the user operates the "Drawing Generation" button 55, the drawing generation process P4 is started. At this time, a drawing representing the entire target location is generated based on the connection point cloud data representing the entire target location. When the user operates the "Drawing Display" button 56, the screen transitions to one displaying the drawing generated in the drawing generation process P4, and the drawing representing the entire target location is displayed.

[0095] Furthermore, when the user operates the "photography data list" button 57, the screen transitions to a screen that displays a list of individual photography data. On this screen, the user can check the individual photography data.

[0096] Next, we will explain the canvas screen 61 displayed on the display 42 of the control device 2 during the preliminary alignment process. Fig. 9 is an explanatory diagram showing the canvas screen 61. Figs. 10, 11, and 12 are explanatory diagrams showing the operation status of the point cloud image 71 and the combined point cloud image 74 on the canvas 72.

[0097] During the preliminary alignment process (see FIG. 7), a canvas screen 61 (point cloud operation screen) shown in FIG.

[0098] The canvas screen 61 is provided with a canvas window 62, an operation window 63, and a selected image window 64.

[0099] Additionally, the canvas screen 61 is provided with a "Load point cloud file" button 65. When the user operates the "Load point cloud file" button 65, a screen (not shown) for specifying a point cloud file (a file or folder in which point cloud data is stored) is displayed. When the user performs an operation to specify a point cloud file here, the point cloud data stored in the specified point cloud file is input, and a point cloud image 71 that visualizes the point cloud data is displayed in the canvas window 62.

[0100] At this time, by inputting multiple pieces of point cloud data to be combined, multiple point cloud images 71, which visualize the multiple pieces of point cloud data to be combined, are displayed side by side on the canvas 72 in the canvas window 62. The point cloud images 71 are visualized as a bird's-eye view of the point cloud data. In the example shown in Fig. 9, point cloud images 71 #1 to #4, which visualize four pieces of point cloud data #1 to #4 to be combined, are displayed.

[0101] Furthermore, in the canvas window 62, the size and relative positional relationship of the point cloud image 71 with respect to the canvas 72 are maintained constant. The point cloud image 71 is displayed on the canvas 72 at a size proportional to the actual size of the corresponding point cloud data. Note that in the initial state when point cloud data is input, multiple point cloud images 71 may be arranged on the canvas 72 so as to correspond to the actual positional relationship of the original point cloud data.

[0102] The selected image window 64 displays thumbnails 73 of the point cloud images 71 that are selected in the canvas window 62. In the example shown in Fig. 9, the point cloud images 71 #1 and #2 are selected. Note that in the canvas window 62, a mark indicating the selected state may be displayed on the point cloud images 71 that are selected.

[0103] Note that the example shown in Figure 9 is for a house, but when the target location is large, such as a factory, photography may be taken multiple times (for example, 10 times), and there may be a large number of point cloud data (for example, 10 pieces) to be combined.In this case, the canvas window 62 displays a large number of point cloud images 71 lined up on the canvas 72.

[0104] Furthermore, in the canvas window 62, the user performs an alignment operation to adjust the position and angle of the point cloud image 71 so that the common areas of the two point cloud images 71 overlap each other on the canvas 72. At this time, the user can perform an operation to translate the point cloud image 71 and an operation to rotate the point cloud image 71 as the alignment operation. For example, the point cloud image 71 is translated by grabbing and moving the point cloud image 71 (for example, a drag operation). Furthermore, the point cloud image 71 is rotated by operating a rotation mark (not shown) that is displayed when an operation to select the point cloud image 71 (for example, a click operation) is performed.

[0105] The operation window 63 is provided with a "Merge" button 81 as an operation unit for instructing the execution of the point cloud merging process. The "Merge" button 81 becomes operable when two point cloud images 71 overlap in response to a user operation in the canvas window 62. Note that the "Merge" button 81 may appear when the point cloud images 71 overlap.

[0106] When the user operates the "Execute merge" button 81, the point cloud alignment process (ST102 in FIG. 5) performs a preliminary alignment process (ST201 in FIG. 6) followed by a common area extraction process, pre-processing, rough alignment process, and fine alignment process (ST202 to ST205 in FIG. 6), as well as a point cloud merging process (ST103 in FIG. 5), and a merged point cloud image that visualizes the merged point cloud data as a result of the process is displayed on the canvas screen 61. This allows the user to visually check whether the point cloud data has been merged appropriately.

[0107] In the examples shown in Figures 10, 11, and 12, first, from the initial state shown in Figure 10(A), the user operates the point cloud images 71 of #1 and #2 on the canvas 72 to overlap the point cloud images 71 of #1 and #2 so that their common areas match, as shown in Figure 10(B). At this time, rotation and translation operations are performed as necessary. Next, when the user operates the "Execute Merge" button 81, the point cloud merging process is executed, and the merged point cloud image 74 of #S1 is displayed as the processing result, as shown in Figure 11(A).

[0108] 11(A), the user operates the point cloud images 71 of #3 and #4 to overlap them so that their common areas are aligned. At this time, rotation and translation operations are performed as necessary. Next, when the user operates the "Execute Merge" button 81, the point cloud merging process is executed, and the merged point cloud image 74 of #S2 is displayed as the processing result, as shown in FIG. 11(B).

[0109] Next, as shown in Figure 12(A), the user operates the connection point cloud images 74 of #S1 and #S2 to overlap the connection point cloud images 74 of #S1 and #S2 so that their common areas match. At this time, rotation and translation operations are performed as necessary. Next, when the user operates the "Execute Merge" button 81, the point cloud merging process is executed, and the connection point cloud image of #S3 is displayed as the processing result, as shown in Figure 12(B).

[0110] In this embodiment, the point cloud merging process for merging the two point cloud data is executed by simultaneously selecting two point cloud images 71 with overlapping common areas and then operating the "Execute Merge" button 81, but it is also possible to simultaneously select three or more point cloud images 71 with overlapping common areas and then operate the "Execute Merge" button 81. In this case, merging of two point cloud data, merging of merged point cloud data with point cloud data, or merging of two merged point cloud data is executed in an appropriate combination.

[0111] The canvas screen 61 also has a "Save connection point cloud file" button 66 as an operation unit with which the user approves the connection results and instructs saving of the data. When the user operates the "Save connection point cloud file" button 66, a connection point cloud file storing the connection point cloud data is generated and accumulated in the storage device 45. When the user operates the "Execute connection" button 81, a simple point cloud connection process is executed, and a connection point cloud image for the user to check the processing results is displayed on the canvas screen 61, while when the user operates the "Save connection point cloud file" button 66, a regular point cloud connection process may be executed.

[0112] In this embodiment, the preliminary alignment process is performed based on a user operation, but the preliminary alignment process may be performed without a user operation. Specifically, the processor 46 may perform a process such as image matching on the two point cloud images to detect similar areas in the two point cloud images as common areas (common area detection process). Furthermore, this common area detection process may be performed to detect similar common areas in three or more point cloud images.

[0113] Next, we will explain the image opacity adjustment function of the canvas screen 61. Fig. 13 is an explanatory diagram showing a situation in which the opacity of the point cloud image 71 is adjusted on the canvas screen 61.

[0114] The operation window 63 of the canvas screen 61 is provided with a slider 83 as an operation unit for specifying the opacity of the point cloud image 71. When the user operates the slider 83 after performing an operation to select the point cloud image 71, the opacity of the selected point cloud image 71 changes.

[0115] When the opacity is 100%, the point cloud image 71 is displayed opaque. When the opacity is less than 100%, the point cloud image 71 is displayed semi-transparently. Therefore, when the opacity of the upper point cloud image 71 of two overlapping point cloud images 71 is set to less than 100%, the lower point cloud image 71 is visible through the overlapping portion of the two point cloud images 71, and at the same time, the upper point cloud image 71 is also visible.

[0116] Furthermore, when two point cloud images 71 overlap vertically, the opacity may be set only for the upper point cloud image 71, or the same opacity may be set for both overlapping point cloud images 71. In the example shown in Fig. 13(A), two point cloud images 71, #1 and #2, which overlap vertically, are both selected and their opacity is specified, and as shown in Fig. 13(B), the two point cloud images 71, #1 and #2, are displayed semi-transparently with the same opacity. This allows the user to easily check whether the two point cloud images 71 are overlapped with their common areas properly aligned.

[0117] When the opacity is specified by operating the slider 83, the point cloud image 71 is displayed in a semi-transparent state according to the specified opacity, but the point cloud image 71 may normally be displayed in a non-transparent state and may be displayed in a semi-transparent state only while the point cloud image 71 is being operated.

[0118] Next, a description will be given of the canvas zoom function of the canvas screen 61. Fig. 14 is an explanatory diagram showing a situation in which the canvas 72 is zoomed in or out on the canvas screen 61.

[0119] The canvas screen 61 is provided with an enlargement button 84 and a reduction button 85 as operation units for instructing the enlargement or reduction of the canvas 72. When the user operates the enlargement button 84, the canvas 72 gradually enlarges. At this time, the canvas 72 transitions from a standard state (see FIG. 9) in which the entire canvas 72 is displayed in the display area of ​​the canvas screen 61 to an enlarged state (see FIG. 14) in which only a portion of the canvas 72 fits within the display area. On the other hand, when the user operates the reduction button 85 while the canvas 72 is in the enlarged state, the canvas 72 gradually reduces and returns to the standard state.

[0120] In this way, when the user operates the zoom in button 84 and the zoom out button 85, the canvas 72 is zoomed in or out, and the point cloud image 71 on the canvas 72 is displayed zoomed in or out accordingly. This allows the user to easily visually check whether the common areas of the point cloud images 71 are aligned with sufficient accuracy. Note that the operation unit for instructing the zoom in or out of the canvas 72 is not limited to the zoom in button 84 and the zoom out button 85, and may be, for example, a slider or a numerical input of the zoom ratio.

[0121] Here, when the user operates the enlarge button 84 to enlarge the canvas 72, only a portion of the canvas 72 fits within the display area. At this time, by selecting only the point cloud image 71 to be aligned, the canvas 72 is enlarged with the selected point cloud image 71 at the center. This prevents the point cloud image 71 to be aligned from being outside the display area. In the example shown in FIG. 14, two point cloud images 71, #1 and #2, are selected, so the canvas 72 is enlarged with the midpoint between the two point cloud images 71 at the center. At this time, only a portion of the point cloud image 71 near the selected point cloud image 71 is displayed, and the point cloud image 71 farther from the selected point cloud image 71 is outside the display area.

[0122] Furthermore, on the canvas screen 61, the canvas 72 is enlarged or reduced by changing the distance of the viewpoint from the canvas 72 while maintaining the size and relative positional relationship of the point cloud image 71 with respect to the canvas 72 constant. That is, when the user operates the enlarge button 84, the viewpoint moves closer to the canvas 72, causing the point cloud image 71 on the canvas 72 to be enlarged (zoomed in). Also, when the user operates the reduce button 85, the viewpoint moves away from the canvas 72, causing the point cloud image 71 on the canvas 72 to be reduced (zoomed out). In this way, the size and relative positional relationship of the point cloud image 71 with respect to the canvas 72 are maintained constant, thereby reducing the load of the display processing on the canvas screen 61. That is, when a large number of point cloud images 71 are arranged on the canvas 72, the display processing of changing the position and angle of the point cloud images 71 in response to user operations becomes easier.

[0123] Next, a description will be given of the merge cancel function on the canvas screen 61. Fig. 15 is an explanatory diagram showing the operation status of merge cancel on the canvas screen 61.

[0124] The canvas screen 61 is provided with an "Undo Merge" button 82 as an operation unit for instructing restoration of the state before merging. When the user operates the "Undo Merge" button 82 after selecting the merged point cloud image 74, the point cloud merging process for the selected merged point cloud image 74 is canceled and the state is restored to its original state. This allows the user to return to the state before merging with a simple operation if they erroneously operate the "Execute Merge" button 81 or if an undesired merged result is obtained.

[0125] In the example shown in Figure 15(A), one connection point cloud image 74 is selected. A thumbnail 73 of the selected connection point cloud image 74 is displayed in the selected image window 64. In the example shown in Figure 15(B), the point cloud combination process for the connection point cloud image 74 selected in the example shown in Figure 15(A) is canceled, and the state is restored to the state before combination, that is, a state in which an alignment operation has been performed and the two point cloud images 71 are superimposed. Note that when the connection point cloud image 74 is returned to the point cloud image 71 before combination, the point cloud image 71 is no longer selected, and the thumbnail 73 of the point cloud image 71 is no longer displayed in the selected image window 64.

[0126] Next, we will explain the image frame display function of the canvas screen 61. Figure 16 is an explanatory diagram showing a situation in which a frame image 78 is displayed on the canvas screen 61 around the point cloud image 71.

[0127] The canvas screen 61 is provided with an "Image Frame Display / Hide" button 86 as an operation unit for instructing the display of a frame image 78 representing the periphery of the point cloud image 71. When the user operates the "Image Frame Display / Hide" button 86 after performing an operation to select the point cloud image 71, the frame image 78 is displayed on the periphery of the selected point cloud image 71.

[0128] The example shown in Fig. 16(A) is the initial state (non-display state), and operating the "Image Frame Display / Hide" button 86 transitions to a state (display state) in which a frame image 78 is displayed, as shown in Fig. 16(B). This makes it easier to distinguish the overlapping areas of the selected point cloud images 71 using the frame image 78, allowing the user to easily check whether the common areas of the point cloud data are appropriately overlapping. In the example shown in Fig. 16, multiple point cloud images 71 are selected, and thumbnails 73 of the selected point cloud images 71 are displayed in the selected image window 64.

[0129] Furthermore, when the user operates the "Image frame display / hide" button 86 in the display state of the frame image 78, the state returns to the original state (non-display state) in which the frame image 78 is not displayed. In this way, when the user operates the "Image frame display / hide" button 86, it is possible to switch between a display state in which the frame image 78 is displayed and a non-display state in which the frame image 78 is not displayed.

[0130] Next, we will explain the image lock function of the canvas screen 61. Fig. 17 is an explanatory diagram showing a situation in which the point cloud image 71 is set to a locked state on the canvas screen 61.

[0131] The canvas screen 61 is provided with an "Image Lock" button 87 as an operation unit for instructing to lock the point cloud image 71. When the user operates the "Image Lock" button 87 after performing an operation to select a point cloud image 71, the selected point cloud image 71 is set to a locked state (a fixed state in which operations are not possible). The user cannot operate a locked point cloud image 71, and can only operate an unlocked point cloud image 71. Furthermore, on the canvas screen 61, when a point cloud image 71 is set to a locked state, a lock mark 79 indicating the locked state is displayed on the corresponding point cloud image 71.

[0132] In the example shown in FIG. 17(A), point cloud image 71 #1 is selected, and a thumbnail 73 of the selected point cloud image #1 is displayed in the selected image window 64. When the user operates the "Lock Image" button 87, the selected point cloud image #1 is set to a locked state, and a lock mark 79 is displayed, as shown in FIG. 17(B). Point cloud image #2 is in an operable state, and is operated so that the common area overlaps with point cloud image #1 71. At this time, point cloud image #1 71 that has been set to a locked state returns to an unselected state, but because point cloud image #2 71 is in a selected state, a thumbnail 73 of point cloud image #2 71 is displayed in the selected image window 64.

[0133] By locking the point cloud image 71 in this way, it is possible to prevent the point cloud image 71 that is not being operated from being moved by an erroneous operation by the user. Furthermore, the lock mark 79 allows the user to easily understand the locked state of the point cloud image 71. Note that, in the example shown in Fig. 17, the lock mark 79 is drawn in a circular shape in a predetermined color (e.g., green), but the form of the lock mark 79 is not limited to this.

[0134] In the example shown in Figure 17, the point cloud image 71 corresponding to the point cloud data to be combined is set to a locked state, but the combined point cloud image 74 (see Figures 11 and 12) corresponding to the combined point cloud data generated by the point cloud combination process can also be set to a locked state by a similar operation.

[0135] Furthermore, one or more point cloud images 71 may be set to the locked state at the same time. For example, during an operation to overlap the common areas of two point cloud images 71, only one point cloud image 71 on the target side (the point cloud image 71 to be aligned) may be set to the locked state. Furthermore, in order to prevent erroneous operation of the point cloud images 71, all of the other point cloud images 71 except for the point cloud image 71 on the source side (the point cloud image to be aligned) may be set to the locked state.

[0136] Next, a description will be given of the combination history screen 101 displayed on the display 42 of the control device 2. FIG.

[0137] The canvas screen 61 (see FIG. 9) is provided with a "Merge History" button 88 as an operation unit for instructing the display of merge history information that shows the history of merging of multiple point cloud data. When the user operates the "Merge History" button 88 after performing an operation to select a merge point cloud image 74 (see FIG. 10(B)), the screen transitions to a merge history screen 101 shown in FIG. 18. Note that on the merge history screen 101, when the user performs an operation to select a merge point cloud image 74, a thumbnail 73 of the selected merge point cloud image 74 is displayed in the selected image window 64.

[0138] The merge history screen 101 displays a system diagram 102 (tree diagram) as merge history information that shows the history of merging multiple point cloud data. By viewing the system diagram 102, the user can easily check the merge status of multiple point cloud data, that is, the combinations in which the point cloud data were merged. In particular, even in cases where the target location is large, such as a factory, and there are many point cloud data to be merged, the user can easily check the merge status of the point cloud data.

[0139] 18, the topmost displayed connection point cloud image 74 corresponds to the latest connection point cloud data, and the bottommost displayed point cloud image 71 corresponds to the source point cloud data. The user can trace back the connection status of the point cloud data by tracing the system diagram 102 downward.

[0140] The merge history screen 101 also displays, for each merge point cloud image 74, a registration match rate (evaluation information) of the common area, which indicates the degree of appropriateness of the registration using the transformation matrix estimated at the time of merging in the merge point cloud data corresponding to that merge point cloud image 74. This allows the user to easily confirm the degree of appropriateness of the registration using the transformation matrix estimated at the time of merging in the merge point cloud data. For example, if the registration match rate of the common area is low, the user can confirm that the accuracy of the registration in the merge point cloud data is low, that is, the operation of merging the common area of ​​the merge source point cloud image 71 may be insufficient.

[0141] Here, the registration match rate of the common area is expressed, for example, as the match rate (%) of the common area based on the transformation matrix for registration estimated when the two point cloud data to be combined are combined. The match rate is calculated based on the distance between corresponding points in the two point cloud data. Specifically, for each point in the source point cloud after alignment, the nearest point in the target point cloud is determined to be the corresponding point. If the distance between the corresponding points is less than a threshold value (e.g., 3 cm), the corresponding point is determined to be acceptable. If the distance between the corresponding points is greater than the threshold value, the corresponding point is determined to be unacceptable. The match rate is then calculated as the ratio of the number of corresponding points determined to be acceptable to the total number of detected corresponding points. For example, if 50 pairs of corresponding points out of 100 pairs are acceptable, the match rate is 50%. A low match rate indicates that the registration accuracy of the combined point cloud data is low.

[0142] The join history screen 101 also has a "cancel join" button 103. When the user operates the "cancel join" button 103 after selecting the join point cloud image 74, the point cloud joining process related to the selected join point cloud image 74 is canceled, and the state is restored to the state before the point cloud joining process was executed.

[0143] At this time, the point cloud combining process that generated the combining point cloud data corresponding to the selected combining point cloud image 74 is canceled. Also, the point cloud combining process based on the combining point cloud data corresponding to the selected combining point cloud image 74 is canceled. In other words, the point cloud combining process from the combining point cloud data corresponding to the selected combining point cloud image 74 to the combining point cloud data corresponding to the latest combining point cloud image 74 is canceled.

[0144] 18, the #S1 connection point cloud image 74 is selected. In this case, the connection point cloud data corresponding to the #1 connection point cloud image 74 is restored to the point cloud data corresponding to the original #1 and #2 point cloud images 71. In addition, the connection point cloud data corresponding to the #S3 connection point cloud image 74, which was generated based on the connection point cloud data corresponding to the #S1 connection point cloud image 74, is restored to its original state.

[0145] (Modification of the first embodiment) Next, a modified example of the first embodiment will be described. Note that points not specifically mentioned here are the same as those in the above-described embodiment. Figure 19 is an explanatory diagram showing a main canvas screen 111 and a sub-canvas screen 112 displayed on the display 42 of the control device 2 according to a modified example of the first embodiment.

[0146] Similar to the canvas screen 61 (see FIG. 9) in the first embodiment, the main canvas screen 111 displays a plurality of point cloud images 71, which are visualizations of a plurality of point cloud data to be combined, side by side. Also, the main canvas screen 111 is provided with a "Sub-canvas" button 68.

[0147] When the user operates the "sub-canvas" button 68 after selecting the point cloud image 71 on the main canvas screen 111, the sub-canvas screen 112 pops up on the main canvas screen 111. Note that the main canvas screen 111 and the sub-canvas screen 112 may be displayed in a switchable manner.

[0148] Only the point cloud images 71 selected by the user on the main canvas screen 111 are displayed side by side on the sub-canvas screen 112. This allows the user to perform the alignment operation by displaying only the point cloud images 71 currently being aligned among the point cloud images 71 displayed on the main canvas screen 111 on the sub-canvas screen 112. This makes it easier for the user to perform the alignment operation of overlapping the common areas of the two point cloud images 71 to be combined.

[0149] Note that the sub-canvas screen 112 may be provided with a "Main Canvas" button for returning to the main canvas screen 111. Also, the user may operate a button to close the sub-canvas screen 112 to return to the main canvas screen 111.

[0150] (Second embodiment) Next, a second embodiment will be described. It should be noted that the points not specifically mentioned here are the same as those of the above-described embodiment. Fig. 20 is an explanatory diagram showing a canvas screen 61 displayed on the display 42 of the control device 2 according to the second embodiment.

[0151] In the first embodiment, a preliminary alignment process is performed based on a user's operation, that is, a preliminary alignment process based on a user's operation to overlap a common area of ​​two point cloud data. On the other hand, in this embodiment, in addition to the preliminary alignment process based on a user's operation, a preliminary alignment process that is not based on a user's operation can also be performed. Note that the rough alignment process and fine alignment process performed after the preliminary alignment process are the same as those in the first embodiment (see FIG. 6).

[0152] In the preliminary alignment process not based on user operation, a process of detecting a common area between two point cloud data (common area detection process) is performed. In the common area detection process, image matching is performed between two point cloud images 71 that visualize point cloud data to measure the similarity between the point cloud images 71, and a common area between the two point cloud images 71 is detected based on the measurement result. Specifically, an area with high similarity between the two point cloud images 71 is detected as a common area. At this time, for example, an area between the point cloud images 71 where the similarity is equal to or greater than a predetermined threshold is detected as a common area. Furthermore, an area that is included in a predetermined percentage of the most similar point cloud images 71 is detected as a common area.

[0153] In this embodiment, a preliminary alignment process not based on a user's operation is first performed, and if the process does not produce an appropriate result, a preliminary alignment process based on a user's operation is performed again. That is, the user first performs an operation to instruct the execution of a preliminary alignment process not based on a user's operation, checks the processing result, and if the processing result is not appropriate, the point cloud combining process is restored and then the preliminary alignment process based on the user's operation is performed.

[0154] Specifically, as shown in Fig. 20, the canvas screen 61 is provided with an "execute merge" button 81, an "cancel merge" button 82, and an "execute automatic merge" button 121. When the user operates the "execute automatic merge" button 121 after performing an operation (for example, a click operation) to select two point cloud images 71 to be merged, a preliminary alignment process not based on a user operation and a point cloud merge process based on the result of that process are executed, and a merged point cloud image 74 (see Figs. 11 and 12) is displayed on the canvas screen 61 as the result of the processing.

[0155] The user visually checks the combined point cloud image 74 to determine whether the processing results are appropriate, and if the processing results are not appropriate, operates the "Cancel Combine" button 82. Next, after performing an operation to overlap the common areas of the two point cloud images 71 to be combined, the user operates the "Execute Combine" button 81. This executes a preliminary alignment process based on the user's operation and a point cloud combination process based on the processing results, and the combined point cloud image 74 as the processing result is displayed on the canvas screen 61.

[0156] Here, the result of the preliminary alignment process not based on a user operation may be appropriate in some cases, but the result of the process may be inappropriate in other cases. For this reason, on the canvas screen 61, a connection point cloud image 74 generated by a preliminary alignment process not based on a user operation and a connection point cloud image 74 generated by a preliminary alignment process based on a user operation may coexist.

[0157] As described above, the embodiments have been described as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited to these, and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made. Furthermore, it is also possible to combine the components described in the above embodiments to create new embodiments. [Industrial Applicability]

[0158] The point cloud combining device and point cloud combining method of the present invention have the effect of improving user operability when aligning multiple point cloud data obtained by 3D reconstruction processing of a target location when combining them, thereby reducing the burden on the user when there is a large amount of point cloud data to be combined, and are useful as a point cloud combining device and point cloud combining method that aligns two point cloud data obtained by 3D reconstruction processing of a target location and then combines the two point cloud data. [Explanation of symbols]

[0159] 1: Imaging device 2: Control device (point cloud connection device) 42: Display 46: Processor 61: Canvas screen 71: Point cloud image 72: Canvas 73: Thumbnail 74: Connected point cloud image 78: Frame image 79: Lock mark 81: "Execute join" button 82: "Undo merge" button 83: Slider 84: Enlarge button 85: Shrink button 86: "Show / hide image frame" button 87: "Image Lock" button 88: "Merge History" button 101: Combined History Screen 102: Family tree

Claims

1. A point cloud merging device that executes a process of merging two pieces of point cloud data acquired by a three-dimensional reconstruction process of a target location after performing mutual alignment, The processor: displaying on a display device a screen including a canvas on which a plurality of point cloud images, which are visualizations of the plurality of point cloud data to be combined, are arranged so as to be operable by a user, and an operation unit for instructing execution of a point cloud combining process; executes the point cloud merging process to merge the two point cloud data in response to a user's operation of aligning the common areas of the two point cloud images on the canvas and the user's operation on the operation unit; The point cloud merging device is characterized in that a screen including a merged point cloud image that visualizes the merged point cloud data generated by the point cloud merging process is displayed on the display device.

2. The processor: Displaying the combined point cloud image, which visualizes combined point cloud data generated by combining the two point cloud data, on the canvas; The point cloud combining device according to claim 1 , wherein the combined point cloud data and the point cloud data are combined in response to a user operation of superimposing a common area of ​​the combined point cloud image and the point cloud image.

3. The processor: A screen including an operation section for instructing the restoration of the state before the combination is displayed, The point cloud combining device according to claim 1, characterized in that, in response to a user's operation of the operation unit, the combined point cloud data is returned to a state before the combination, and the point cloud image corresponding to the point cloud data before the combination is displayed.

4. The processor: displaying a screen including an operation unit for specifying the opacity of the point cloud image; The point cloud combining device according to claim 1 , wherein the point cloud image is displayed in a semi-transparent state with a designated opacity in response to an operation of the operation unit by a user.

5. The processor: displaying a screen including an operation unit for instructing the enlargement or reduction of the canvas; The point cloud combining device according to claim 1 , wherein the canvas is enlarged or reduced in size in response to an operation of the operation unit by a user.

6. The processor: The point cloud combining device according to claim 5, characterized in that the canvas is enlarged or reduced by changing the distance from the viewpoint to the canvas while keeping the position and size of the point cloud image relative to the canvas constant.

7. The processor: displaying a screen including an operation unit for instructing the display of a frame image representing the periphery of the point cloud image; The point cloud combining device according to claim 1, characterized in that, in response to an operation of the operation unit by a user, a display state in which the frame image is displayed and a non-display state in which the frame image is not displayed are switched.

8. The processor: displaying a screen including an operation unit for instructing to lock the point cloud image; The point cloud combining device according to claim 1 , wherein the point cloud image is set to a locked state in response to an operation of the operation unit by a user.

9. The processor: displaying a screen including an operation unit for instructing display of combination history information representing a history of combination of the two point cloud data; 2. The point cloud merging device according to claim 1, wherein a screen including the merging history information is displayed on the display device in response to an operation of the operation unit by a user.

10. The processor: The point cloud combining device according to claim 9, characterized in that it displays a screen including the combining history information and evaluation information representing the degree of appropriateness of alignment using a transformation matrix estimated at the time of combining in the combined point cloud data generated by the point cloud combining process.

11. A point cloud merging method that causes a processor to perform a process of merging two point cloud data acquired by a three-dimensional reconstruction process of a target location after mutual alignment, comprising: displaying on a display device a screen including a canvas on which a plurality of point cloud images, which are visualizations of the plurality of point cloud data to be combined, are arranged so as to be operable by a user, and an operation unit for instructing execution of a point cloud combining process; executes the point cloud merging process to merge the two point cloud data in response to a user's operation of aligning the common areas of the two point cloud images on the canvas and the user's operation on the operation unit; A point cloud merging method characterized in that a screen including a connection point cloud image that visualizes the connection point cloud data generated by the point cloud merging process is displayed on the display device.

Images