Point cloud merging device and point cloud merging method

Abstract

[Problem] To make it possible to improve user operability during position matching when merging two sets of point cloud data that were acquired by three-dimensional reconstruction processing of a target place, and reduce the burden on the user when there is a large quantity of point cloud data to merge. [Solution] Displayed on a display is a canvas screen 61 including a canvas 72, on which two point cloud images 71 that result from visualizing two sets of point cloud data targeted for merging are positioned such that a user can perform operations thereon, and a button 81 that directs the implementation of a point cloud merging process. The point cloud merging process, which merges the two sets of point cloud data, is implemented in response to a position matching operation by the user for superimposing shared regions of the two point cloud images on the canvas, and an operation of the button by the user, and a screen including a merged point cloud image that results from visualizing point cloud data generated by the point cloud merging process is displayed.

Description

Point cloud combining device and point cloud combining method

[0001] The present disclosure relates to a point cloud merging device and a point cloud merging method that align and merge two pieces of point cloud data acquired by three-dimensional reconstruction processing of a target location.

[0002] A three-dimensional reconstruction technique is known that generates point cloud data as three-dimensional spatial information about a target location based on photographed images of the target location. Among these three-dimensional reconstruction techniques, the SLAM (Simultaneous Localization and Mapping) method has recently attracted attention. The SLAM method allows a mobile object (e.g., a worker) to carry a photographing device and capture images of the target location while moving around the target location, thereby acquiring point cloud data of the target location.

[0003] On the other hand, when the target location is large, such as a factory, or when the target location is divided into complex sections, such as a house, it is not possible to capture the entire target location at once. In such cases, the user must capture the target location multiple times. Furthermore, a process is performed in which multiple point cloud data generated by capturing partial images of the target location are combined to generate combined point cloud data representing the entire target location. In this case, since the positional relationships between the multiple point cloud data cannot be accurately recognized, combining the multiple point cloud data requires alignment to align common areas where the same subject is captured in the multiple point cloud data.

[0004] A known technique for aligning multiple point cloud data is to display a point cloud image that visualizes the point cloud data on a screen and allow a user to manipulate the point cloud image so that the common areas of the two point cloud images overlap (see Patent Document 1). In this technique, multiple markers (targets) that serve as references for alignment are placed at target locations, and the markers are displayed on the point cloud image, allowing the user to perform alignment operations based on the markers.

[0005] Japanese Patent Application Laid-Open No. 2018-147065

[0006] However, conventional techniques require the user to place multiple markers (targets) as alignment references at the target location and measure the marker locations using a surveying instrument (such as a total station), which is extremely time-consuming. Therefore, a technology that allows the user to perform alignment operations without using markers as references is desired.

[0007] Furthermore, when the target location is large, such as a factory, photography may be taken multiple times (e.g., 10 times), resulting in a large number of point cloud data sets (e.g., 10 sets) to be combined, which poses a problem of the laborious task of combining the point cloud data sets. Therefore, a technology that can reduce the burden on users when there is a large number of point cloud data sets to be combined is desired.

[0008] Therefore, the main objective of the present disclosure is to provide a point cloud combining device and a point cloud combining method that can improve user operability when aligning multiple point cloud data obtained by 3D reconstruction processing of a target location, thereby reducing the burden on the user when there is a large amount of point cloud data to be combined.

[0009] The point cloud combining device of the present disclosure is a point cloud combining device that uses a processor to execute a process of combining two point cloud data acquired by a three-dimensional reconstruction process of a target location after aligning them with each other, and the processor displays on a display device a screen including a canvas on which multiple point cloud images, which visualize the multiple point cloud data to be combined, are arranged so that they can be operated by a user, and an operation unit that instructs the execution of the point cloud combining process, and executes the point cloud combining process to combine the two point cloud data in accordance with a user's alignment operation to overlap the common areas of the two point cloud images on the canvas and a user's operation on the operation unit, and displays on the display device a screen including a combined point cloud image that visualizes the combined point cloud data generated by the point cloud combining process.

[0010] The point cloud combining method of the present disclosure also causes a processor to perform a process of combining two point cloud data acquired by a three-dimensional reconstruction process of a target location after aligning them with each other, and is configured to display on a display device a screen including a canvas on which a plurality of point cloud images visualizing the plurality of point cloud data to be combined are arranged so that the user can operate them, and an operation unit for instructing the execution of the point cloud combining process, and to execute the point cloud combining process to combine the two point cloud data in accordance with a user's operation of aligning the common areas of the two point cloud images on the canvas and a user's operation on the operation unit, and to display on the display device a screen including a combined point cloud image visualizing the combined point cloud data generated by the point cloud combining process.

[0011] According to the present disclosure, multiple point cloud images that visualize multiple point cloud data to be combined are arranged on a canvas, and the user can select and align the point cloud images on the canvas as appropriate. This improves user operability and reduces the burden on the user when there is a large amount of point cloud data to be combined.

[0012] FIG. 1 is an explanatory diagram showing the status of a photographing operation performed by a user using a photographing system according to the first embodiment; FIG. 2 is an explanatory diagram showing an example of point cloud combination processing performed by a control device; FIG. 3 is a block diagram showing the general configuration of a photographing device and a control device; FIG. 4 is a block diagram showing an overview of processing performed by a processor of the control device; FIG. 5 is a flow diagram showing the procedure of processing performed by the control device; FIG. 6 is a flow diagram showing the procedure of point cloud alignment processing performed by the control device; FIG. 7 is a flow diagram showing the procedure of preliminary alignment processing performed by the control device; FIG. 8 is a menu screen displayed on the display of the control device; FIG. 9 is a canvas screen displayed on the display of the control device; FIG. 1 is an explanatory diagram showing the operation status of a point cloud image and a combined point cloud image in the canvas screen; FIG. 2 is an explanatory diagram showing the state where the opacity of a point cloud image is adjusted on the canvas screen; FIG. 3 is an explanatory diagram showing the state where the canvas is enlarged or reduced on the canvas screen; FIG. 4 is an explanatory diagram showing the operation status of canceling a merge on the canvas screen; FIG. 5 is an explanatory diagram showing the state where a frame image is displayed on a point cloud image on the canvas screen; FIG. 6 is an explanatory diagram showing the state where a point cloud image is set to a locked state on the canvas screen;

[0013] The first invention made to solve the above problem is a point cloud merging device that uses a processor to execute a process of merging two point cloud data acquired by a 3D reconstruction process of a target location after aligning them with each other, wherein the processor displays on a display device a screen including a canvas on which a plurality of point cloud images that visualize the plurality of point cloud data to be merged are arranged so that the user can operate them, and an operation unit that instructs the execution of the point cloud merging process, and executes the point cloud merging process to merge the two point cloud data in accordance with a user's operation of aligning the common areas of the two point cloud images on the canvas and a user's operation on the operation unit, and displays on the display device a screen including a combined point cloud image that visualizes the combined point cloud data generated by the point cloud merging process.

[0014] According to this, multiple point cloud images that visualize multiple point cloud data to be merged are arranged on a canvas, and the user can appropriately select the point cloud image on the canvas and perform the alignment operation. This improves user operability and reduces the burden on the user when there is a large number of point cloud data to be merged. Note that the point cloud merging process can be used to merge two source point cloud data, merge source point cloud data with merged point cloud data, or merge two merged point cloud data.

[0015] In addition, in a second invention, the processor is configured to display on the canvas the connected point cloud image, which visualizes the connected point cloud data generated by combining two pieces of point cloud data, and to combine the connected point cloud data and the point cloud data in accordance with a user's operation to superimpose the common areas of the connected point cloud image and the point cloud image.

[0016] According to this, three or more point cloud data can be sequentially combined two by two to obtain point cloud data in which three or more point cloud data are combined. Note that in addition to combining two point cloud data or combining combined point cloud data with other point cloud data, combining two combined point cloud data is also possible.

[0017] In addition, in a third invention, the processor displays a screen including an operation unit that instructs restoration of the state before combination, and in response to a user's operation of the operation unit, returns the combined point cloud data to the state before combination and displays the point cloud image corresponding to the point cloud data before combination.

[0018] This allows the user to restore the point cloud data to its pre-combination state if the user makes an operation error or if an undesirable combination result is obtained. The operation unit may be a button provided on the screen.

[0019] In addition, a fourth invention is configured such that the processor displays a screen including an operation unit for specifying the opacity of the point cloud image, and displays the point cloud image in a semi-transparent state with the specified opacity in response to a user's operation of the operation unit.

[0020] By displaying the point cloud image in a semi-transparent state, the user can easily check the alignment state of the common area when the point cloud images are superimposed. Note that the operation unit may be a slider provided on the screen.

[0021] In a fifth aspect of the present invention, the processor displays a screen including an operation unit for instructing enlargement or reduction of the canvas, and enlarges or reduces the canvas in response to a user's operation of the operation unit.

[0022] This allows the user to easily check the alignment of the common area when the point cloud images are superimposed by enlarging or reducing the point cloud image on the canvas. The operation unit may be a button or a slider provided on the screen.

[0023] In addition, a sixth invention is configured such that the processor enlarges or reduces the canvas by changing the distance of the viewpoint from the canvas while keeping the position and size of the point cloud image relative to the canvas constant.

[0024] This allows the point cloud image on the canvas to be scaled by scaling the canvas. Furthermore, when a large number of point cloud images are arranged on the canvas, display processing for changing the positions and angles of those point cloud images in response to user operations becomes easier. In this case, when a user performs an operation to select multiple point cloud images, the canvas may be scaled around the midpoint of the selected multiple point cloud images.

[0025] In addition, a seventh invention is configured such that the processor displays a screen including an operation unit that instructs the display of a frame image representing the outer periphery of the point cloud image, and switches between a display state in which the frame image is displayed and a non-display state in which the frame image is not displayed depending on the user's operation of the operation unit.

[0026] This allows the user to easily check the degree of overlap of common areas of the point cloud data using the frame image as a guide. The operation unit may be a button provided on the screen. The point cloud image on which the frame image is displayed may be selected in advance in response to a user operation. The point cloud image on which the frame image is displayed may be one or multiple.

[0027] In addition, an eighth invention is configured such that the processor displays a screen including an operation unit for instructing locking of the point cloud image, and sets the point cloud image to a locked state in response to operation of the operation unit by a user.

[0028] This allows the point cloud image to be locked, i.e., fixed and inoperable, thereby preventing the user from accidentally moving the point cloud image. The operation unit may be a button provided on the screen. The point cloud image to be set to the locked state may be selected in advance in response to a user operation. The number of point cloud images to be set to the locked state at the same time may be one or more.

[0029] In addition, a ninth invention is configured such that the processor displays a screen including an operation unit that instructs the display of combination history information representing the history of combining two of the point cloud data, and displays a screen including the combination history information on the display device in response to a user's operation of the operation unit.

[0030] This allows the user to easily check the history of combining multiple point cloud data, i.e., the combinations in which the point cloud data have been combined up to now. The operation unit may be buttons provided on the screen.

[0031] In addition, in a tenth invention, the processor is configured to display a screen including the combination history information and evaluation information representing the degree of appropriateness of alignment using a transformation matrix estimated at the time of combination in the combination point group data generated by the point group combination processing.

[0032] This allows the user to easily check the degree of appropriateness of bonding in the bonding point group data.

[0033] An eleventh invention is a point cloud merging method that causes a processor to perform a process of merging two point cloud data acquired by a three-dimensional reconstruction process of a target location after aligning them with each other, wherein a screen including a canvas on which a plurality of point cloud images visualizing the plurality of point cloud data to be merged are arranged so that the user can operate, and an operation unit for instructing the execution of the point cloud merging process is displayed on a display device, the point cloud merging process is executed to merge the two point cloud data in accordance with a user's operation of aligning the common areas of the two point cloud images on the canvas and the user's operation on the operation unit, and a screen including a combined point cloud image visualizing the combined point cloud data generated by the point cloud merging process is displayed on the display device.

[0034] As with the first invention, this improves user operability when aligning multiple point cloud data obtained by 3D reconstruction processing of the target location, thereby reducing the burden on the user when there is a large number of point cloud data to be combined.

[0035] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0036] First Embodiment FIG. 1 is an explanatory diagram showing the situation of a photographing operation performed by a user using a photographing system according to a first embodiment.

[0037] As shown in FIG. 1, the photography system includes a photography device 1 and a control device 2 (point cloud combining device).

[0038] The photographing device 1 includes a sensor unit 11, a display / input panel unit 12, and a rod-shaped support member 13. The sensor unit 11 and the display / input panel unit 12 of the photographing device 1 are connected via a first cable 18. The display / input panel unit 12 of the photographing device 1 and the control device 2 are connected via a second cable 19.

[0039] The control device 2 is configured as a laptop or tablet PC that can be carried by a user (worker). In the example shown in Fig. 1, the control device 2 can be stored in a shoulder bag and carried by the user, but the manner in which the user carries the control device 2 is not limited to this.

[0040] A user performs a photographing task by walking around a target location while holding the photographing device 1 in his / her hand and causing the photographing device 1 to photograph the target location. At this time, the user can change the position (height) of the sensor unit 11 by moving the arm holding the photographing device 1.

[0041] Next, a description will be given of the point cloud combining process performed by the control device 2. Fig. 2 is an explanatory diagram showing an example of the point cloud combining process.

[0042] When the target location is large, such as a factory, or when the target location is divided into complex sections, such as a house, it is difficult to capture the entire target location at once. In such cases, the user takes multiple photographs of the target location. The control device 2 also performs a process of combining multiple point cloud data generated by partially photographing the target location to generate combined point cloud data representing the entire target location.

[0043] In the example shown in FIG. 2, the target location is a house. The house is equipped with a living room, kitchen, bedroom, toilet, bathroom, etc. In this example, the interior of the house is photographed four times, and four point cloud data #1 to #4 are generated. The point cloud data #1 and #2 are combined to generate combined point cloud data #S1, and the point cloud data #3 and #4 are combined to generate combined point cloud data #S2. The combined point cloud data #S1 and #S2 are further combined to generate combined point cloud data #S3, which represents the entire target location. Note that the order in which multiple point cloud data are combined is not limited to this example.

[0044] Next, a description will be given of the general configuration of the photographing device 1 and the control device 2. Fig. 3 is a block diagram showing the general configuration of the photographing device 1 and the control device 2. Fig. 4 is a block diagram showing an overview of the processing performed by the processor 46 of the control device 2.

[0045] As shown in FIG. 3, the sensor unit 11 of the image capturing device 1 includes a visible camera 21, a depth sensor 22, an IMU (Inertial Measurement Unit) 23, and an input / output interface 24.

[0046] The visible light camera 21 (color camera) takes color photographs and outputs color photographed images.

[0047] The depth sensor 22 outputs depth information (distance image) as a detection result based on images captured by left and right infrared cameras (not shown).

[0048] The IMU 23 detects the motion state of the device itself, specifically, three-dimensional angular velocity and acceleration. Based on the detection results of the IMU 23, the position, orientation, and velocity of the sensor unit 11 can be detected.

[0049] The input / output interface 24 inputs and outputs data to and from the control device 2 via the display / input panel unit 12. Specifically, the input / output interface 24 transmits detection data from the visible camera 21, the depth sensor 22, and the IMU 23. The input / output interface 24 may be based on the USB (registered trademark) standard.

[0050] The visible camera 21, the depth sensor 22, and the IMU 23 may not be integrated. Alternatively, the depth sensor 22 and the IMU 23 may be omitted, and only the visible camera 21 may be provided. Alternatively, the visible camera 21 may be provided together with either the depth sensor 22 or the IMU 23.

[0051] The display / input panel unit 12 of the photographing device 1 includes a touch panel display 31 , a repeater 32 , and an input / output interface 33 .

[0052] The touch panel display 31 displays a screen that assists the user in taking photographs, etc., under the control of the control device 2.

[0053] The repeater 32 repeats data communication between the control device 2 and the sensor unit 11. The repeater 32 may be a hub based on the USB (registered trademark) standard.

[0054] The input / output interface 33 inputs and outputs data to and from the control device 2. Specifically, the input / output interface 33 receives display information such as a screen that assists the user in taking photographs from the control device 2. The input / output interface 33 may be based on the USB (registered trademark) standard.

[0055] The control device 2 includes an input / output interface 41 , a display 42 (display device), an input device 43 , a memory 44 , a storage device 45 , and a processor 46 .

[0056] The input / output interface 41 inputs and outputs data to and from the image capturing device 1. The input / output interface 41 may be based on the USB (registered trademark) standard.

[0057] The display 42 displays a screen for managing previously acquired photographic data, a screen for setting the operating conditions of the photographic device 1, a screen for combining point cloud data, and the like.

[0058] The input device 43 is used by the user to perform input operations. The input device 43 may be a keyboard, a mouse, a touchpad, a touch panel, etc. If the control device 2 is configured as a tablet PC, a touch panel display is provided in which the touch panel as the input device 43 and the display panel as the display 42 are integrated.

[0059] The memory 44 stores programs executed by the processor 46 and the like.

[0060] The storage device 45 stores the shooting data (shooting information) acquired from the image capturing device 1. The shooting data includes the image captured by the visible camera 21, the detection results (distance information) of the depth sensor 22, the detection results of the IMU 23, the shooting time, etc. The storage device 45 also stores point cloud data generated by the processor 46 as a 3D reconstruction result.

[0061] The processor 46 performs various processes by executing programs stored in the memory 44. In this embodiment, the processor 46 performs a three-dimensional reconstruction process P1.

[0062] In the three-dimensional reconstruction process P1, the processor 46 generates point cloud data (environment map) as three-dimensional spatial information related to the target location using the SLAM (Simultaneous Localization And Mapping) method based on images captured by the visible camera 21. In the three-dimensional reconstruction process P1, self-position estimation is performed in addition to the generation of the point cloud data, and the self-position for each time, i.e., the position of the photographing point, is acquired.

[0063] As shown in FIG. 4, the three-dimensional reconstruction process P1 includes a feature extraction process P11, a tracking process P12, a position and orientation correction process P13, and a point cloud generation process P14.

[0064] In the feature extraction process P11, the processor 46 extracts feature information (feature points, etc.) from the image (frame) captured by the visible camera 21.

[0065] In tracking processing P12, the processor 46 compares the currently extracted feature points with previously extracted feature points to estimate the amount of transition related to the position and orientation of the image capture device 1, and updates the position and orientation trajectory data based on the amount of transition. Note that the position and orientation trajectory data is obtained by tracking the position and orientation of the image capture device 1, and includes the tracking results of the position and orientation at the time of capturing each captured image (frame), i.e., information regarding the position and orientation of the image capture device 1 at each time of capturing.

[0066] In the position and orientation correction process P13, the processor 46 corrects the position and orientation trajectory data acquired in the tracking process P12 based on the detection data of the IMU 23. Here, the position and orientation trajectory data is corrected so as to compensate for the measurement results of locations with few features, such as walls and ceilings.

[0067] In the point cloud generation process P14, the processor 46 generates point cloud data based on the distance information from the depth sensor 22 and the position and orientation trajectory data acquired in the tracking process P12 and the position and orientation correction process P13. The point cloud generation process P14 includes a process (standard point cloud generation process) for generating regular point cloud data acquired as a 3D reconstruction result, and a process (simplified point cloud generation process) for generating simplified point cloud data that allows the user to check the shooting conditions (point cloud generation conditions). The simplified point cloud generation process must be performed in real time during shooting, but the standard point cloud generation process may be performed after shooting.

[0068] The processor 46 also performs a point cloud alignment process P2, a point cloud joining process P3, a drawing generation process P4, a display information generation process P5, and a display process P6.

[0069] In the point cloud alignment process P2, the processor 46 aligns the common areas of the multiple point cloud data to be combined. Specifically, as an alignment condition for aligning the common areas of the two point cloud data on the source side (alignment source) and the target side (alignment destination), a transformation matrix for coordinate transformation of each point of the source side point cloud data is calculated, and the transformation matrix is ​​applied to the source side point cloud data to perform coordinate transformation of the source side point cloud data.

[0070] In the point cloud combining process P3, the processor 46 combines the point cloud data that have been aligned in the point cloud alignment process P2 to generate a single combined point cloud data that represents the entire target location.

[0071] In the drawing generation process P4, the processor 46 generates a 3D model representing the entire target location based on the point cloud data representing the entire target location generated by the 3D reconstruction process P1, and generates a layout drawing representing the entire target location based on the 3D model. At this time, a 2D layout drawing (plan view) representing the entire target location is generated by projecting the 3D model representing the entire target location onto a horizontal plane. In addition, a 3D layout drawing representing the entire target location is generated by projecting the 3D model representing the entire target location based on a predetermined line of sight.

[0072] In the display information generation process P5, the processor 46 generates display information for a screen to be displayed on the display 42 of the control device 2. The display 42 displays a screen on which the user performs operations such as managing the imaging data, issuing instructions for post-imaging processes (point cloud generation process P14, point cloud alignment process P2, point cloud joining process P3, and drawing generation process P4), and setting processing conditions. The processor 46 also generates display information for a screen to be displayed on the touch panel display 31 of the imaging device 1. The touch panel display 31 displays a screen to assist the user in the imaging operation.

[0073] In the display process P6, the processor 46 displays a screen on the display 42 of the control device 2 based on the display information generated in the display information generation process P5, and also displays a screen on the touch panel display 31 of the photographing device 1.

[0074] In this embodiment, the processing related to the merging of point cloud data is performed on a control device 2 that can be carried by a user, but the processing related to the merging of point cloud data may also be performed on another device (e.g., a server device) that can communicate with the control device 2.

[0075] Next, a description will be given of the procedure of the process performed by the control device 2. FIG.

[0076] First, the processor 46 acquires the point cloud data to be combined in response to a user's operation to specify a point cloud file in which the point cloud data to be combined is stored (ST101). The point cloud data generated by the 3D reconstruction process P1 is stored in the storage device 45.

[0077] Next, the processor 46 performs alignment to align the common areas of the multiple point cloud data to be combined (point cloud alignment process) (ST102). At this time, as an alignment condition for aligning the common areas of the two point cloud data on the source side and the target side, a transformation matrix for transforming the coordinates of each point of the source side point cloud data is calculated, and the transformation matrix is ​​applied to the source side point cloud data to perform coordinate transformation of the source side point cloud data.

[0078] Next, the processor 46 combines the point cloud data that have been aligned in the point cloud alignment process (ST102) to generate one combined point cloud data that represents the entire target location (point cloud combination process) (ST103). At this time, in addition to combining two point cloud data, combining one combined point cloud data with another point cloud data or combining one combined point cloud data with another combined point cloud data is performed as necessary.

[0079] Next, a description will be given of the procedure of the point cloud alignment process performed by the control device 2. Fig. 6 is a flowchart showing the procedure of the point cloud alignment process.

[0080] Since multiple point cloud data sets generated by taking multiple photographs of the target location cannot accurately recognize their relative positions, the control device 2 performs a process (point cloud alignment process) to align the common areas of the multiple point cloud data sets when combining the multiple point cloud data sets.

[0081] First, the processor 46 displays a point cloud image that visualizes each of the multiple point cloud data to be combined on the screen of the display 42, and in response to a user's operation to overlap the common parts of the two point cloud images, obtains information (preliminary alignment information) regarding the relative positional relationship between the two overlapping point cloud images (preliminary alignment process) (ST201).

[0082] Next, based on the preliminary alignment information acquired in the preliminary alignment process, the processor 46 extracts point clouds contained in the common area from each of the two point cloud data on the source side and the target side, and generates two extracted point cloud data on the source side and the target side (common area extraction process) (ST202).

[0083] Next, the processor 46 performs preprocessing (ST203) on the two extracted point cloud data on the source side and the target side extracted in the common area extraction process (ST202) to properly perform the next rough alignment process (ST204).

[0084] Next, the processor 46 determines the rough alignment conditions (transformation matrix) for roughly aligning the two extracted point cloud data on the source side and the target side, and applies the rough alignment conditions to the extracted point cloud data on the source side to perform coordinate transformation of the extracted point cloud data on the source side (rough alignment process) (ST204). By performing the rough alignment process in this way prior to the fine alignment process, it is possible to avoid the problem of getting stuck in a local solution in the fine alignment process and obtain appropriate fine alignment conditions.

[0085] Next, the processor 46 uses the extracted point cloud data that has been aligned using the rough alignment conditions (transformation matrix) acquired in the rough alignment process (ST204) to determine fine alignment conditions (transformation matrix) for precisely aligning the two point cloud data on the source side and the target side, and applies these fine alignment conditions to the point cloud data on the source side to perform coordinate transformation of the point cloud data on the source side (fine alignment process) (ST205).

[0086] Next, a description will be given of the preliminary alignment process performed by the control device 2. Fig. 7 is a flowchart showing the procedure of the preliminary alignment process.

[0087] In the preliminary alignment process (ST201 in FIG. 6), the processor 46 first generates a point cloud image that visualizes the point cloud data in a bird's-eye view (point cloud image generation process) (ST301). Specifically, the point cloud image is generated by projecting each point of the point cloud data onto a two-dimensional horizontal plane. At this time, multiple points in the point cloud data may be compressed into one point to reduce the size of the point cloud image.

[0088] Next, the processor 46 displays a canvas screen 61 (point cloud operation screen, see FIG. 9 ) on the display 42 (ST302). On the canvas screen 61, the user can perform an operation to superimpose the common area of ​​two point cloud images, which are visualized versions of two sets of point cloud data.

[0089] Next, the processor 46 acquires information (preliminary alignment information) regarding the relative positional relationship between the two overlapping point cloud images based on the user's operation on the canvas screen 61 (ST303).

[0090] Next, a description will be given of the menu screen 51 displayed on the display 42 of the control device 2. FIG.

[0091] The menu screen 51 has a "Generate point cloud" button 52, a "Display point cloud" button 53, a "Combine point clouds" button 54, a "Generate drawing" button 55, a "Display drawing" button 56, and a "List of shooting data" button 57.

[0092] When the user operates the "Generate Point Cloud" button 52, the point cloud generation process P14 is started. At this time, individual point cloud data is generated based on individual photographic data generated by partially photographing the target location. When the user operates the "Display Point Cloud" button 53, the screen transitions to one that displays the point cloud data generated in the point cloud generation process P14, and a point cloud image that visualizes the individual point cloud data is displayed.

[0093] Furthermore, when the user operates the "Point Cloud Combination" button 54, the point cloud alignment process P2 is initiated, and the screen transitions to a screen where the user can perform operations related to the combination of individual point cloud data, specifically the canvas screen 61 (see FIG. 9). Note that when an operation is performed on the canvas screen 61 to instruct the execution of the point cloud combination process P3, the point cloud combination process is initiated, and as a processing result, a combined point cloud image is displayed, which visualizes the combined point cloud data representing the entire target location.

[0094] Furthermore, when the user operates the "Drawing Generation" button 55, the drawing generation process P4 is started. At this time, a drawing representing the entire target location is generated based on the connection point cloud data representing the entire target location. When the user operates the "Drawing Display" button 56, the screen transitions to one displaying the drawing generated in the drawing generation process P4, and the drawing representing the entire target location is displayed.

[0095] Furthermore, when the user operates the "photography data list" button 57, the screen transitions to a screen displaying a list of individual photography data. On this screen, the user can check the individual photography data.

[0096] Next, the canvas screen 61 displayed on the display 42 of the control device 2 during the preliminary alignment process will be described. Fig. 9 is an explanatory diagram showing the canvas screen 61. Figs. 10, 11, and 12 are explanatory diagrams showing the operation status of the point cloud image 71 and the combined point cloud image 74 on the canvas 72.

[0097] During the preliminary alignment process (see FIG. 7), a canvas screen 61 (point cloud operation screen) shown in FIG.

[0098] The canvas screen 61 is provided with a canvas window 62, an operation window 63, and a selected image window 64.

[0099] The canvas screen 61 also has a "Load point cloud file" button 65. When the user operates the "Load point cloud file" button 65, a screen (not shown) for specifying a point cloud file (a file or folder in which point cloud data is stored) is displayed. When the user performs an operation to specify a point cloud file, the point cloud data stored in the specified point cloud file is input, and a point cloud image 71 that visualizes the point cloud data is displayed in the canvas window 62.

[0100] At this time, by inputting multiple pieces of point cloud data to be combined, multiple point cloud images 71, which visualize the multiple pieces of point cloud data to be combined, are displayed side by side on the canvas 72 in the canvas window 62. The point cloud images 71 are visualized as a bird's-eye view of the point cloud data. In the example shown in Fig. 9, point cloud images 71 #1 to #4, which visualize four pieces of point cloud data #1 to #4 to be combined, are displayed.

[0101] Furthermore, in the canvas window 62, the size and relative positional relationship of the point cloud image 71 with respect to the canvas 72 are maintained constant. The point cloud image 71 is displayed on the canvas 72 at a size proportional to the actual size of the corresponding point cloud data. Note that in the initial state when point cloud data is input, multiple point cloud images 71 may be arranged on the canvas 72 so as to correspond to the actual positional relationship of the original point cloud data.

[0102] The selected image window 64 displays thumbnails 73 of the point cloud images 71 that are selected in the canvas window 62. In the example shown in Fig. 9, the point cloud images 71 #1 and #2 are selected. Note that in the canvas window 62, a mark indicating the selected state may be displayed on the selected point cloud images 71.

[0103] Note that the example shown in FIG. 9 is for a house, but when the target location is large, such as a factory, photography may be taken multiple times (for example, 10 times), resulting in a large number of point cloud data (for example, 10 pieces) to be combined. In this case, the canvas window 62 displays a large number of point cloud images 71 lined up on the canvas 72.

[0104] Furthermore, in the canvas window 62, the user performs an alignment operation to adjust the position and angle of the point cloud image 71 so that the common areas of the two point cloud images 71 overlap each other on the canvas 72. At this time, the user can perform the alignment operation by translating the point cloud image 71 and by rotating the point cloud image 71. For example, the point cloud image 71 is translated by grabbing and moving the point cloud image 71 (e.g., a drag operation). Furthermore, the point cloud image 71 is rotated by operating a rotation mark (not shown) that is displayed when the point cloud image 71 is selected (e.g., a click operation).

[0105] The operation window 63 is provided with a "Merge" button 81 as an operation unit for instructing the execution of the point cloud merging process. The "Merge" button 81 becomes operable when two point cloud images 71 overlap in response to a user operation in the canvas window 62. Note that the "Merge" button 81 may appear when the point cloud images 71 overlap.

[0106] When the user operates the "Execute Merge" button 81, the point cloud alignment process (ST102 in FIG. 5) performs the preliminary alignment process (ST201 in FIG. 6) followed by the common area extraction process, pre-processing, rough alignment process, and fine alignment process (ST202 to ST205 in FIG. 6), as well as the point cloud merging process (ST103 in FIG. 5), and a merged point cloud image that visualizes the merged point cloud data as a result of the process is displayed on the canvas screen 61. This allows the user to visually check whether the point cloud data has been merged appropriately.

[0107] In the examples shown in Figures 10, 11, and 12, starting from the initial state shown in Figure 10(A), the user first manipulates the point cloud images 71 of #1 and #2 on the canvas 72 to overlap the point cloud images 71 of #1 and #2 so that their common areas align, as shown in Figure 10(B). At this time, rotation and translation operations are performed as necessary. Next, when the user operates the "Execute Merge" button 81, the point cloud merging process is executed, and the merged point cloud image 74 of #S1 is displayed as the processing result, as shown in Figure 11(A).

[0108] 11(A), the user operates the point cloud images 71 of #3 and #4 to overlap them so that their common areas are aligned. At this time, rotation and translation operations are performed as necessary. Next, when the user operates the "Execute Merge" button 81, the point cloud merging process is executed, and the merged point cloud image 74 of #S2 is displayed as the processing result, as shown in FIG. 11(B).

[0109] Next, as shown in Figure 12(A), the user operates the connection point cloud images 74 of #S1 and #S2 to overlap the connection point cloud images 74 of #S1 and #S2 so that their common areas match. At this time, rotation and translation operations are performed as necessary. Next, when the user operates the "Execute Merge" button 81, the point cloud merging process is executed, and the connection point cloud image of #S3 is displayed as the processing result, as shown in Figure 12(B).

[0110] In this embodiment, the point cloud merging process for merging the two point cloud data is executed by simultaneously selecting two point cloud images 71 with overlapping common areas and then operating the "Execute Merge" button 81, but it is also possible to simultaneously select three or more point cloud images 71 with overlapping common areas and then operate the "Execute Merge" button 81. In this case, merging of two point cloud data, merging of merged point cloud data with point cloud data, or merging of two merged point cloud data is executed in an appropriate combination.

[0111] The canvas screen 61 also has a "Save connection point cloud file" button 66 as an operation unit with which the user approves the connection results and instructs saving of the data. When the user operates the "Save connection point cloud file" button 66, a connection point cloud file storing the connection point cloud data is generated and accumulated in the storage device 45. When the user operates the "Execute connection" button 81, a simple point cloud connection process is executed, and a connection point cloud image for the user to check the processing results is displayed on the canvas screen 61, while when the user operates the "Save connection point cloud file" button 66, a regular point cloud connection process may be executed.

[0112] In this embodiment, the preliminary alignment process is performed based on a user operation, but the preliminary alignment process may also be performed without a user operation. Specifically, the processor 46 may perform a process such as image matching on the two point cloud images to detect similar areas in the two point cloud images as common areas (common area detection process). Furthermore, this common area detection process may detect similar common areas in three or more point cloud images.

[0113] Next, a description will be given of the image opacity adjustment function of the canvas screen 61. Fig. 13 is an explanatory diagram showing a situation in which the opacity of the point cloud image 71 is adjusted on the canvas screen 61.

[0114] The operation window 63 of the canvas screen 61 is provided with a slider 83 as an operation unit for specifying the opacity of the point cloud image 71. When the user operates the slider 83 after performing an operation to select the point cloud image 71, the opacity of the selected point cloud image 71 changes.

[0115] When the opacity is 100%, the point cloud image 71 is displayed in an opaque state. When the opacity is less than 100%, the point cloud image 71 is displayed in a semi-transparent state. Therefore, when the opacity of the upper point cloud image 71 of two overlapping point cloud images 71 is set to less than 100%, the lower point cloud image 71 is visible through the overlapping portion of the two point cloud images 71, and at the same time, the upper point cloud image 71 is also visible.

[0116] Furthermore, when two point cloud images 71 overlap vertically, the opacity may be set only for the upper point cloud image 71, or the same opacity may be set for both overlapping point cloud images 71. In the example shown in Fig. 13(A), two point cloud images 71, #1 and #2, which overlap vertically, are both selected and their opacity is specified, and as shown in Fig. 13(B), the two point cloud images 71, #1 and #2, are displayed semi-transparently with the same opacity. This allows the user to easily check whether the two point cloud images 71 are overlapped with their common areas properly aligned.

[0117] When the opacity is specified by operating the slider 83, the point cloud image 71 is displayed in a semi-transparent state according to the specified opacity, but the point cloud image 71 is usually displayed in a non-transparent state, and may be displayed in a semi-transparent state only while the point cloud image 71 is being operated.

[0118] Next, a description will be given of the canvas zoom function of the canvas screen 61. Fig. 14 is an explanatory diagram showing a state in which the canvas 72 is zoomed in or out on the canvas screen 61.

[0119] The canvas screen 61 is provided with an enlargement button 84 and a reduction button 85 as operation units for instructing the enlargement or reduction of the canvas 72. When the user operates the enlargement button 84, the canvas 72 gradually enlarges. At this time, the canvas 72 transitions from a standard state (see FIG. 9 ) in which the entire canvas 72 is displayed in the display area of ​​the canvas screen 61 to an enlarged state (see FIG. 14 ) in which only a portion of the canvas 72 fits within the display area. On the other hand, when the user operates the reduction button 85 while the canvas 72 is in the enlarged state, the canvas 72 gradually reduces and returns to the standard state.

[0120] In this way, when the user operates the zoom in button 84 and the zoom out button 85, the canvas 72 is zoomed in or out, and the point cloud image 71 on the canvas 72 is accordingly zoomed in or out. This allows the user to easily visually check whether the common areas of the point cloud images 71 are aligned with sufficient accuracy. Note that the operation unit for instructing the zoom in or out of the canvas 72 is not limited to the zoom in button 84 and the zoom out button 85, and may be, for example, a slider or a numerical input of the zoom ratio.

[0121] Here, when the user operates the enlarge button 84 to enlarge the canvas 72, only a portion of the canvas 72 fits within the display area. At this time, by selecting only the point cloud image 71 to be aligned, the canvas 72 is enlarged with the selected point cloud image 71 at its center. This prevents the point cloud image 71 to be aligned from being outside the display area. In the example shown in FIG. 14 , two point cloud images 71, #1 and #2, are selected, so the canvas 72 is enlarged with the midpoint between the two point cloud images 71 at its center. At this time, only a portion of the point cloud image 71 near the selected point cloud image 71 is displayed, and the point cloud image 71 farther from the selected point cloud image 71 is outside the display area.

[0122] Furthermore, on the canvas screen 61, the canvas 72 is enlarged or reduced by changing the distance of the viewpoint from the canvas 72 while maintaining the size and relative positional relationship of the point cloud image 71 with respect to the canvas 72 constant. That is, when the user operates the enlarge button 84, the viewpoint moves closer to the canvas 72, causing the point cloud image 71 on the canvas 72 to be enlarged (zoomed in). Also, when the user operates the reduce button 85, the viewpoint moves away from the canvas 72, causing the point cloud image 71 on the canvas 72 to be reduced (zoomed out). In this way, maintaining the size and relative positional relationship of the point cloud image 71 with respect to the canvas 72 constant reduces the load of the display processing on the canvas screen 61. That is, when a large number of point cloud images 71 are arranged on the canvas 72, the display processing of changing the position and angle of the point cloud images 71 in response to user operations becomes easier.

[0123] Next, a description will be given of the merge cancel function of the canvas screen 61. Fig. 15 is an explanatory diagram showing the operation status of merge cancel on the canvas screen 61.

[0124] The canvas screen 61 is provided with an "Undo Merge" button 82 as an operation unit for instructing restoration of the state before merging. When the user operates the "Undo Merge" button 82 after performing an operation to select a merged point cloud image 74, the point cloud merging process for the selected merged point cloud image 74 is canceled and the state is restored to its original state. This allows the user to restore the state before merging with a simple operation if they erroneously operate the "Execute Merge" button 81 or if an undesirable merged result is obtained.

[0125] In the example shown in Fig. 15(A), one connection point cloud image 74 is selected. A thumbnail 73 of the selected connection point cloud image 74 is displayed in the selected image window 64. In the example shown in Fig. 15(B), the point cloud combination process for the connection point cloud image 74 selected in the example shown in Fig. 15(A) is canceled, and the state is restored to the state before combination, that is, the state in which the alignment operation has been performed and the two point cloud images 71 are superimposed. Note that when the connection point cloud image 74 is returned to the point cloud image 71 before combination, the point cloud image 71 is no longer selected, and the thumbnail 73 of the point cloud image 71 is no longer displayed in the selected image window 64.

[0126] Next, a description will be given of the image frame display function of the canvas screen 61. Fig. 16 is an explanatory diagram showing a situation in which a frame image 78 is displayed on the canvas screen 61 around the point cloud image 71.

[0127] The canvas screen 61 is provided with an "Image frame display / hide" button 86 as an operation unit for instructing the display of a frame image 78 representing the periphery of the point cloud image 71. When the user operates the "Image frame display / hide" button 86 after performing an operation to select the point cloud image 71, the frame image 78 is displayed on the periphery of the selected point cloud image 71.

[0128] The example shown in Fig. 16(A) is the initial state (non-display state), and by operating the "Image Frame Display / Hide" button 86, the state transitions to a state (display state) in which a frame image 78 is displayed, as shown in Fig. 16(B). This makes it easier to distinguish the overlapping areas of the selected point cloud images 71 using the frame image 78, allowing the user to easily check whether the common areas of the point cloud data are appropriately overlapping. Note that in the example shown in Fig. 16, multiple point cloud images 71 are selected, and thumbnails 73 of the selected point cloud images 71 are displayed in the selected image window 64.

[0129] Furthermore, when the user operates the "Image Frame Display / Hide" button 86 while the frame image 78 is displayed, the display returns to the original state (non-display state) in which the frame image 78 is not displayed. In this way, when the user operates the "Image Frame Display / Hide" button 86, it is possible to switch between a display state in which the frame image 78 is displayed and a non-display state in which the frame image 78 is not displayed.

[0130] Next, a description will be given of the image lock function of the canvas screen 61. Fig. 17 is an explanatory diagram showing a situation in which the point cloud image 71 is set to a locked state on the canvas screen 61.

[0131] The canvas screen 61 is provided with an "Image Lock" button 87 as an operation unit for instructing to lock the point cloud image 71. When the user operates the "Image Lock" button 87 after performing an operation to select a point cloud image 71, the selected point cloud image 71 is set to a locked state (a fixed state in which operations are not possible). The user cannot operate a locked point cloud image 71, and can only operate an unlocked point cloud image 71. Furthermore, on the canvas screen 61, when a point cloud image 71 is set to a locked state, a lock mark 79 indicating the locked state is displayed on the corresponding point cloud image 71.

[0132] In the example shown in FIG. 17(A), point cloud image 71 #1 is selected, and a thumbnail 73 of the selected point cloud image #1 is displayed in the selected image window 64. When the user operates the "Image Lock" button 87, the selected point cloud image #1 is set to a locked state, and a lock mark 79 is displayed, as shown in FIG. 17(B). Point cloud image #2 is in an operable state, and is operated so that the common area overlaps with point cloud image #1 71. At this time, point cloud image #1 71 that has been set to a locked state returns to an unselected state, but because point cloud image #2 71 is in a selected state, a thumbnail 73 of point cloud image #2 71 is displayed in the selected image window 64.

[0133] By locking the point cloud image 71 in this way, it is possible to prevent the point cloud image 71 that is not being operated from being moved by an erroneous operation by the user. Furthermore, the lock mark 79 allows the user to easily understand the locked state of the point cloud image 71. Note that, in the example shown in Fig. 17, the lock mark 79 is drawn in a circular shape in a predetermined color (e.g., green), but the shape of the lock mark 79 is not limited to this.

[0134] In the example shown in Figure 17, the point cloud image 71 corresponding to the source point cloud data is set to a locked state, but the combined point cloud image 74 (see Figures 11 and 12) corresponding to the combined point cloud data generated by the point cloud combining process can also be set to a locked state by a similar operation.

[0135] One or more point cloud images 71 may be set to the locked state at the same time. For example, during an operation to overlap the common areas of two point cloud images 71, only one point cloud image 71 on the target side (the point cloud image 71 to be aligned) may be set to the locked state. In addition, in order to prevent erroneous operation of the point cloud images 71, all of the other point cloud images 71 except for the point cloud image 71 on the source side (the point cloud image to be aligned) may be set to the locked state.

[0136] Next, a description will be given of the combined history screen 101 displayed on the display 42 of the control device 2. FIG.

[0137] The canvas screen 61 (see FIG. 9 ) is provided with a "Merge History" button 88 as an operation unit for instructing the display of merge history information that shows the history of merging of multiple point cloud data. When the user operates the "Merge History" button 88 after performing an operation to select a merge point cloud image 74 (see FIG. 10(B)), the screen transitions to a merge history screen 101 shown in FIG. 18 . Note that, on the merge history screen 101, when the user performs an operation to select a merge point cloud image 74, a thumbnail 73 of the selected merge point cloud image 74 is displayed in the selected image window 64.

[0138] The merge history screen 101 displays a system diagram 102 (tree diagram) as merge history information that shows the history of merging multiple point cloud data. By viewing the system diagram 102, the user can easily check the merge status of multiple point cloud data, i.e., the combinations in which the point cloud data were merged. In particular, even in cases where the target location is large, such as a factory, and there are many point cloud data to be merged, the user can easily check the merge status of the point cloud data.

[0139] 18, the topmost displayed connection point cloud image 74 corresponds to the latest connection point cloud data, and the bottommost displayed point cloud image 71 corresponds to the point cloud data from which the connection was made. The user can trace back the connection status of the point cloud data by tracing downward on the system diagram 102.

[0140] Furthermore, the merge history screen 101 displays, for each merge point cloud image 74, a registration match rate (evaluation information) of the common area, which indicates the degree of appropriateness of the registration using the transformation matrix estimated at the time of merging in the merge point cloud data corresponding to that merge point cloud image 74. This allows the user to easily check the degree of appropriateness of the registration using the transformation matrix estimated at the time of merging in the merge point cloud data. For example, if the registration match rate of the common area is low, the user can confirm that the accuracy of the registration in the merge point cloud data is low, that is, the operation of merging the common area of ​​the merge source point cloud image 71 may be insufficient.

[0141] Here, the registration match rate of the common area is expressed, for example, as the match rate (%) of the common area based on the transformation matrix for registration estimated when the two point cloud data to be combined are combined. The match rate is calculated based on the distance between corresponding points in the two point cloud data. Specifically, for each point in the source point cloud after alignment, the nearest point in the target point cloud is determined to be the corresponding point. If the distance between the corresponding points is equal to or less than a threshold value (e.g., 3 cm), the matching is determined as acceptable. If the distance between the corresponding points is greater than the threshold value, the matching is determined as unacceptable. The matching rate is then calculated as the ratio of the number of corresponding points determined to be acceptable to the total number of detected corresponding points. For example, if 50 pairs of corresponding points out of 100 pairs are acceptable, the matching rate is 50%. A low match rate indicates low registration accuracy in the combined point cloud data.

[0142] The merge history screen 101 also has a "cancel merge" button 103. When the user operates the "cancel merge" button 103 after selecting a merged point cloud image 74, the point cloud merge process related to the selected merged point cloud image 74 is canceled, and the state is restored to the state before the point cloud merge process was executed.

[0143] At this time, the point cloud combining process that generated the connecting point cloud data corresponding to the selected connecting point cloud image 74 is canceled. Also, the point cloud combining process based on the connecting point cloud data corresponding to the selected connecting point cloud image 74 is canceled. In other words, the point cloud combining process from the connecting point cloud data corresponding to the selected connecting point cloud image 74 to the connecting point cloud data corresponding to the latest connecting point cloud image 74 is canceled.

[0144] 18, the connection point cloud image 74 of #S1 is selected. In this case, the connection point cloud data corresponding to the connection point cloud image 74 of #1 is restored to the point cloud data corresponding to the original point cloud images 71 of #1 and #2. In addition, the connection point cloud data corresponding to the connection point cloud image 74 of #S3, which was generated based on the connection point cloud data corresponding to the connection point cloud image 74 of #S1, is restored to its original state.

[0145] (Modification of the First Embodiment) Next, a modification of the first embodiment will be described. Note that points not specifically mentioned here are the same as those of the above-described embodiment. Figure 19 is an explanatory diagram showing a main canvas screen 111 and a sub-canvas screen 112 displayed on the display 42 of the control device 2 according to a modification of the first embodiment.

[0146] Similar to the canvas screen 61 (see FIG. 9) in the first embodiment, the main canvas screen 111 displays a plurality of point cloud images 71, which are visualizations of the plurality of point cloud data to be combined, side by side. The main canvas screen 111 also has a "sub-canvas" button 68.

[0147] When the user operates the "sub-canvas" button 68 after selecting the point cloud image 71 on the main canvas screen 111, the sub-canvas screen 112 pops up on the main canvas screen 111. Note that the main canvas screen 111 and the sub-canvas screen 112 may be displayed in a switchable manner.

[0148] Only the point cloud images 71 selected by the user on the main canvas screen 111 are displayed side by side on the sub-canvas screen 112. This allows the user to perform the alignment operation by displaying only the point cloud images 71 currently being aligned among the point cloud images 71 displayed on the main canvas screen 111 on the sub-canvas screen 112. This makes it easier for the user to perform the alignment operation by overlapping the common areas of the two point cloud images 71 to be combined.

[0149] Note that the sub-canvas screen 112 may be provided with a "main canvas" button for returning to the main canvas screen 111. Also, the user may operate a button to close the sub-canvas screen 112 to return to the main canvas screen 111.

[0150] Second Embodiment Next, a second embodiment will be described. It should be noted that the points not specifically mentioned here are the same as those of the above-described embodiment. Fig. 20 is an explanatory diagram showing a canvas screen 61 displayed on the display 42 of the control device 2 according to the second embodiment.

[0151] In the first embodiment, a preliminary alignment process is performed based on a user's operation, that is, a preliminary alignment process based on a user's operation to overlap a common area of ​​two point cloud data. On the other hand, in this embodiment, in addition to the preliminary alignment process based on a user's operation, a preliminary alignment process that is not based on a user's operation can also be performed. Note that the rough alignment process and fine alignment process performed after the preliminary alignment process are the same as those in the first embodiment (see FIG. 6 ).

[0152] In the preliminary alignment process not based on user operation, a process of detecting a common area between two point cloud data (common area detection process) is performed. In the common area detection process, image matching is performed between two point cloud images 71 that visualize point cloud data to measure the similarity between the point cloud images 71, and a common area between the two point cloud images 71 is detected based on the measurement result. Specifically, an area with high similarity between the two point cloud images 71 is detected as a common area. At this time, for example, an area between the point cloud images 71 whose similarity is equal to or greater than a predetermined threshold is detected as a common area. Furthermore, an area that is included in a predetermined percentage of the most similar point cloud images 71 is detected as a common area.

[0153] In this embodiment, a preliminary alignment process not based on a user's operation is first performed, and if the process does not produce an appropriate result, a preliminary alignment process based on a user's operation is performed again. That is, the user first performs an operation to instruct the execution of a preliminary alignment process not based on a user's operation, checks the processing result, and if the processing result is not appropriate, the point cloud combining process is restored and then the preliminary alignment process based on the user's operation is performed.

[0154] 20, in addition to a "Merge" button 81 and a "Cancel Merge" button 82, an "Auto Merge" button 121 is provided on the canvas screen 61. When the user performs an operation (e.g., a click operation) to select two point cloud images 71 to be merged and then operates the "Auto Merge" button 121, a preliminary alignment process not based on a user operation and a point cloud merge process based on the results of that process are executed, and a merged point cloud image 74 (see FIGS. 11 and 12) is displayed on the canvas screen 61 as the processing result.

[0155] The user visually checks the combined point cloud image 74 to determine whether the processing results are appropriate, and if the processing results are not appropriate, operates the "Cancel Combine" button 82. Next, after performing an operation to overlap the common areas of the two point cloud images 71 to be combined, the user operates the "Execute Combine" button 81. This executes a preliminary alignment process based on the user's operation and a point cloud combination process based on the processing results, and the combined point cloud image 74 as the processing result is displayed on the canvas screen 61.

[0156] Here, the result of the preliminary alignment process not based on user operation may be appropriate in some cases, but the result of the process may be inappropriate in other cases. For this reason, on the canvas screen 61, a connection point cloud image 74 generated by the preliminary alignment process not based on user operation and a connection point cloud image 74 generated by the preliminary alignment process based on user operation may coexist.

[0157] As described above, the embodiments have been described as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited to these, and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made. Furthermore, it is also possible to combine the components described in the above embodiments to create new embodiments.

[0158] The point cloud combining device and point cloud combining method disclosed herein have the effect of improving user operability when aligning multiple point cloud data obtained by 3D reconstruction processing of a target location when combining them, thereby reducing the burden on the user when there is a large amount of point cloud data to be combined, and are useful as a point cloud combining device and point cloud combining method that aligns two point cloud data obtained by 3D reconstruction processing of a target location and then combines the two point cloud data.

[0159] 1: Camera device 2: Control device (point cloud combining device) 42: Display 46: Processor 61: Canvas screen 71: Point cloud image 72: Canvas 73: Thumbnail 74: Combined point cloud image 78: Frame image 79: Lock mark 81: "Execute combine" button 82: "Cancel combine" button 83: Slider 84: Zoom in button 85: Zoom out button 86: "Show / hide image frame" button 87: "Image lock" button 88: "Combine history" button 101: Combine history screen 102: System diagram

Claims

1. A point cloud merging device that uses a processor to execute a process of merging two point cloud data obtained by a three-dimensional reconstruction process of a target location after aligning them with each other, wherein the processor: displays on a display device a screen including a canvas on which multiple point cloud images that visualize the multiple point cloud data to be merged are arranged so that the user can operate them, and an operation unit that instructs the execution of the point cloud merging process; executes the point cloud merging process to merge the two point cloud data in accordance with the user's operation of aligning the common areas of the two point cloud images on the canvas and the user's operation on the operation unit; and displays on the display device a screen including a combined point cloud image that visualizes the combined point cloud data generated by the point cloud merging process.

2. The point cloud combining device according to claim 1, characterized in that the processor displays on the canvas the combined point cloud image, which visualizes the combined point cloud data generated by combining two pieces of point cloud data, and combines the combined point cloud data and the point cloud data in response to a user operation to superimpose the common areas of the combined point cloud image and the point cloud image.

3. The point cloud combining device of claim 1, characterized in that the processor displays a screen including an operation unit for instructing restoration of the state before combination, and in response to a user's operation of the operation unit, returns the combined point cloud data to the state before combination and displays the point cloud image corresponding to the point cloud data before combination.

4. The point cloud combining device described in claim 1, characterized in that the processor displays a screen including an operation unit for specifying the opacity of the point cloud image, and displays the point cloud image in a semi-transparent state with the specified opacity in response to the user's operation of the operation unit.

5. The point cloud combining device according to claim 1, characterized in that the processor displays a screen including an operation unit for instructing the enlargement or reduction of the canvas, and enlarges or reduces the canvas in response to the user's operation of the operation unit.

6. The point cloud combining device described in claim 5, characterized in that the processor enlarges or reduces the canvas by changing the distance from the viewpoint to the canvas while keeping the position and size of the point cloud image relative to the canvas constant.

7. The point cloud combining device according to claim 1, characterized in that the processor displays a screen including an operation unit for instructing the display of a frame image representing the outer periphery of the point cloud image, and switches between a display state in which the frame image is displayed and a non-display state in which the frame image is not displayed in accordance with the user's operation of the operation unit.

8. The point cloud combining device according to claim 1, characterized in that the processor displays a screen including an operation unit for instructing locking of the point cloud image, and sets the point cloud image to a locked state in response to operation of the operation unit by the user.

9. The point cloud merging device described in claim 1, characterized in that the processor displays a screen including an operation unit that instructs the display of merging history information that represents the history of merging of the two point cloud data, and displays a screen including the merging history information on the display device in response to a user's operation of the operation unit.

10. The point cloud combining device described in claim 9, characterized in that the processor displays a screen including the combining history information and evaluation information representing the degree of appropriateness of alignment using the transformation matrix estimated at the time of combining in the combined point cloud data generated by the point cloud combining process.

11. A point cloud merging method that causes a processor to perform a process of merging two point cloud data obtained by a three-dimensional reconstruction process of a target location after mutual alignment, the method comprising: displaying on a display device a screen including a canvas on which multiple point cloud images that visualize the multiple point cloud data to be merged are arranged so that the user can operate, and an operation unit that instructs the execution of the point cloud merging process; executing the point cloud merging process to merge the two point cloud data in accordance with the user's operation of aligning the common areas of the two point cloud images on the canvas and the user's operation on the operation unit; and displaying on the display device a screen including a merged point cloud image that visualizes the merged point cloud data generated by the point cloud merging process.

Images