Point cloud combining device and point cloud combining method

Abstract

In the alignment when combining two point cloud data obtained by three-dimensional restoration processing of a target location, improve the operability of the user and reduce the burden on the user when there are a large number of point cloud data to be combined. 【Solution means】A canvas screen 61 including two point cloud images 71 visualizing two point cloud data to be combined, a canvas 72 arranged so that it can be operated by the user, and a button 81 for instructing the execution of point cloud combination processing is displayed on the display. According to the alignment operation by the user of overlapping the common areas of the two point cloud images on the canvas and the operation of the button by the user, a point cloud combination process for combining the two point cloud data is executed, and a screen including a combined point cloud image visualizing the point cloud data generated by the point cloud combination process is displayed.

Description

Technical Field

[0001] The present invention relates to a point cloud combining device and a point cloud combining method for combining two pieces of point cloud data obtained by three-dimensional restoration processing of a target location after performing mutual alignment.

Background Art

[0002] A three-dimensional restoration technique for generating point cloud data as three-dimensional spatial information regarding a target location based on a captured image of the target location is known. In this three-dimensional restoration technique, in recent years, the SLAM (Simultaneous Localization And Mapping) method has attracted attention. In the SLAM method, point cloud data of a target location can be acquired by causing a moving body (for example, an operator) to hold a photographing device and photographing the target location while moving.

[0003] On the other hand, when the target location is large like a factory or when the target location is divided in a complicated manner like a house, the entire target location cannot be photographed at once. In this case, the user performs photographing of the target location in multiple times. Further, a process of combining a plurality of pieces of point cloud data generated by partially photographing the target location to generate combined point cloud data representing the entire target location is performed. At this time, since the positional relationship between the plurality of pieces of point cloud data cannot be accurately recognized, when combining the plurality of pieces of point cloud data, alignment for aligning a common area in which the same subject of the plurality of pieces of point cloud data appears is required.

[0004] As a technique regarding alignment of such a plurality of pieces of point cloud data, conventionally, a technique is known in which a point cloud image obtained by visualizing point cloud data is displayed on a screen, and the user operates the point cloud image so that a common area of two point cloud screens overlaps (see Patent Document 1). In this technique, by arranging a plurality of markers (targets) serving as alignment references in the target location, the markers are displayed on the point cloud image, and the user can perform an operation for alignment based on the markers.

Prior Art Documents

Patent Document

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, in the conventional technology, it is necessary to previously arrange a plurality of markers (targets) serving as alignment references at the target location and measure the installation positions of the markers using a surveying instrument (such as a total station), which is very time-consuming. For this reason, a technology that allows a user to perform an alignment operation without using a marker as a reference is desired.

[0007] Also, when the target location is large, such as in a factory, imaging may be performed in multiple times (for example, 10 times), and the number of point cloud data to be combined may be large (for example, 10). In this case, there is a problem that the work for combining the point cloud data is very time-consuming. For this reason, a technology that can reduce the burden on the user when there is a large number of point cloud data to be combined is desired.

[0008] Therefore, the main object of the present invention is to provide a point cloud combining device and a point cloud combining method that can improve the operability of the user and reduce the burden on the user when there is a large number of point cloud data to be combined in the alignment when combining a plurality of point cloud data obtained by three-dimensional restoration processing of a target location.

Means for Solving the Problems

[0009] The point cloud combining device of the present invention is a point cloud combining device that executes, by a processor, a process of combining two point cloud data obtained by three-dimensional restoration processing of a target location after performing mutual alignment, wherein the processor includes a canvas on which a plurality of point cloud images visualizing the plurality of point cloud data to be combined are arranged so as to be operable by a user, and an instruction to execute point cloud combining processing First operation unit And a screen including the above is displayed on a display device, and a user alignment operation of overlapping common regions of the two point cloud images on the canvas, and a user The first operation unit In response to the operation of, the point cloud combining process of combining the two point cloud data is executed, and a screen including a combined point cloud image visualizing the combined point cloud data generated by the point cloud combining process is displayed on the display device Then, a screen including a second operation unit for instructing restoration of the state before combination is further displayed, and in response to an operation of the second operation unit by the user, the combined point cloud data is returned to the state before combination, and the point cloud image corresponding to the point cloud data before the combination is displayed It is configured as follows.

[0010] Further, the point cloud combining method of the present invention is a point cloud combining method in which a processor performs a process of combining two point cloud data obtained by three-dimensional restoration processing of a target location after performing mutual alignment, wherein a plurality of point cloud images visualizing the plurality of point cloud data to be combined are arranged on a canvas so as to be operable by a user, and an instruction to execute point cloud combining processing First operation unit And a screen including the above is displayed on a display device, and a user alignment operation of overlapping common regions of the two point cloud images on the canvas, and a user The first operation unit In response to the operation of, the point cloud combining process of combining the two point cloud data is executed, and a screen including a combined point cloud image visualizing the combined point cloud data generated by the point cloud combining process is displayed on the display device Then, a screen including a second operation unit for instructing restoration of the state before combination is further displayed, and in response to an operation of the second operation unit by the user, the combined point cloud data is returned to the state before combination, and the point cloud image corresponding to the point cloud data before the combination is displayed It is configured as follows.

Advantages of the Invention

[0011] According to the present invention, a plurality of point cloud images visualizing a plurality of point cloud data to be combined are arranged on a canvas, and a user can appropriately select the point cloud images on the canvas and perform an alignment operation It is possible to return the point cloud data to the state before combination when the user makes a wrong operation or when an undesirable combination result is obtained Thereby, the operability of the user can be improved, and the burden on the user when there are a large number of point cloud data to be combined can be reduced.

Brief Description of Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Mode for Carrying Out the Invention

[0013] A first invention made to solve the above problems is a point cloud combining device that executes, by a processor, a process of combining two point cloud data obtained by three - dimensional restoration processing of a target location after performing mutual alignment, wherein the processor visualizes a plurality of the point cloud data to be combined as a plurality of point cloud images, a canvas on which operations by a user are possible, and a screen including an instruction to execute point cloud combining processing is displayed on a display device, an alignment operation by the user of overlapping common regions of the two point cloud images on the canvas, and according to an operation by the user First operation unit and, the point cloud combining process of combining the two point cloud data is executed, and a screen including a combined point cloud image visualizing the combined point cloud data generated by the point cloud combining process is displayed on the display device The first operation unit configured. Then, a screen including a second operation unit for instructing restoration of the state before combination is further displayed, and in response to an operation of the second operation unit by the user, the combined point cloud data is returned to the state before combination, and the point cloud image corresponding to the point cloud data before the combination is displayed According to this, a plurality of point cloud images visualizing a plurality of point cloud data to be combined are arranged on a canvas, and the user can appropriately select the point cloud images on the canvas and perform an alignment operation

[0014] Thereby, the operability of the user is improved, and the burden on the user when there are a large number of point cloud data to be combined can be reduced. Note that the point cloud combining process includes, in addition to the combination of two point cloud data as the combination sources, the combination of the point cloud data of the combination source and the combined point cloud data, and the combination of two combined point cloud data. It is possible to return the point cloud data to the state before combination when the user makes a wrong operation or when an undesirable combination result is obtained Thereby, the operability of the user is improved, and the burden on the user when there are a large number of point cloud data to be combined can be reduced. In addition to the combination of two point cloud data as the combination sources, the point cloud combining process includes the combination of the point cloud data of the combination source and the combined point cloud data, and the combination of two combined point cloud data.

[0015] Further, in the second invention, the processor displays the combined point cloud image obtained by visualizing the combined point cloud data generated by combining the two point cloud data on the canvas, and combines the combined point cloud data and the point cloud data according to an operation of a user who overlaps a common area of the combined point cloud image and the point cloud image.

[0016] According to this, it is possible to sequentially combine two point cloud data out of three or more point cloud data to obtain point cloud data in which three or more point cloud data are combined. In addition to the combination of two point cloud data and the combination of the combined point cloud data and the point cloud data, the combination of two combined point cloud data is also possible.

[0019] Also, Third the invention is such that the processor displays a screen including specifying the opacity of the point cloud image, and displays the point cloud image in a semi-transparent state with the specified opacity according to an operation by the user. Third operation unit including The third operation unit According to this, since the point cloud image is displayed in a semi-transparent state, the user can easily confirm the alignment state of the common area when the point cloud images are overlapped. The operation unit may be a slider provided on the screen.

[0020] Also,

[0021] the invention is such that the processor displays a screen including instructing the enlargement / reduction of the canvas, and performs the enlargement / reduction of the canvas according to an operation by the user. Fourth including Fourth operation unit instructing The fourth operation unit According to this, since the point cloud image on the canvas is enlarged / reduced, the user can easily confirm the alignment state of the common area when the point cloud images are overlapped. The operation unit may be a button provided on the screen or a slider.

[0022] Also,

[0023] the invention is such that the processor displays a screen including instructing the enlargement / reduction of the canvas, and performs the enlargement / reduction of the canvas according to an operation by the user. FifthThe invention is configured such that the processor enlarges or reduces the canvas by changing the separation distance of the viewpoint with respect to the canvas while keeping the position and size of the point cloud image with respect to the canvas constant.

[0024] According to this, by operating to enlarge or reduce the canvas, it becomes possible to enlarge or reduce the point cloud image on the canvas. Also, in a state where a large number of point cloud images are arranged on the canvas, it becomes easier to perform display processing for changing the position and angle of those point cloud images according to the user's operation. In this case, when the user performs an operation of selecting a plurality of point cloud images, the canvas may be enlarged or reduced centered on the midpoint of the selected plurality of point cloud images.

[0025] Also, Sixth The invention is such that the processor displays a screen including an instruction to display a frame image representing the outer periphery of the point cloud image, and switches between a display state in which the frame image is displayed and a non-display state in which the frame image is not displayed according to an operation by the user. Fifth operation unit and The fifth operation unit of

[0026] According to this, the user can easily confirm the overlapping state of the common regions of the point cloud data using the frame image as a reference. Note that the operation unit may be a button provided on the screen. Also, the point cloud image on which the frame image is to be displayed may be pre-selected according to the user's operation. Also, the number of point cloud images on which the frame image is displayed simultaneously may be one or a plurality.

[0027] Also, Seventh The invention is such that the processor displays a screen including an instruction to lock the point cloud image, and sets the point cloud image in a locked state according to an operation by the user. Sixth operation unit and The sixth operation unit of

[0028] According to this, by making the point cloud image in a locked state, that is, a fixed state where it cannot be operated, it is possible to prevent the user from accidentally moving the point cloud image. Note that the operation unit may be a button provided on the screen. Also, the point cloud image to be set in the locked state may be preselected according to the user's operation. Further, the number of point cloud images set in the locked state simultaneously may be one or a plurality.

[0029] Also, Eighth the invention is such that the processor instructs the display of combination history information representing the history of the combination of the two pieces of point cloud data Seventh operation unit to display a screen including the above, and according to the operation by the user The seventh operation unit of the above, the screen including the combination history information is displayed on the display device.

[0030] According to this, the user can easily confirm the history of the combination of a plurality of pieces of point cloud data, that is, how each piece of point cloud data has been combined so far. Note that the operation unit may be a button provided on the screen.

[0031] Also, Ninth the invention is such that the processor displays a screen including the combination history information and evaluation information representing the degree of appropriateness of the alignment by the transformation matrix estimated at the time of combination in the combined point cloud data generated by the point cloud combination process.

[0032] According to this, the user can easily confirm the degree of appropriateness of the combination in the combined point cloud data.

[0033] Also, Tenth the invention is a point cloud combination method for causing a processor to perform a process of combining two pieces of point cloud data acquired by three-dimensional restoration processing of a target location after performing mutual alignment, wherein a plurality of point cloud images visualizing a plurality of the point cloud data to be combined are arranged on a canvas that can be operated by the user, and an instruction to execute the point cloud combination process First operation unitA screen including [specific content] is displayed on a display device, and a user performs an alignment operation to overlap a common area of the two point cloud images on the canvas, and according to the operation of [user operation], the point cloud combining process for combining the two point cloud data is executed, and a screen including a combined point cloud image obtained by visualizing the combined point cloud data generated by the point cloud combining process is displayed on the display device The first operation unit and in response to the operation of [user operation], a combined point cloud image including a combined point cloud image obtained by visualizing the combined point cloud data generated by the point cloud combining process is displayed on the display device Then, a screen including a second operation unit for instructing restoration of the state before combination is further displayed, and in response to an operation of the second operation unit by the user, the combined point cloud data is returned to the state before combination, and the point cloud image corresponding to the point cloud data before the combination is displayed is configured as follows

[0034] According to this, similar to the first invention, in the alignment when combining a plurality of point cloud data obtained by three-dimensional restoration processing of a target location, the operability of the user is improved, and the burden on the user when there are a large number of point cloud data to be combined can be reduced

[0035] Hereinafter, embodiments of the present invention will be described with reference to the drawings

[0036] (First Embodiment) FIG. 1 is an explanatory diagram showing a situation of a photographing operation performed by a user using a photographing system according to the first embodiment

[0037] As shown in FIG. 1, the photographing system includes a photographing device 1 and a control device 2 (point cloud combining device)

[0038] The photographing device 1 includes a sensor unit 11, a display input panel unit 12, and a rod-shaped support 13. The sensor unit 11 and the display input panel unit 12 of the photographing device 1 are connected via a first cable 18. The display input panel unit 12 of the photographing device 1 and the control device 2 are connected via a second cable 19

[0039] The control device 2 is configured by a laptop-type or tablet-type PC that can be carried by a user (operator). In the example shown in FIG. 1, the control device 2 is stored in a shoulder bag and can be carried by the user, but the mode in which the user carries the control device 2 is not limited to this

[0040] The user performs a shooting operation in which the user holds the imaging device 1 by hand and walks around the target location while causing the imaging device 1 to shoot the target location. At this time, the user can change the position (height) of the sensor unit 11 by moving the arm holding the imaging device 1.

[0041] Next, the point cloud combination process performed by the control device 2 will be described. FIG. 2 is an explanatory diagram showing an example of the point cloud combination process.

[0042] When the target location is wide like a factory or when the target location is complexly partitioned like a house, it is difficult to shoot the entire target location at once. In this case, the user performs shooting of the target location in multiple parts. Further, in the control device 2, a process is performed in which a plurality of point cloud data generated by partially shooting the target location are combined to generate combined point cloud data representing the entire target location.

[0043] The example shown in FIG. 2 is a case where the target location is a house. The house is provided with a living room, a kitchen, a bedroom, a toilet, a bathroom, and the like. Further, in this example, the inside of the house is photographed in four parts, and four pieces of point cloud data #1 to #4 are generated. Further, the point cloud data of #1 and #2 are combined to generate combined point cloud data #S1, and the point cloud data of #3 and #4 are combined to generate combined point cloud data #S2. Furthermore, the combined point cloud data of #S1 and #S2 are combined to generate combined point cloud data #S3 representing the entire target location. Note that the order of combining a plurality of pieces of point cloud data is not limited to such an example.

[0044] Next, the schematic configurations of the imaging device 1 and the control device 2 will be described. FIG. 3 is a block diagram showing the schematic configurations of the imaging device 1 and the control device 2. FIG. 4 is a block diagram showing an outline of the processes performed by the processor 46 of the control device 2.

[0045] As shown in FIG. 3, the sensor unit 11 of the imaging device 1 includes a visible camera 21, a depth sensor 22, an IMU 23 (Inertial Measurement Unit), and an input / output interface 24.

[0046] The visible camera 21 (color camera) performs color photography and outputs a color photographed image.

[0047] The depth sensor 22 outputs depth information (distance image) as a detection result based on photographed images by left and right infrared cameras (not shown).

[0048] The IMU 23 detects the motion state of its own device, specifically, three-dimensional angular velocity and acceleration. Based on the detection result of the IMU 23, the position, attitude, and velocity of the sensor unit 11 can be detected.

[0049] The input / output interface 24 performs data input / output with the control device 2 via the display input panel unit 12. Specifically, the detection data of the visible camera 21, the depth sensor 22, and the IMU 23 are transmitted. The input / output interface 24 may be based on the USB (registered trademark) standard.

[0050] Note that the visible camera 21, the depth sensor 22, and the IMU 23 may not be integrated. Also, a configuration in which the depth sensor 22 and the IMU 23 are omitted and only the visible camera 21 is provided may be used. Further, a configuration in which either one of the depth sensor 22 or the IMU 23 and the visible camera 21 are provided may be used.

[0051] The display input panel unit 12 of the photographing device 1 includes a touch panel display 31, a repeater 32, and an input / output interface 33.

[0052] The touch panel display 31 displays a screen or the like that supports the user's photographing operation based on the control by the control device 2.

[0053] The repeater 32 relays data communication between the control device 2 and the sensor unit 11. The repeater 32 may be a hub based on the USB (registered trademark) standard.

[0054] The input / output interface 33 performs data input / output with the control device 2. Specifically, it receives display information such as a screen for assisting the user's shooting operation from the control device 2. The input / output interface 33 may be based on the USB (registered trademark) standard.

[0055] The control device 2 includes an input / output interface 41, a display 42 (display device), an input device 43, a memory 44, a storage device 45, and a processor 46.

[0056] The input / output interface 41 performs data input / output with the imaging device 1. The input / output interface 41 may be based on the USB (registered trademark) standard.

[0057] The display 42 displays a screen related to the management of captured data acquired in the past, a screen related to settings such as the operation conditions of the imaging device 1, a screen related to the combination of point cloud data, and the like.

[0058] The input device 43 is for the user to perform input operations. The input device 43 may be a keyboard, a mouse, a touch pad, a touch panel, or the like. When the control device 2 is configured as a tablet PC, a touch panel display in which the touch panel as the input device 43 and the display panel as the display 42 are integrated is provided.

[0059] The memory 44 stores programs executed by the processor 46 and the like.

[0060] The storage device 45 stores captured data (imaging information) acquired from the imaging device 1. The captured data includes a captured image of the visible camera 21, a detection result (distance information) of the depth sensor 22, a detection result of the IMU 23, and the imaging time. Further, the storage device 45 stores point cloud data as a three-dimensional restoration result generated by the processor 46.

[0061] The processor 46 performs various processes by executing a program stored in the memory 44. In the present embodiment, the processor 46 performs a three-dimensional restoration process P1.

[0062] In the three-dimensional restoration process P1, the processor 46 uses the SLAM (Simultaneous Localization And Mapping) method based on a captured image by the visible camera 21 or the like to generate point cloud data (environmental map) as three-dimensional space information regarding the target location. Also, in the three-dimensional restoration process P1, self-position estimation is performed in conjunction with the generation of the point cloud data, and the self-position at each time, that is, the position of the shooting point, is acquired.

[0063] As shown in FIG. 4, the three-dimensional restoration process P1 includes a feature extraction process P11, a tracking process P12, a position and orientation correction process P13, and a point cloud generation process P14.

[0064] In the feature extraction process P11, the processor 46 extracts feature information (feature points, etc.) from the captured image (frame) of the visible camera 21.

[0065] In the tracking process P12, the processor 46 compares the feature points extracted this time with the feature points extracted previously, estimates the transition amount regarding the position and orientation of the imaging device 1, and updates the position and orientation trajectory data based on the transition amount. The position and orientation trajectory data is the result of tracking the position and orientation of the imaging device 1 and includes the tracking result of the position and orientation at the time of shooting each captured image (frame), that is, information regarding the position and orientation of the imaging device 1 at each time of shooting.

[0066] In the position and orientation correction process P13, the processor 46 corrects the position and orientation trajectory data acquired in the tracking process P12 based on the detection data of the IMU 23. Here, for example, the position and orientation trajectory data is corrected to supplement the measurement results in locations with few features such as walls and ceilings.

[0067] In the point cloud generation process P14, the processor 46 generates point cloud data based on the distance information of the depth sensor 22 and the position and orientation trajectory data obtained in the tracking process P12 and the position and orientation correction process P13. Further, the point cloud generation process P14 includes a process of generating normal point cloud data to be obtained as a three-dimensional restoration result (standard point cloud generation process) and a process of generating simple point cloud data for the user to confirm the shooting situation (point cloud generation situation) (simple point cloud generation process). The simple point cloud generation process needs to be performed in real time during shooting, but the standard point cloud generation process may be performed after shooting.

[0068] Further, the processor 46 performs a point cloud alignment process P2, a point cloud combination process P3, a drawing generation process P4, a display information generation process P5, and a display process P6.

[0069] In the point cloud alignment process P2, the processor 46 performs alignment to align the common areas of a plurality of point cloud data to be combined. Specifically, as an alignment condition for aligning the common areas of the two point cloud data on the source side (alignment source) and the target side (alignment destination), a transformation matrix for coordinate transformation of each point of the point cloud data on the source side is obtained, and by applying this transformation matrix to the point cloud data on the source side, the coordinate transformation of the point cloud data on the source side is performed.

[0070] In the point cloud combination process P3, the processor 46 combines the point cloud data aligned in the point cloud alignment process P2 to generate one combined point cloud data representing the entire target location.

[0071] In the drawing generation process P4, the processor 46 generates a 3D model representing the entire target location based on the point cloud data representing the entire target location generated by the 3D restoration process P1, and generates a layout diagram representing the entire target location based on the 3D model. At this time, a 2D layout diagram (plan view) representing the entire target location is generated by projecting the 3D model representing the entire target location onto a horizontal plane. In addition, a 3D layout diagram representing the entire target location is generated by projecting the 3D model representing the entire target location based on a predetermined line-of-sight direction.

[0072] In the display information generation process P5, the processor 46 generates display information for the screen to be displayed on the display 42 of the control device 2. On the display 42, a screen is displayed for the user to perform operations such as management of shooting data, instructions for post-shooting processing (point cloud generation process P14, point cloud alignment process P2, point cloud combination process P3, and drawing generation process P4), and settings regarding processing conditions. In addition, the processor 46 generates display information for the screen to be displayed on the touch panel display 31 of the imaging device 1. On the touch panel display 31, a screen for assisting the user's shooting operation is displayed.

[0073] In the display process P6, the processor 46 displays a screen on the display 42 of the control device 2 and also displays a screen on the touch panel display 31 of the imaging device 1 based on the display information generated in the display information generation process P5.

[0074] Note that in this embodiment, the process related to the combination of point cloud data is performed in the control device 2 that can be carried by the user, but the process related to the combination of point cloud data may be performed in another device (for example, a server device) that can communicate with the control device 2.

[0075] Next, the procedure of the process performed by the control device 2 will be described. FIG. 5 is a flowchart showing the procedure of the process performed by the control device 2.

[0076] First, in response to an operation by the user that designates a point cloud file storing the point cloud data to be combined, the processor 46 acquires the point cloud data to be combined (ST101). Note that the point cloud data generated by the three-dimensional restoration process P1 is stored in the storage device 45.

[0077] Next, the processor 46 performs alignment to align the common regions of a plurality of point cloud data to be combined (point cloud alignment process) (ST102). At this time, as an alignment condition for aligning the common regions of the two pieces of point cloud data on the source side and the target side, a transformation matrix for coordinate transformation of each point of the point cloud data on the source side is obtained, and by applying this transformation matrix to the point cloud data on the source side, coordinate transformation of the point cloud data on the source side is performed.

[0078] Next, the processor 46 combines the point cloud data aligned in the point cloud alignment process (ST102) to generate one combined point cloud data representing the entire target location (point cloud combination process) (ST103). At this time, in addition to combining two pieces of point cloud data, if necessary, combination of the combined point cloud data and point cloud data, and combination of the combined point cloud data and the combined point cloud data are performed.

[0079] Next, the procedure of the point cloud alignment process performed by the control device 2 will be described. FIG. 6 is a flowchart showing the procedure of the point cloud alignment process.

[0080] In the plurality of point cloud data generated by performing photographing of the target location in multiple times, their relative positions cannot be accurately recognized. Therefore, in the control device 2, when combining a plurality of point cloud data, a process for aligning the common regions of the plurality of point cloud data (point cloud alignment process) is performed.

[0081] First, the processor 46 displays on the screen of the display 42 point cloud images obtained by visualizing each of a plurality of point cloud data to be combined, and acquires information (preliminary alignment information) regarding the relative positional relationship between two point cloud images that overlap each other in response to an operation by the user of overlapping the common portions of the two point cloud images (preliminary alignment process) (ST201).

[0082] Next, based on the preliminary alignment information acquired in the preliminary alignment process, the processor 46 extracts point clouds included in the common region from each of the two point cloud data on the source side and the target side, and generates two extracted point cloud data on the source side and the target side (common region extraction process) (ST202).

[0083] Next, the processor 46 performs preprocessing for appropriately performing the following rough alignment process (ST204) on the two extracted point cloud data on the source side and the target side extracted in the common region extraction process (ST202) (ST203).

[0084] Next, the processor 46 obtains a rough alignment condition (transformation matrix) for roughly aligning the two extracted point cloud data on the source side and the target side, and performs coordinate transformation on the extracted point cloud data on the source side by applying this rough alignment condition to the extracted point cloud data on the source side (rough alignment process) (ST204). By performing the rough alignment process prior to the fine alignment process in this way, it is possible to avoid a problem of falling into a local solution in the fine alignment process and obtain appropriate fine alignment conditions.

[0085] Next, using the extracted point cloud data aligned using the rough alignment condition (transformation matrix) obtained in the rough alignment process (ST204), the processor 46 obtains a fine alignment condition (transformation matrix) for precisely aligning the two point cloud data on the source side and the target side, and performs coordinate transformation on the point cloud data on the source side by applying this fine alignment condition to the point cloud data on the source side (fine alignment process) (ST205).

[0086] Next, the preliminary alignment process performed by the control device 2 will be described. FIG. 7 is a flowchart showing the procedure of the preliminary alignment process.

[0087] In the preliminary alignment process (ST201 in FIG. 6), first, the processor 46 generates a point cloud image in which the point cloud data is visualized in an overhead view (point cloud image generation process) (ST301). Specifically, the point cloud image is generated by projecting each point of the point cloud data onto a two-dimensional horizontal plane. At this time, a plurality of points of the point cloud data may be compressed into one point to reduce the weight of the point cloud image.

[0088] Next, the processor 46 causes the canvas screen 61 (point cloud operation screen, see FIG. 9) to be displayed on the display 42 (ST302). On the canvas screen 61, the user can perform an operation of overlapping the common areas of two point cloud images in which each of the two point cloud data is visualized.

[0089] Next, based on the user's operation on the canvas screen 61, the processor 46 acquires information (preliminary alignment information) regarding the relative positional relationship between the two overlapping point cloud images (ST303).

[0090] Next, the menu screen 51 displayed on the display 42 of the control device 2 will be described. FIG. 8 is an explanatory diagram showing the menu screen 51.

[0091] The menu screen 51 is provided with a button 52 for "point cloud generation", a button 53 for "point cloud display", a button 54 for "point cloud combination", a button 55 for "drawing generation", a button 56 for "drawing display", and a button 57 for "photograph data list".

[0092] When the user operates the "Point Cloud Generation" button 52, the point cloud generation process P14 is started. At this time, individual point cloud data is generated based on the individual shooting data generated by partially shooting the target location. When the user operates the "Point Cloud Display" button 53, the screen transitions to a screen for displaying the point cloud data generated in the point cloud generation process P14, and a point cloud image visualizing the individual point cloud data is displayed.

[0093] Also, when the user operates the "Point Cloud Merging" button 54, the point cloud alignment process P2 is started, and the screen transitions to a screen where the user performs operations related to the merging of individual point cloud data, specifically the canvas screen 61 (see FIG. 9). When an operation for instructing the execution of the point cloud merging process P3 is performed on the canvas screen 61, the point cloud merging process is started, and as a processing result, a merged point cloud image visualizing the merged point cloud data representing the entire target location is displayed.

[0094] Also, when the user operates the "Drawing Generation" button 55, the drawing generation process P4 is started. At this time, a drawing representing the entire target location is generated based on the merged point cloud data representing the entire target location. When the user operates the "Drawing Display" button 56, the screen transitions to a screen for displaying the drawing generated in the drawing generation process P4, and a drawing representing the entire target location is displayed.

[0095] Also, when the user operates the "Shooting Data List" button 57, the screen transitions to a screen where the individual shooting data is listed. On this screen, the user can check the individual shooting data.

[0096] Next, the canvas screen 61 displayed on the display 42 of the control device 2 during the preliminary alignment process will be described. FIG. 9 is an explanatory diagram showing the canvas screen 61. FIGS. 10, 11, and 12 are explanatory diagrams showing the operation status of the point cloud image 71 and the merged point cloud image 74 on the canvas 72.

[0097] During the preliminary alignment process (see FIG. 7), the canvas screen 61 (point cloud operation screen) shown in FIG. 9 is displayed on the display 42 of the control device 2.

[0098] On the canvas screen 61, a canvas window 62, an operation window 63, and a selection image window 64 are provided.

[0099] Also, on the canvas screen 61, a button 65 for "point cloud file loading" is provided. When the user operates the button 65 for "point cloud file loading", a screen (not shown) for specifying a point cloud file (a file or folder storing point cloud data) is displayed. Here, when the user performs an operation to specify a point cloud file, the point cloud data stored in the specified point cloud file is input, and a point cloud image 71 obtained by visualizing the point cloud data is displayed in the canvas window 62.

[0100] At this time, when a plurality of point cloud data to be combined are input, in the canvas window 62, a plurality of point cloud images 71 obtained by visualizing the plurality of point cloud data to be combined are arranged and displayed on the canvas 72. Note that the point cloud image 71 is a visualization of the point cloud data in a bird's-eye view state. In the example shown in FIG. 9, point cloud images 71 #1 to #4 obtained by visualizing four pieces of point cloud data #1 to #4 as the combination targets are displayed.

[0101] Also, in the canvas window 62, the size and relative positional relationship of the point cloud image 71 with respect to the canvas 72 are kept constant. The point cloud image 71 is displayed on the canvas 72 in a size proportional to the actual size of the corresponding point cloud data. Note that in the initial state when the point cloud data is input, the plurality of point cloud images 71 may be arranged on the canvas 72 so as to correspond to the actual positional relationship of the point cloud data from which they are derived.

[0102] In the selection image window 64, a thumbnail 73 of the point cloud image 71 in the selected state in the canvas window 62 is displayed. In the example shown in FIG. 9, the point cloud images 71 #1 and #2 are in the selected state. Note that in the canvas window 62, a mark indicating the selected state may be displayed on the point cloud image 71 in the selected state.

[0103] Note that the example shown in FIG. 9 is for a house, but when the target location is large like a factory, the shooting may be carried out in multiple times (for example, 10 times), and the number of point cloud data to be combined may reach a large number (for example, 10). In this case, in the canvas window 62, a large number of point cloud images 71 are arranged and displayed on the canvas 72.

[0104] Also, in the canvas window 62, the user performs an alignment operation to adjust the position and angle of the point cloud image 71 so that the common areas of two point cloud images 71 overlap each other on the canvas 72. At this time, as the alignment operation, the user can perform an operation to translate the point cloud image 71 and an operation to rotate the point cloud image 71. For example, by an operation of grabbing and moving the point cloud image 71 (for example, a drag operation), the point cloud image 71 is translated. Also, by operating a rotation mark (not shown) displayed when an operation of selecting the point cloud image 71 (for example, a click operation) is performed, the point cloud image 71 is rotated.

[0105] In the operation window 63, a button 81 of "Execute Combination" is provided as an operation part for instructing the execution of the point cloud combination process. The button 81 of "Execute Combination" changes to an operable state when two point cloud images 71 overlap according to the user's operation in the canvas window 62. Note that the button 81 of "Execute Combination" may appear when the point cloud images 71 overlap.

[0106] When the user operates the button 81 of "Execute Combination", a common area extraction process, a pre-process, a rough alignment process, and a precise alignment process (ST202 to ST205 in FIG. 6) that are performed subsequent to the preliminary alignment process (ST201 in FIG. 6) in the point cloud alignment process (ST102 in FIG. 5), and the point cloud combination process (ST103 in FIG. 5) are executed, and a combined point cloud image visualizing the combined point cloud data as a processing result is displayed on the canvas screen 61. Thereby, the user can visually confirm whether the point cloud data is appropriately combined.

[0107] In the example shown in FIGS. 10, 11, and 12, first, from the initial state shown in FIG. 10(A), the user operates the point cloud images 71 at points #1 and #2 on the canvas 72 and overlays the point cloud images 71 at #1 and #2 so that their common regions match as shown in FIG. 10(B). At this time, a rotation operation and a translation operation are performed as necessary. Next, when the user operates the "Execute Merge" button 81, a point cloud merging process is executed, and as shown in FIG. 11(A), a merged point cloud image 74 of #S1 is displayed as a processing result.

[0108] Also, as shown in FIG. 11(A), the user operates the point cloud images 71 at #3 and #4 and overlays the point cloud images 71 at #3 and #4 so that their common regions match. At this time, a rotation operation and a translation operation are performed as necessary. Next, when the user operates the "Execute Merge" button 81, a point cloud merging process is executed, and as shown in FIG. 11(B), a merged point cloud image 74 of #S2 is displayed as a processing result.

[0109] Next, as shown in FIG. 12(A), the user operates the merged point cloud images 74 of #S1 and #S2 and overlays the merged point cloud images 74 of #S1 and #S2 so that their common regions match. At this time, a rotation operation and a translation operation are performed as necessary. Next, when the user operates the "Execute Merge" button 81, a point cloud merging process is executed, and as shown in FIG. 12(B), a merged point cloud image of #S3 is displayed as a processing result.

[0110] Note that in this embodiment, a point cloud merging process for merging two point cloud data is executed by operating the "Execute Merge" button 81 after two point cloud images 71 with overlapping common regions are simultaneously selected. However, the "Execute Merge" button 81 may be operated after three or more point cloud images 71 with overlapping common regions are simultaneously selected. In this case, combinations of merging two point cloud data, merging combined point cloud data and point cloud data, and merging two combined point cloud data are executed as appropriate.

[0111] In addition, on the canvas screen 61, a button 66 for "saving the combined point cloud file" is provided as an operation unit for the user to approve the combination result and instruct data storage. When the user operates the button 66 for "saving the combined point cloud file", a combined point cloud file storing the combined point cloud data is generated and stored in the storage device 45. When the user operates the button 81 for "executing combination", a simple point cloud combination process is executed, and a combined point cloud image for the user to confirm the processing result is displayed on the canvas screen 61. On the other hand, when the user operates the button 66 for "saving the combined point cloud file", a regular point cloud combination process may be executed.

[0112] In this embodiment, a preliminary alignment process is performed based on the user's operation, but the preliminary alignment process may be performed without depending on the user's operation. Specifically, a process (common area detection process) of detecting similar areas of two point cloud images as a common area by a process such as image matching for two point cloud images may be executed by the processor 46. Further, this common area detection process may be for detecting a similar common area for three or more point cloud images.

[0113] Next, the image opacity adjustment function of the canvas screen 61 will be described. FIG. 13 is an explanatory diagram showing a situation where the opacity of the point cloud image 71 is adjusted on the canvas screen 61.

[0114] On the operation window 63 of the canvas screen 61, a slider 83 is provided as an operation unit for specifying the opacity of the point cloud image 71. When the user selects the point cloud image 71 and then operates the slider 83, the opacity of the selected point cloud image 71 changes.

[0115] When the opacity is 100%, the point cloud image 71 is displayed in an opaque state. When the opacity is less than 100%, the point cloud image 71 is displayed in a semi-transparent state. Therefore, when the opacity of the upper point cloud image 71 among the two overlapping point cloud images 71 is set to less than 100%, the lower point cloud image 71 can be seen through at the overlapping part of the two point cloud images 71, and at the same time, the upper point cloud image 71 can also be seen.

[0116] Also, when two point cloud images 71 overlap vertically, the opacity may be set only for the upper point cloud image 71, or both of the two overlapping point cloud images 71 may be set to the same opacity. In the example shown in FIG. 13(A), both of the two point cloud images 71 of #1 and #2 that overlap vertically are selected and the opacity is specified. As shown in FIG. 13(B), the two point cloud images 71 of #1 and #2 are displayed in a semi-transparent state with the same opacity. Thereby, the user can easily confirm whether the two point cloud images 71 are superimposed in a state where the common areas are properly aligned.

[0117] Note that when the opacity is specified by operating the slider 83, the point cloud image 71 is displayed in a semi-transparent state according to the specified opacity. Normally, the point cloud image 71 is displayed in an opaque state, and the point cloud image 71 may be displayed in a semi-transparent state only during the operation of the point cloud image 71.

[0118] Next, the canvas magnification / reduction function of the canvas screen 61 will be described. FIG. 14 is an explanatory diagram showing a situation where the canvas 72 is magnified and reduced on the canvas screen 61.

[0119] On the canvas screen 61, an enlargement button 84 and a reduction button 85 are provided as operation parts for instructing the enlargement and reduction of the canvas 72. When the user operates the enlargement button 84, the canvas 72 is gradually enlarged. At this time, the state transitions from the standard state (see Fig. 9) where the entire canvas 72 is displayed in the display area of the canvas screen 61 to the enlarged state (see Fig. 14) where only a part of the canvas 72 fits within the display area. On the other hand, when the user operates the reduction button 85 in the enlarged state of the canvas 72, the canvas 72 is gradually reduced and returns to the standard state.

[0120] In this way, when the user operates the enlargement button 84 and the reduction button 85, the canvas 72 is enlarged and reduced, and accordingly, the point cloud image 71 on the canvas 72 is enlarged and reduced for display. Therefore, the user can easily visually confirm whether the common areas of the point cloud images 71 are aligned with sufficient accuracy. Note that the operation parts for instructing the enlargement and reduction of the canvas 72 are not limited to the enlargement button 84 and the reduction button 85, and may be, for example, a slider or numerical input of the enlargement and reduction ratio.

[0121] Here, when the user operates the enlargement button 84 and the canvas 72 is enlarged, only a part of the canvas 72 fits within the display area. At this time, by selecting only the point cloud image 71 that is the target of alignment, the canvas 72 is enlarged with the selected point cloud image 71 as the center. Therefore, it is possible to prevent the point cloud image 71 that is the target of alignment from going outside the display area. In the example shown in Fig. 14, since two point cloud images 71 of #1 and #2 are selected, the canvas 72 is enlarged with the midpoint between the two point cloud images 71 as the center. At this time, only a part of the point cloud image 71 near the selected point cloud image 71 is displayed, and the point cloud image 71 far from the selected point cloud image 71 is outside the display area.

[0122] Also, on the canvas screen 61, by changing the distance between the viewpoints with respect to the canvas 72 while keeping the size and relative positional relationship of the point cloud image 71 with respect to the canvas 72 constant, the canvas 72 is enlarged or reduced. That is, when the user operates the zoom-in button 84, the viewpoint approaches the canvas 72, and the point cloud image 71 on the canvas 72 is enlarged and displayed (zoomed in). Also, when the user operates the zoom-out button 85, the viewpoint moves away from the canvas 72, and the point cloud image 71 on the canvas 72 is reduced and displayed (zoomed out). By keeping the size and relative positional relationship of the point cloud image 71 with respect to the canvas 72 constant in this way, the load of the display process on the canvas screen 61 is reduced. That is, it becomes easier to perform a display process of changing the position and angle of the point cloud image 71 according to the user's operation in a state where a large number of point cloud images 71 are arranged on the canvas 72.

[0123] Next, the unmerge function of the canvas screen 61 will be described. FIG. 15 is an explanatory diagram showing the operation status of unmerging on the canvas screen 61.

[0124] On the canvas screen 61, a "Unmerge" button 82 is provided as an operation unit for instructing the restoration of the state before merging. When the user operates the "Unmerge" button 82 after selecting the merged point cloud image 74, the point cloud merging process related to the selected merged point cloud image 74 is canceled and returned to the original state. As a result, when the user accidentally operates the "Execute Merge" button 81 or when an undesirable merge result is obtained, the state before merging can be restored with a simple operation.

[0125] In the example shown in FIG. 15(A), one combined point group image 74 is selected. In the selection image window 64, a thumbnail 73 of the one combined point group image 74 being selected is displayed. In the example shown in FIG. 15(B), the point group combining process related to the combined point group image 74 selected in the example shown in FIG. 15(A) is canceled, and it is in the state before combination, that is, the state where the alignment operation is performed and the two point group images 71 are superimposed. Note that when the combined point group image 74 is returned to the point group image 71 before combination, the point group image 71 is no longer in the selected state, and the thumbnail 73 of the point group image 71 is not displayed in the selection image window 64.

[0126] Next, the image frame display function of the canvas screen 61 will be described. FIG. 16 is an explanatory diagram showing a situation where a frame image 78 is displayed on the point group image 71 on the canvas screen 61.

[0127] On the canvas screen 61, a button 86 for "image frame display / non-display" is provided as an operation unit for instructing the display of the frame image 78 representing the outer periphery of the point group image 71. When the user operates the button 86 for "image frame display / non-display" after performing an operation of selecting the point group image 71, the frame image 78 is displayed on the outer peripheral portion of the selected point group image 71.

[0128] The example shown in FIG. 16(A) is the initial state (non-display state). By operating the button 86 for "image frame display / non-display", it transitions to the state where the frame image 78 is displayed as shown in FIG. 16(B) (display state). As a result, the area where the selected multiple point group images 71 overlap becomes easier to distinguish by the frame image 78, so that the user can easily confirm whether the common areas of the point group data overlap appropriately. Note that in the example shown in FIG. 16, a plurality of point group images 71 are selected, and thumbnails 73 of the selected multiple point group images 71 are displayed in the selection image window 64.

[0129] Also, when the user operates the "Image Frame Display / Non - display" button 86 while the frame image 78 is in the display state, the frame image 78 returns to the original state (non - display state) where it is not displayed. In this way, when the user operates the "Image Frame Display / Non - display" button 86, it is possible to switch between the display state where the frame image 78 is displayed and the non - display state where the frame image 78 is not displayed.

[0130] Next, the image lock function of the canvas screen 61 will be described. FIG. 17 is an explanatory diagram showing a situation where the point cloud image 71 is set to the locked state on the canvas screen 61.

[0131] On the canvas screen 61, a "Image Lock" button 87 is provided as an operation part for instructing the lock of the point cloud image 71. When the user selects the point cloud image 71 and then operates the "Image Lock" button 87, the selected point cloud image 71 is set to the locked state (a fixed state where it cannot be operated). The user cannot operate the point cloud image 71 in the locked state and can only operate the point cloud image 71 in the non - locked state. Also, on the canvas screen 61, when the point cloud image 71 is set to the locked state, a lock mark 79 indicating the locked state is displayed on the corresponding point cloud image 71.

[0132] In the example shown in FIG. 17(A), the point cloud image 71 of #1 is selected, and a thumbnail 73 of the selected point cloud image 71 of #1 is displayed in the selection image window 64. Here, when the user operates the "Image Lock" button 87, as shown in FIG. 17(B), the selected point cloud image 71 of #1 is set to the locked state and the lock mark 79 is displayed. Also, the point cloud image 71 of #2 is in an operable state and is operated so that the common area overlaps with the point cloud image 71 of #1. At this time, although the point cloud image 71 of #1 set to the locked state returns to the non - selected state, since the point cloud image 71 of #2 is in the selected state, a thumbnail 73 of the point cloud image 71 of #2 is displayed in the selection image window 64.

[0133] By locking the point cloud image 71 in this way, it is possible to prevent the point cloud image 71 that is not being operated on from being moved by a user's accidental operation. In addition, the lock mark 79 allows the user to easily grasp the locked state of the point cloud image 71. In the example shown in FIG. 17, the lock mark 79 is drawn in a circular predetermined color (for example, green), but the form of the lock mark 79 is not limited to this.

[0134] Note that in the example shown in FIG. 17, the point cloud image 71 corresponding to the source point cloud data is set to the locked state, but the combined point cloud image 74 (see FIGS. 11 and 12) corresponding to the combined point cloud data generated by the point cloud combination process can also be set to the locked state by the same operation.

[0135] Also, the number of point cloud images 71 set to the locked state at the same time may be one or more. For example, during the operation of overlapping the common areas of two point cloud images 71, only one point cloud image 71 on the target side (the alignment destination) may be set to the locked state. Also, for the purpose of preventing accidental operations on the point cloud image 71, all other point cloud images 71 except the point cloud image 71 on the source side (the alignment source) may be set to the locked state.

[0136] Next, the combined history screen 101 displayed on the display 42 of the control device 2 will be described. FIG. 18 is an explanatory diagram showing the combined history screen 101.

[0137] On the canvas screen 61 (see FIG. 9), a button 88 for "combined history" is provided as an operation unit for instructing the display of combined history information representing the history of combination of a plurality of point cloud data. When the user performs an operation of selecting the combined point cloud image 74 (see FIG. 10(B)) and then operates the button 88 for "combined history", the screen transitions to the combined history screen 101 shown in FIG. 18. Note that on the combined history screen 101, when the user performs an operation of selecting the combined point cloud image 74, the thumbnail 73 of the selected combined point cloud image 74 is displayed in the selection image window 64.

[0138] On the combination history screen 101, a system diagram 102 (tree diagram) is displayed as combination history information representing the history of combining a plurality of point cloud data. By viewing the system diagram 102, the user can easily check the combination status of the plurality of point cloud data, that is, how the point cloud data was combined. In particular, even when the target location is wide like a factory and there are a large number of source point cloud data, the user can easily check the combination status of the point cloud data.

[0139] In the example shown in FIG. 18, the combined point cloud image 74 displayed at the top corresponds to the latest combined point cloud data, and the point cloud image 71 displayed at the bottom corresponds to the source point cloud data. By tracing the system diagram 102 downward, the user can check the combination status of the point cloud data retroactively.

[0140] In addition, on the combination history screen 101, for each combined point cloud image 74, a registration matching rate (evaluation information) of the common area representing the degree of appropriateness of registration by the transformation matrix estimated at the time of combination in the combined point cloud data corresponding to the combined point cloud image 74 is displayed. Thereby, the user can easily check the degree of appropriateness of the registration by the transformation matrix estimated at the time of combination in the combined point cloud data. For example, when the registration matching rate of the common area is low, the user can confirm that the registration accuracy in the combined point cloud data is low, that is, there may be insufficient registration operation for overlapping the common areas of the source point cloud images 71.

[0141] Here, the alignment matching rate of the common area is represented by, for example, the matching rate (%) of the common area by the transformation matrix for alignment estimated when two point cloud data to be combined are combined. The matching rate is calculated based on the point-to-point distances between corresponding points in the two point cloud data. Specifically, first, for each point in the source-side point cloud after alignment, the nearest neighbor point in the target-side point cloud is set as the corresponding point. If the point-to-point distance between the corresponding points is equal to or less than a threshold value (for example, 3 cm), it is acceptable; if the point-to-point distance between the corresponding points is greater than the threshold value, it is determined to be unacceptable. Then, the ratio of the number of corresponding points determined to be acceptable to the total number of detected corresponding points is calculated as the matching rate. For example, if 50 out of 100 pairs of corresponding points are acceptable, the matching rate is 50%. If the matching rate is low, it is determined that the alignment accuracy in the combined point cloud data is low.

[0142] In addition, the combined history screen 101 is provided with a "Cancel combination" button 103. When the user performs an operation to select the combined point cloud image 74 and then operates the "Cancel combination" button 103, the point cloud combination process related to the selected combined point cloud image 74 is canceled, and the state before the point cloud combination process was executed is restored.

[0143] At this time, the point cloud combination process that generated the combined point cloud data corresponding to the selected combined point cloud image 74 is canceled. In addition, the point cloud combination process based on the combined point cloud data corresponding to the selected combined point cloud image 74 is canceled. That is, the point cloud combination process from the combined point cloud data corresponding to the selected combined point cloud image 74 to the combined point cloud data corresponding to the latest combined point cloud image 74 is canceled.

[0144] In the example shown in FIG. 18, the combined point cloud image 74 of #S1 is selected. In this case, the combined point cloud data corresponding to the combined point cloud image 74 of #1 is restored to the point cloud data corresponding to the original point cloud images 71 of #1 and #2. In addition, the combined point cloud data corresponding to the combined point cloud image 74 of #S3 generated based on the combined point cloud data corresponding to the combined point cloud image 74 of #S1 is restored.

[0145] (Modification of the First Embodiment) Next, a modification of the first embodiment will be described. Note that points not specifically mentioned here are the same as those in the above-described embodiment. FIG. 19 is an explanatory diagram showing a main canvas screen 111 and a sub-canvas screen 112 displayed on a display 42 of a control device 2 according to a modification of the first embodiment.

[0146] Similar to the canvas screen 61 (see FIG. 9) in the first embodiment, the main canvas screen 111 displays a plurality of point cloud images 71 obtained by visualizing a plurality of point cloud data to be combined, arranged side by side. Further, a button 68 for "sub-canvas" is provided on the main canvas screen 111.

[0147] When the user selects a point cloud image 71 on the main canvas screen 111 and then operates the button 68 for "sub-canvas", the sub-canvas screen 112 is pop-up displayed on the main canvas screen 111. Note that the main canvas screen 111 and the sub-canvas screen 112 may be displayed so as to be switched.

[0148] Only the point cloud image 71 selected by the user on the main canvas screen 111 is arranged and displayed on the sub-canvas screen 112. As a result, among the point cloud images 71 displayed on the main canvas screen 111, only the point cloud image 71 being aligned can be displayed on the sub-canvas screen 112 to perform the alignment operation. Therefore, it becomes easier for the user to perform an alignment operation of overlapping the common regions of two point cloud images 71 to be combined.

[0149] Note that a button for "main canvas" to return to the main canvas screen 111 may be provided on the sub-canvas screen 112. Also, the user may operate a button to close the sub-canvas screen 112 to return to the main canvas screen 111.

[0150] (Second Embodiment) Next, the second embodiment will be described. Note that points not specifically mentioned here are the same as those in the above-described embodiment. FIG. 20 is an explanatory diagram showing a canvas screen 61 displayed on a display 42 of a control device 2 according to the second embodiment.

[0151] In the first embodiment, a preliminary alignment process based on a user operation, that is, a preliminary alignment process based on a user operation of overlapping a common area of two point cloud data is performed. On the other hand, in this embodiment, in addition to the preliminary alignment process based on a user operation, a preliminary alignment process not based on a user operation can be performed. Note that the rough alignment process and the fine alignment process performed after the preliminary alignment process are the same as those in the first embodiment (see FIG. 6).

[0152] In the preliminary alignment process not based on a user operation, a process of detecting a common area of two point cloud data (common area detection process) is performed. In the common area detection process, the similarity is measured between two point cloud images 71 in which the point cloud data is visualized, and the common area of the two point cloud images 71 is detected based on the measurement result. Specifically, an area with a high similarity between the two point cloud images 71 is detected as the common area. At this time, for example, an area where the similarity between the point cloud images 71 is equal to or greater than a predetermined threshold is detected as the common area. Also, an area included in the top predetermined ratio from the one with the higher similarity between the point cloud images 71 is detected as the common area.

[0153] Also, in this embodiment, first, a preliminary alignment process not based on a user operation is performed, and if an appropriate result is not obtained in that process, the preliminary alignment process based on a user operation is performed again. That is, the user first performs an operation instructing the execution of the preliminary alignment process not based on a user operation, checks the processing result, and if the processing result is not appropriate, after restoring the point cloud combination process, performs the operation of the preliminary alignment process based on a user operation.

[0154] Specifically, as shown in FIG. 20, in addition to the "Execute Merge" button 81 and the "Cancel Merge" button 82 on the canvas screen 61, an "Auto-Execute Merge" button 121 is provided. After the user performs an operation (e.g., a click operation) to select two point cloud images 71 to be merged, when the user operates the "Auto-Execute Merge" button 121, a preliminary alignment process not based on the user's operation and a point cloud merging process based on the result of that process are executed, and a merged point cloud image 74 (see FIGS. 11 and 12) as the processing result is displayed on the canvas screen 61.

[0155] The user visually checks the merged point cloud image 74 to determine whether the processing result is appropriate. If the processing result is not appropriate, the user operates the "Cancel Merge" button 82. Next, after performing an operation to overlap the common regions of the two point cloud images 71 to be merged, the user operates the "Execute Merge" button 81. Thereby, a preliminary alignment process based on the user's operation and a point cloud merging process based on the result of that process are executed, and a merged point cloud image 74 as the processing result is displayed on the canvas screen 61.

[0156] Here, the result of the preliminary alignment process not based on the user's operation may be appropriate, or the result of that process may not be appropriate. For this reason, on the canvas screen 61, there may be a state where the merged point cloud image 74 generated by the preliminary alignment process not based on the user's operation and the merged point cloud image 74 generated by the preliminary alignment process based on the user's operation are mixed.

[0157] As described above, embodiments have been described as examples of the technology disclosed in the present application. However, the technology in the present disclosure is not limited thereto, and can also be applied to embodiments with changes, replacements, additions, omissions, etc. Further, it is also possible to combine the respective components described in the above embodiments to form a new embodiment.

Industrial Applicability

[0158] The point cloud combining apparatus and the point cloud combining method according to the present invention can improve the operability of the user in alignment when combining a plurality of point cloud data obtained by three-dimensional restoration processing of a target location, and can reduce the burden on the user when there are a large number of point cloud data to be combined. It is useful as a point cloud combining apparatus and a point cloud combining method that perform alignment of two point cloud data obtained by three-dimensional restoration processing of a target location and then combine the two point cloud data.

Explanation of Signs

[0159] 1: Imaging device 2: Control device (point cloud combining device) 42: Display 46: Processor 61: Canvas screen 71: Point cloud image 72: Canvas 73: Thumbnail 74: Combined point cloud image 78: Frame image 79: Lock mark 81: Button for "Execute combination" 82: Button for "Cancel combination" 83: Slider 84: Zoom-in button 85: Zoom-out button 86: Button for "Display / hide image frame" 87: Button for "Image lock" 88: Button for "Combination history" 101: Combination history screen 102: System diagram

Claims

1. A point cloud merging device that executes, by a processor, a process of merging two pieces of point cloud data obtained by three-dimensional restoration processing of a target location after performing mutual alignment, wherein the processor: displays, on a display device, a screen including a canvas on which a plurality of point cloud images visualizing the plurality of pieces of point cloud data to be merged are arranged so as to be operable by a user, and a first operation unit for instructing execution of point cloud merging processing; executes the point cloud merging process of merging the two pieces of point cloud data in accordance with an alignment operation by the user of overlapping common regions of the two point cloud images on the canvas and an operation of the first operation unit by the user; displays, on the display device, a screen including a merged point cloud image visualizing the merged point cloud data generated by the point cloud merging process; further displays a screen including a second operation unit for instructing restoration to the state before merging; and in response to an operation of the second operation unit by the user, returns the merged point cloud data to the state before merging and displays the point cloud image corresponding to the point cloud data before the merging, characterized by the point cloud merging device.

2. The processor: displays the merged point cloud image visualizing the merged point cloud data generated by merging the two pieces of point cloud data on the canvas; and merges the merged point cloud data and the point cloud data in response to an operation of the user of overlapping the common regions of the merged point cloud image and the point cloud image, the point cloud merging device according to claim 1.

3. The processor: displays a screen including a third operation unit for designating the opacity of the point cloud image; and displays the point cloud image in a semi-transparent state with the designated opacity in response to an operation of the third operation unit by the user, the point cloud merging device according to claim 1.

4. The processor: displays a screen including a fourth operation unit for instructing enlargement and reduction of the canvas; and enlarges and reduces the canvas in response to an operation of the fourth operation unit by the user, the point cloud merging device according to claim 1.

5. The processor: enlarges and reduces the canvas by changing the separation distance of the viewpoint with respect to the canvas while keeping the position and size of the point cloud image with respect to the canvas constant, the point cloud merging device according to claim 4.

6. The processor: A screen including a fifth operation unit for instructing display of a frame image representing the outer periphery of the point group image is displayed. The point group combining apparatus according to claim 1, wherein in response to an operation of the fifth operation unit by a user, a display state in which the frame image is displayed and a non-display state in which the frame image is not displayed are switched. **Claim 7** The processor displays a screen including a sixth operation unit for instructing locking of the point group image, and sets the point group image to a locked state in response to an operation of the sixth operation unit by a user, the point group combining apparatus according to claim 1. **Claim 8** The processor displays a screen including a seventh operation unit for instructing display of combination history information representing a history of combination of the two point group data, and displays a screen including the combination history information on the display device in response to an operation of the seventh operation unit by a user, the point group combining apparatus according to claim 1. **Claim 9** The processor displays a screen including the combination history information and evaluation information representing the degree of appropriateness of alignment by a transformation matrix estimated at the time of combination in the combined point group data generated by the point group combination process, the point group combining apparatus according to claim 8. **Claim 10** A point group combining method for causing a processor to perform a process of combining two point group data obtained by three-dimensional restoration processing of a target location after performing mutual alignment, displaying on a display device a screen including a canvas on which a plurality of point group images visualizing a plurality of the point group data to be combined are arranged so as to be operable by a user, and a first operation unit for instructing execution of a point group combination process, executing the point group combination process for combining the two point group data in response to an alignment operation by the user of overlapping a common area of the two point group images on the canvas and an operation of the first operation unit by the user, displaying on the display device a screen including a combined point group image visualizing the combined point group data generated by the point group combination process, further displaying a screen including a second operation unit for instructing restoration to a state before combination, and returning the combined point group data to a state before combination in response to an operation of the second operation unit by a user and displaying the point group image corresponding to the point group data before combination, the point group combining method being characterized by this.

Images