Point cloud processing device and point cloud processing system

Abstract

A point cloud processing apparatus includes: a first point cloud generation unit that generates a first three-dimensional point cloud of a target area using a plurality of images captured by a camera included in a moving body; and a second point cloud generation unit that generates a second three-dimensional point cloud of a partial area within the target area using an image selected based on information about a terminal device installed within the target area among the plurality of images.

Description

Technical Field

[0001] The present technology relates to a point cloud processing device and a point cloud processing system.

Background Art

[0002] As a conventional technique, there is a technique for generating a three-dimensional point cloud representing the ground surface from a plurality of image data obtained by photographing the ground surface with a camera mounted on a moving body such as a drone flying in the air.

[0003] In the generation of a three-dimensional point cloud, errors may occur based on the position detection accuracy of a position detection sensor or the like. Therefore, a technique has been proposed in which a marker 400 is installed in an area to be the target of three-dimensional point cloud generation, and the marker 400 is detected from an image obtained by photographing the area where the marker 400 is installed to generate a three-dimensional point cloud (Patent Document 1).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In the technique described in Patent Document 1, the efficiency of the marker 400 detection operation for generating a three-dimensional point cloud can be improved, but further improvement in accuracy is required in the generation of the three-dimensional point cloud.

[0006] The present technology has been made in view of such problems, and an object thereof is to provide a point cloud processing device and a point cloud processing system capable of generating another three-dimensional point cloud for improving the accuracy of the three-dimensional point cloud.

Means for Solving the Problems

[0007] To solve the above problems, a first technique includes a first point cloud generation unit that generates a first three-dimensional point cloud of a target area using a plurality of images captured by a camera included in a moving body, and a second point cloud generation unit that generates a second three-dimensional point cloud of a partial area within the target area using an image selected based on information about a terminal device installed within the target area among the plurality of images. A point group correction unit that calculates a correction amount for correcting a first three-dimensional point group based on a second three-dimensional point group It is a point cloud processing device including these.

[0008] A second technique is a point cloud processing system including a moving body equipped with a camera, a first point cloud generation unit that generates a first three-dimensional point cloud of a target area using a plurality of images captured by the camera included in the moving body, and a second point cloud generation unit that generates a second three-dimensional point cloud of a partial area within the target area using an image selected based on information about a terminal device installed within the target area among the plurality of images.

[0009] A third technique is a point cloud processing device including a first point cloud generation unit that generates a first three-dimensional point cloud of a target area using a plurality of images captured by a camera included in a moving body, and a third point cloud generation unit that generates a third three-dimensional point cloud of an area around a marker within the target area using an image selected based on information about a marker installed within the target area among the plurality of images.

[0010] A fourth technique is a point cloud processing system including a moving body equipped with a camera, a first point cloud generation unit that generates a first three-dimensional point cloud of a target area using a plurality of images captured by the camera included in the moving body, and a third point cloud generation unit that generates a third three-dimensional point cloud of an area around a marker within the target area using an image selected based on information about a marker installed within the target area among the plurality of images.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

[0012] Hereinafter, embodiments of the present technology will be described with reference to the drawings. The description will be made in the following order. <First Embodiment> [Configuration of Point Cloud Processing System 1000] [Configuration of Mobile Body 100] [Configuration of Terminal Device 200] [Configuration and Processing of Point Cloud Processing Apparatus 300] <Second Embodiment> [Configuration of Point Cloud Processing System 1000] [Configuration and Processing of Point Cloud Processing Apparatus 300] <Modification Example>

[0013] <First Embodiment> [Configuration of Point Cloud Processing System 1000] As shown in FIGS. 1 and 2, the point cloud processing system 1000 is composed of a mobile body 100 and a terminal device 200 having functions as a point cloud processing apparatus 300.

[0014] The mobile body 100 is an unmanned aircraft such as a drone that flies in the air. The mobile body 100 is equipped with a camera 105 and can automatically fly based on a pre-set flight path and perform automatic shooting. The flight path can be set by existing control software for mobile bodies or the like.

[0015] When performing automatic flight and automatic shooting, route information, shooting position, shooting direction, shooting timing, etc. are set in advance, and the mobile body 100 performs movement control and shooting control according to the set content. As shown in FIG. 2, the mobile body 100 flies over an area (target area) that is the object for creating a three-dimensional point cloud, and periodically shoots the ground surface with the camera 105 during flight to acquire a plurality of images. Note that in addition to automatic flight, the mobile body 100 may also be capable of flying by manual operation by an operator. The target area is an area where the ground surface can be photographed from above.

[0016] Note that the mobile body 100 can also shoot while changing the flight path according to an instruction via wireless communication from the user during movement. Furthermore, it is also possible to add or change the shooting timing according to an instruction via wireless communication from the user during movement.

[0017] As shown in FIG. 1, the terminal device 200 is installed as a fixed base station supported by a tripod, a stand, etc. within the target area. It is desirable to install the terminal device 200 outdoors and not shielded by other objects so that it can be photographed by the camera 105 of the mobile body 100 and can surely communicate with GNSS satellites by a GNSS (Global Navigation Satellite System) device. In the present embodiment, the terminal device 200 will be described as having a circular shape in plan view, but the shape of the terminal device 200 is not limited to this, and any shape may be used.

[0018] The terminal device 200 has a function as a point cloud processing device 300. The point cloud processing device 300 creates a first three-dimensional point cloud, which is a three-dimensional point cloud of the target area, based on a plurality of images obtained by the camera 105 of the mobile body 100 photographing the ground surface of the target area. Also, the point cloud processing device 300 creates a second three-dimensional point cloud, which is a three-dimensional point cloud of a partial area within the target area, based on a plurality of images obtained by the camera 105 of the mobile body 100 photographing the ground surface of the target area. The partial area includes the terminal device 200 installed within the target area as shown in FIG. 1 and is an area smaller than the target area. In the embodiment, the target area and the partial area are rectangular areas, but they may be other shapes such as circular, trapezoidal, or free form. Note that there may be objects such as buildings and automobiles on the ground surface.

[0019] A three-dimensional point cloud is set of data of a plurality of points having position information and color information. It can represent terrain, objects, etc. as a collection of a large number of points and is point cloud data that can be used in various fields such as civil engineering, architecture, and manufacturing.

[0020] In this embodiment, since the mobile body 100 is an unmanned aerial vehicle flying in the air, the mobile body 100 is connected to the terminal device 200 by wireless connection. However, the supply of data and information from the mobile body 100 to the terminal device 200 may also be performed via a recording medium such as a USB flash memory or an SD memory card.

[0021] Examples of wireless connection methods include networks such as Wi-Fi, wireless LAN (Local Area Network), 4G (4th generation mobile communication system), 5G (5th generation mobile communication system), Bluetooth (registered trademark), NFC (Near Field Communication), and Ethernet (registered trademark).

[0022] [Configuration of Mobile Body 100] The configuration of the mobile body 100 will be described with reference to FIG. 3. Although illustration is omitted, the mobile body 100 includes a housing, a rotor, a rotor support shaft, etc. as an external configuration. However, it may be a fixed wing instead of a rotor.

[0023] The UAV (Unmanned Aerial Vehicle) control unit 101 is composed of a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), etc., and controls the entire mobile unit 100 and each part thereof by executing processing according to a program and issuing a command. Further, the UAV control unit 101 controls the moving speed, moving direction, turning direction, etc. of the mobile unit 100 by supplying a control signal for controlling the output of the actuator 102 to the actuator 102.

[0024] Also, the UAV control unit 101 controls the mobile unit 100 to move along a pre-set moving path by controlling the output of the actuator 102 while comparing the current position of the mobile unit 100 with the pre-set moving path.

[0025] The actuator 102 is a driving source for driving a rotary wing and is provided at the tip of a rotary wing support shaft or the like. The actuator 102 operates under the control of the UAV control unit 101.

[0026] The gimbal 103 rotatably supports the camera 105 at the lower part or the like of the mobile unit 100. The gimbal 103 is a kind of support tool that supports with, for example, a two-axis or three-axis shaft.

[0027] The gimbal control unit 104 controls the operation of the gimbal 103 that supports the camera 105. By controlling the rotation of the shaft of the gimbal 103 with the gimbal control unit 104, the posture of the camera 105 can be freely adjusted.

[0028] The camera 105 includes an imaging element, a signal processing circuit, etc., and captures an RGB (Red, Green, Blue) or monochromatic image. As the imaging element, a CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor), etc. are used.

[0029] The camera 105 is mounted so as to be suspended via a gimbal 103, for example, on the bottom surface of the housing of the moving body 100 in order to photograph the ground surface of the target area. The camera 105 can be driven to direct the lens in any direction from the 360-degree horizontal direction to the vertically downward direction for photographing. Note that the camera 105 may be mounted on the moving body 100 by any method as long as it can direct the lens in all directions from the 360-degree horizontal direction to the vertically downward direction for photographing.

[0030] The position information acquisition unit 106 acquires the position (for example, latitude, longitude, altitude) of the camera 105 using, for example, a GNSS device or a GPS (Global Positioning System) device.

[0031] The attitude information acquisition unit 107 is constituted by, for example, an IMU (Inertial Measurement Unit), and acquires the attitudes of the moving body 100 and the camera 105 by means of an acceleration sensor, an angular velocity sensor, a gyro sensor, etc. in two-axis or three-axis directions. The attitude information is, for example, a three-axis rotation vector (Yaw, Pitch, Roll) calculated from the acceleration and angular velocity of the moving body 100. Note that the acquisition of the position information and attitude information of the camera 105 is also possible by, for example, a laser sensor capable of acquiring three-dimensional information and a SLAM (Simultaneous Localization and Mapping) technique based on the output of the laser sensor.

[0032] The sensor control unit 108 controls so as to synchronize the acquisition of an image by photographing with the camera 105 and the acquisition of the camera position information by the position information acquisition unit 106. Further, the sensor control unit 108 associates the image with the camera position information acquired in synchronization with the photographing of the image. Thus, the camera position information is associated with each of a plurality of images photographed by the camera 105. Note that the sensor control unit 108 may attach time information such as a time stamp indicating that the image and the camera position information are synchronized to the image and the camera position information.

[0033] The communication unit 109 is various communication terminals or communication modules that perform communication processing via a transmission path such as the Internet, wired / wireless communication with various devices, and bus communication. The mobile body 100 transmits the image and the camera position information to the point cloud processing device 300 by the communication unit 109. Note that the mobile body 100 may transfer those pieces of information to the point cloud processing device 300 via an external storage medium such as a USB flash memory or an SD memory card.

[0034] Note that by using a recording medium such as a semiconductor memory, removing the recording medium from the mobile body 100 after the mobile body 100 has landed, and connecting it to a recording medium connection slot (not shown) of the point cloud processing device 300, the image and the camera position information may be transferred.

[0035] For example, the mobile body 100 which is a drone enables desired flight by controlling the output of the actuator 102. In a hovering state where it is stationary in the air, the inclination is detected by the attitude information acquisition unit 107 or a separately provided gyro sensor, and the output of the actuator 102 on the side where the body has dropped is increased, and the output of the actuator 102 on the side where the body has risen is decreased to keep the body horizontal. Further, when moving forward, the output of the actuator 102 in the traveling direction is decreased, and the output of the actuator 102 in the reverse direction is increased to make the body take a forward inclination posture and generate a propulsive force in the traveling direction.

[0036] [Configuration of Terminal Device 200] Next, with reference to FIG. 4, the configuration of the terminal device 200 will be described.

[0037] The CPU 201 functions as an arithmetic processing unit that performs various processes and controls the entire terminal device 200 and each of its components. The CPU 201 executes various processes according to programs stored in the ROM 202 or the non-volatile memory section 204, or programs loaded from the storage section 208 into the RAM 203. For the non-volatile memory section 204, for example, an EEPROM (Electrically Erasable Programmable Read Only Memory) or the like is used. Also, in the RAM 203, data and the like necessary for the CPU 201 to execute various processes are appropriately stored.

[0038] The CPU 201, ROM 202, RAM 203, and non-volatile memory section 204 are interconnected via a bus. Also, an input / output interface 205 is connected to the bus.

[0039] Connected to the input / output interface 205 are a GNSS device 206, an input section 207, a storage section 208, a communication section 209, and a drive 210.

[0040] The GNSS device 206 acquires the three-dimensional coordinates of the terminal device 200 with reference to GNSS satellites by receiving information such as the positions of the satellites transmitted from the GNSS satellites in the satellite positioning system and the signal transmission times. By including the GNSS device 206, the terminal device 200 can acquire its own three-dimensional coordinates. The three-dimensional coordinates of the terminal device 200 acquired by the GNSS device 206 are supplied to the image selection section 305 and the point cloud correction section 308 of the point cloud processing device 300.

[0041] The input section 207 is various operators and operation devices such as, for example, a keyboard, a mouse, keys, a dial, a touch panel, a touch pad, and a remote controller. Operations by the user are detected by the input section 207, and signals corresponding to the input operations are interpreted by the CPU 201.

[0042] The storage unit 208 is a large-capacity storage medium such as a hard disk or a flash memory. Various applications, data, information, etc. are stored in the storage unit 208.

[0043] The communication unit 209 is various communication terminals or communication modules that perform communication processing via a transmission path such as the Internet, wired / wireless communication with various devices, and communication by bus communication, etc. The point cloud processing device 300 receives the image and camera position information transmitted from the mobile body 100 by the communication unit 209. Also, the point cloud processing device 300 can transmit the three-dimensional point cloud to the outside by the communication unit 209. The point cloud processing device 300 may transfer information and data between the mobile body 100 and an external device via an external storage medium such as a USB flash memory or an SD memory card.

[0044] A drive 210 is connected to the input / output interface 205 as needed. A removable storage medium 211 is appropriately mounted via the drive 210. The removable storage medium 211 includes a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc.

[0045] The drive 210 can read data files such as programs used for each process from the removable storage medium 211. The read data file is stored in the storage unit 208. Also, computer programs etc. read from the removable storage medium 211 are installed in the storage unit 208 as needed.

[0046] In the terminal device 200, for example, software for the processing of this technology can be installed via network communication by the communication unit 209 or via the removable storage medium 211. Also, the software may be stored in advance in the ROM 202, the storage unit 208, etc. Also, the image captured by the camera 105 and the processing result by AI image processing may be received, and the image and the processing result may be stored in the storage unit 208 or the removable storage medium 211.

[0047] The terminal device 200 needs to be equipped with a function that can acquire its own three-dimensional coordinates using the GNSS device 206. Also, it needs to be installed at a position where it can be photographed by the camera 105 of the moving body 100 without being blocked by objects, etc. Furthermore, the terminal device 200 has a role as one that specifies a partial area within the target area. Therefore, the terminal device 200 needs to be installed within the target area, and it is desirable to be installed outdoors.

[0048] As the terminal device 200, for example, a surveying device equipped with a GNSS function and a point cloud generation function that is installed and used at a civil engineering site or a construction site can be used.

[0049] [Configuration and Processing of Point Cloud Processing Device 300] Next, with reference to FIG. 5, the configuration of the point cloud processing device 300 that operates in the terminal device 200 and the processing in the point cloud processing device 300 will be described.

[0050] As a prerequisite for the processing in the point cloud processing device 300, first, based on a pre-created movement route, the moving body 100 moves (flies) over the target area and performs photographing with the camera 105.

[0051] As shown in FIG. 2, while the moving body 100 is moving along the pre-movement route, the camera 105 repeatedly and continuously performs photographing at a predetermined fixed interval. Also, in synchronization with the photographing by the camera 105, the position information acquisition unit 106 acquires camera position information that is the position of the camera at the time of photographing. As described above, camera position information is associated with each of the plurality of images photographed by the camera 105. Then, the moving body 100 transmits the plurality of images and the camera position information to the terminal device 200.

[0052] The information acquisition unit 301 acquires the plurality of images and the camera position information transmitted from the moving body 100 and received by the terminal device 200, and supplies them to the feature extraction unit 302.

[0053] The feature extraction unit 302 extracts feature points from each of the plurality of images. Examples of feature points include corners of buildings or objects, boundaries of luminance changes, and boundaries of color changes.

[0054] The feature matching unit 303 identifies the correspondence relationships of the plurality of images by matching the feature points in each image extracted by the feature extraction unit 302. Further, the feature matching unit 303 may identify the correspondence relationships of the plurality of images using camera position information, or may identify the correspondence relationships of the plurality of images using both the feature points and the camera position information.

[0055] The sparse point cloud generation unit 304 generates a three-dimensional point cloud (three-dimensional sparse point cloud) with a sparse point distribution using the plurality of images for which the correspondence relationships have been identified. Further, the sparse point cloud generation unit 304 may estimate the shooting position of the camera 105 based on the plurality of images by means of SfM (Structure From Motion) or the like.

[0056] The image selection unit 305 selects images for three-dimensional point cloud generation from the plurality of images captured by the camera 105 and supplies them to the first point cloud generation unit 306 and the second point cloud generation unit 307.

[0057] The image selection unit 305 selects an image (target area image) for generating a first three-dimensional point cloud of the target area from the plurality of images based on the camera position information and supplies it to the first point cloud generation unit 306. Based on the camera position information, the image selection unit 305 supplies the first point cloud generation unit 306 with a plurality of images in which redundancy has been eliminated, for example, by excluding images taken outside the target area or overlapping images. When there are no such images outside the target area or overlapping images, all of the plurality of images captured by the camera 105 are supplied to the first point cloud generation unit 306 as target area images.

[0058] In order to select an image that the image selection unit 305 supplies to the first point cloud generation unit 306, camera attitude information may be used in addition to the camera position information. For this purpose, the moving body 100 also transmits the camera attitude information to the terminal device 200 together with the camera position information. Further, the point cloud processing device 300 may include an estimation processing unit that estimates the attitude of the camera 105 at the time of shooting based on the image. There is SfM as a method for obtaining the attitude of the camera 105 from the image.

[0059] Further, the image selection unit 305 selects an image (partial area image) for generating a second three-dimensional point cloud of a partial area from a plurality of images and supplies it to the second point cloud generation unit 307. The second three-dimensional point cloud is a three-dimensional point cloud of a partial area. The partial area image for generating the second three-dimensional point cloud is an image in which the terminal device 200 exists as a subject among the plurality of images captured by the moving body 100.

[0060] The image selection unit 305 selects a partial area image from a plurality of images using the three-dimensional coordinates of the terminal device 200 supplied from the GNSS device 206. The position where the image was taken can be specified by the camera position information acquired in synchronization with the shooting. Also, the position of the terminal device 200 can be specified by the three-dimensional coordinates acquired by the GNSS device 206. Therefore, an image in which the terminal device 200 exists as a subject can be specified based on the three-dimensional coordinates of the terminal device 200 and the camera position information. The image selection unit 305 selects the image in which the terminal device 200 exists as a subject as the partial area image. Note that the partial area image may be one or a plurality.

[0061] Further, the image selection unit 305 can also detect the terminal device 200 from a plurality of images by an AI for terminal device detection, a known subject detection method, etc. without using the three-dimensional coordinates of the terminal device 200, and select an image in which the terminal device 200 exists as a subject as the partial area image based on the detection result. In this case, the GNSS device 206 does not need to supply the three-dimensional coordinates of the terminal device 200 to the image selection unit 305 of the point cloud processing device 300.

[0062] The three-dimensional coordinates of the terminal device 200 and the detection results of the terminal device 200 in a plurality of images correspond to the information regarding the terminal device 200 in the claims.

[0063] The first point cloud generation unit 306 generates a depth image of the target area based on the target area image supplied from the image selection unit 305, and further generates a first three-dimensional point cloud, which is a dense three-dimensional point cloud of the target area, using the depth image. The first point cloud generation unit 306 supplies the first three-dimensional point cloud to the point cloud correction unit 308.

[0064] The second point cloud generation unit 307 generates a depth image of the partial area based on the partial area image supplied from the image selection unit 305, and generates a second three-dimensional point cloud, which is a dense three-dimensional point cloud of the partial area, using the depth image. By the image selection unit 305 supplying the partial area image to the second point cloud generation unit 307, the second point cloud generation unit 307 can generate a second three-dimensional point cloud, which is a three-dimensional point cloud of the partial area where the terminal device 200 exists. The second point cloud generation unit 307 supplies the second three-dimensional point cloud to the point cloud correction unit 308.

[0065] In this way, in the present technology, as shown in FIG. 6, in addition to the first three-dimensional point cloud for the target area, a second three-dimensional point cloud for the partial area where the terminal device 200 exists is generated. FIG. 6 is a side view of the target area and the partial area shown in FIG. 1. The first point cloud generation unit 306 and the second point cloud generation unit 307 can generate a three-dimensional point cloud by SfM. The second three-dimensional point cloud is of higher quality than the first three-dimensional point cloud. Higher quality means that there is less variation among the plurality of points constituting the three-dimensional point cloud (less dispersion of the point cloud), high accuracy (high average positional accuracy of the point cloud), high point density, etc. The second three-dimensional point cloud does not necessarily need to satisfy all of these, and may satisfy any one of them, or may satisfy all of them, or may satisfy any combination of them. The first three-dimensional point cloud and the second three-dimensional point cloud are generated in the same coordinate system.

[0066] When the first point cloud generation unit 306 and the second point cloud generation unit 307 use the same depth image generation algorithm, the quality of the point cloud can be adjusted by the resolution of the depth image and the parameter settings during depth calculation (such as the number of iterations of the iterative calculation process when performing correspondence of stereo images).

[0067] Specifically, by increasing the resolution of the depth image, the density of the point cloud is increased (the processing speed is increased).

[0068] Also, the quality of the point cloud is improved by changing the parameter settings during depth calculation. As a method for improving the quality of the point cloud during depth calculation, there is a method of performing correspondence by changing the matching block size during matching cost calculation in stereo matching step by step from large to small, that is, the so-called "coarse-to-fine search" method. There is also a method of changing the setting of which images to associate with stereo matching (performing association with more images). Furthermore, in the process of obtaining a disparity map of the entire image by stereo matching cost optimization, there is also a method of increasing the number of iterations of the matching cost minimization operation, which is a non-linear optimization operation.

[0069] Note that the second three-dimensional point cloud may be made denser and of higher quality than the first three-dimensional point cloud by making the depth image generation algorithm in the second point cloud generation unit 307 different from the depth image generation algorithm in the first point cloud generation unit 306. For example, in the first point cloud generation unit 306, an algorithm with lightweight processing and medium accuracy is used, and in the second point cloud generation unit 307, an algorithm with very heavy processing but high accuracy and robustness is used, and so on.

[0070] Since the terminal device 200 is installed in the target area, when generating the three-dimensional point cloud of the target area, the three-dimensional point cloud includes the point cloud corresponding to the top surface of the terminal device 200. Here, the relationship between the three-dimensional point cloud of the target area and the position of the top surface of the terminal device 200 will be described.

[0071] First, the side view direction (Z direction) will be described. As shown in FIG. 7A, the terminal device 200 is supported by a tripod or the like and installed on the ground surface of the target area. When generating a three-dimensional point cloud for the target area where the terminal device 200 exists, as shown in FIG. 7B, the three-dimensional point cloud includes a point cloud corresponding to the top surface of the terminal device 200.

[0072] When there is no error in the three-dimensional point cloud (including the case where the error is slightly below a predetermined amount), the relationship between the center position of the top surface of the terminal device 200 identified from the three-dimensional coordinates of the terminal device 200 acquired by the GNSS device 206 and the three-dimensional point cloud corresponding to the top surface of the terminal device 200 is as shown in FIG. 7C. Since there is no error, the height in the Z direction of the center position of the top surface of the terminal device 200 and the first three-dimensional point cloud of the terminal device 200 coincides. It is desirable that the three-dimensional point cloud is in such a state without error.

[0073] On the other hand, when there is an error in the Z direction of the three-dimensional point cloud, as shown in FIG. 8B or FIG. 8C, the position in the Z direction of the center position of the top surface of the terminal device 200 and the three-dimensional point cloud of the terminal device 200 is shifted. Also, a shift occurs between the three-dimensional point cloud of the ground surface and the ground surface itself. In the present technology, by using the second three-dimensional point cloud, such a shift of the three-dimensional point cloud can be corrected.

[0074] Next, the plan view direction (XY direction) will be described. As shown in FIG. 9A, the terminal device 200 is installed on the ground surface of the target area. When generating a three-dimensional point cloud for the target area where the terminal device 200 exists, as shown in FIG. 9B, the three-dimensional point cloud includes a point cloud corresponding to the top surface of the terminal device 200.

[0075] When there is no error in the three-dimensional point cloud (including the case where the error is slightly below a predetermined amount), the relationship between the center position of the top surface of the terminal device 200 identified from the three-dimensional coordinates of the terminal device 200 acquired by the GNSS device 206 and the three-dimensional point cloud corresponding to the top surface of the terminal device 200 is as shown in FIG. 9B. Since there is no error, the center position of the top surface of the terminal device 200 coincides with the approximate center of the three-dimensional point cloud of the top surface of the terminal device 200. It is desirable that the three-dimensional point cloud is in such a state without error.

[0076] On the other hand, when there is an XY-direction error in the three-dimensional point group, as shown in FIG. 10B, the position of the center of the top surface of the terminal device 200 and the position of the three-dimensional point group of the terminal device 200 in the XY direction deviate. In this technology, such a deviation of the three-dimensional point group can be corrected by using the second three-dimensional point group.

[0077] Returning to the description of FIG. 5. Next, the point group correction unit 308 calculates a correction amount for correcting the first three-dimensional point group based on the second three-dimensional point group and the three-dimensional coordinates of the terminal device 200, and corrects the first three-dimensional point group based on the correction amount.

[0078] A method for calculating the correction amount in the Z direction of the first three-dimensional point group will be described with reference to FIG. 11. First, the second three-dimensional point group in a partial region shown in FIG. 11A is clipped within a predetermined range. The predetermined range is, for example, a cylindrical range with a radius r and a height h centered on the center position of the top surface of the terminal device 200 that can be specified from the three-dimensional coordinates of the terminal device 200. As a result, as shown in FIG. 11B, the second three-dimensional point group outside the predetermined range is excluded from the processing target.

[0079] Next, as shown in FIG. 11C, a predetermined value in the Z direction of the clipped second three-dimensional point group is determined. The predetermined value is, for example, the average or median of the Z coordinates of a plurality of points constituting the clipped second three-dimensional point group, the height of a plane obtained as a result of aligning a plane parallel to the XY plane with the clipped second three-dimensional point group, and the like.

[0080] Since the first three-dimensional point group and the second three-dimensional point group are generated in the same coordinate system, as shown in FIG. 11D, the difference between the predetermined value in the Z direction and the center position of the top surface of the terminal device 200 becomes the correction amount in the Z direction for correcting the first three-dimensional point group.

[0081] By adjusting the position in the Z direction of the first three-dimensional point group based on the correction amount in the Z direction calculated in this way, the first three-dimensional point group can be corrected to be highly accurate.

[0082] Next, a method for calculating the correction amount in the XY direction of the first three-dimensional point group will be described with reference to FIG. 12. First, clip the second three-dimensional point group in a partial region shown in FIG. 12A within a predetermined range. The predetermined range is, for example, a range with a radius r centered on the center position of the top surface of the terminal device 200 that can be specified from the three-dimensional coordinates of the terminal device 200. As a result, as shown in FIG. 12B, the second three-dimensional point group outside the predetermined range is excluded from the processing target.

[0083] Next, as shown in FIG. 12C, determine a predetermined value in the XY direction of the clipped second three-dimensional point group. The predetermined value is, for example, the average or median of the XY coordinates of a plurality of points constituting the clipped second three-dimensional point group, the center of a circle obtained as a result of fitting a circle to the projected points after projecting the second three-dimensional point group onto the XY plane, and the like.

[0084] Since the first three-dimensional point group and the second three-dimensional point group are generated in the same coordinate system, as shown in FIG. 12D, the difference between the predetermined value in the XY direction and the center position of the top surface of the terminal device 200 becomes the correction amount in the XY direction for correcting the first three-dimensional point group.

[0085] By adjusting the position in the XY direction of the first three-dimensional point group based on the correction amount in the XY direction calculated in this way, the first three-dimensional point group can be corrected to be highly accurate.

[0086] Note that the point group correction unit 308 may correct only the Z direction of the first three-dimensional point group, or may correct only the XY direction. The user may be able to select which direction to correct.

[0087] Then, the output unit 309 outputs the corrected first three-dimensional point group to an external device, a cloud server, etc. via a network or the like. Further, the output unit 309 may output the corrected first three-dimensional point group to a storage unit 208 of the terminal device 200 or a storage medium such as a USB flash memory or an SD memory card.

[0088] In addition, the output unit 309 may output, as information, the correction amount calculated by the point cloud correction unit 308 to an external device or the like, instead of or in addition to the corrected first three-dimensional point cloud. Further, the output unit 309 may output, instead of or in addition to the corrected first three-dimensional point cloud, the uncorrected first three-dimensional point cloud or the second three-dimensional point cloud to an external device or the like. The output unit 309 may output the first three-dimensional point cloud and the correction amount to an external device so that the external device corrects the first three-dimensional point cloud.

[0089] The processing in the present technology is performed as described above. According to the present technology, it is possible to generate a second three-dimensional point cloud for correcting the first three-dimensional point cloud, which is the three-dimensional point cloud of the entire target area, to improve the accuracy. Then, the first three-dimensional point cloud can be corrected using the correction amount calculated using the second three-dimensional point cloud.

[0090] Since the terminal device 200 is installed as a fixed base station within the target area, by generating the second three-dimensional point cloud using the three-dimensional coordinates of the terminal device 200, it is not necessary to separately install the marker 400 for generating the second three-dimensional point cloud within the target area.

[0091] Note that if there is one point cloud generation unit and an algorithm that prioritizes quality is used for generating the depth image in that point cloud generation unit, the point cloud becomes dense and of high quality, and the degree of restoration of the shape of the ground surface and the terminal device 200 becomes high, but there is a problem that the processing time becomes long.

[0092] Also, if there is one point cloud generation unit and an algorithm that prioritizes processing speed is used for generating the depth image in that point cloud generation unit, the processing time becomes short, but the point cloud becomes sparse, and further variations occur, resulting in a low degree of restoration of the shape of the ground surface and the terminal device 200.

[0093] In addition, if there is one point cloud generation unit and an algorithm that prioritizes processing speed for depth image generation in the point cloud generation unit is used, and further noise filtering processing is performed on the depth image, the processing time will be shortened and the variation of the point cloud can be suppressed. However, there is a problem that the degree of restoration of the shape of the ground surface and the terminal device 200 becomes low.

[0094] In addition, if there is one point cloud generation unit and an algorithm that prioritizes processing speed for depth image generation in the point cloud generation unit is used, and further noise filtering processing is performed on the point cloud, the processing time will be shortened and the variation of the point cloud can be suppressed. However, there is a problem that the degree of restoration of the shape of the ground surface and the terminal device 200 becomes low.

[0095] On the other hand, in the present technology, the point cloud processing device 300 includes two point cloud generation units, namely, a first point cloud generation unit 306 and a second point cloud generation unit 307. An algorithm that prioritizes processing speed may be used for depth calculation in the first point cloud generation unit 306, and an algorithm that prioritizes the quality of the point cloud may be used for depth calculation in the second point cloud generation unit 307. Further, the first point cloud generation unit 306 may perform noise filtering processing on the generated depth image. Thereby, first and second three-dimensional point clouds with a short processing time, little variation in the point cloud, good quality, and a high degree of restoration of the shape of the ground surface and the terminal device 200 can be generated.

[0096] The point cloud processing device 300 may include a synthesis processing unit that synthesizes the first three-dimensional point cloud and the second three-dimensional point cloud to generate a synthesized three-dimensional point cloud. The synthesized three-dimensional point cloud is obtained by superimposing the second three-dimensional point cloud on the first three-dimensional point cloud. By displaying the synthesized three-dimensional point cloud, the first three-dimensional point cloud and the second three-dimensional point cloud can be presented to the user together. Thereby, the user can easily grasp the positional relationship between the first three-dimensional point cloud and the second three-dimensional point cloud. When displaying on a display device for three-dimensional point clouds or the like, they may be displayed individually, or the first three-dimensional point cloud and the second three-dimensional point cloud may be synthesized to display a synthesized three-dimensional point cloud. The user may be able to select which three-dimensional point cloud to display.

[0097] On the top surface of the terminal device 200 shown in the images captured by the camera 105 of the moving body 100, air-to-air marks, patterns, textures, etc. as shown in FIGS. 13A to H may be provided. This can improve the accuracy of recognizing the terminal device 200 from the images captured by the camera 105 and can improve the position estimation accuracy of the terminal device 200. Also, when generating the second three-dimensional point cloud, the depth image of the terminal device 200 can be generated more robustly and accurately. As a result, the quality of the second three-dimensional point cloud can be further improved. When the position of the terminal device 200 can be estimated with high accuracy from the image, the estimation result may be used instead of the three-dimensional coordinates obtained by the GNSS device 206, or the estimation result and the three-dimensional coordinates obtained by the GNSS device 206 may be used in combination. Note that the air-to-air marks, patterns, and textures shown in FIG. 13 are merely examples, and any patterns or textures with random patterns may be used.

[0098] Even if the top surface of the terminal device 200 is plain white, since it itself becomes a pattern in the image, the position of the terminal device 200 can be estimated from the image captured by the camera 105 by template matching or AI recognition processing. Also, by painting the top surface of the terminal device 200 with a special color, the terminal device 200 can be detected from the image captured by the camera 105 by color detection processing. In this way, there are methods for estimating the position of the terminal device 200 from the image and for detecting the terminal device 200, but if the method of providing air-to-air marks, patterns, or textures as shown in FIG. 13 on the top surface of the terminal device 200 is adopted, the accuracy of recognizing the terminal device 200 can be further improved and the position estimation accuracy of the terminal device 200 can be improved.

[0099] The terminal device 200 may be provided with the function of the point cloud processing device 300 in advance, or the point cloud processing device 300 and the point cloud processing method may be realized by the terminal device 200 executing a program. The program may be installed in the terminal device 200 in advance, or may be distributed by downloading, a storage medium, etc. so that a user or the like installs it.

[0100] Further, the terminal device 200 does not have the function as the point cloud processing device 300, and an electronic device having an information processing function and a communication function existing outside the target area may function as the point cloud processing device 300. In that case, the mobile body 100 transmits a plurality of images and camera position information to the electronic device, and the terminal device 200 transmits its own three-dimensional coordinates to the electronic device. The electronic device is a personal computer, a smartphone, a tablet terminal, or the like. Also, a cloud server may function as the point cloud processing device 300. However, when the terminal device 200 installed in the target area functions as the point cloud processing device 300, there is an advantage that a three-dimensional point cloud can be generated on-site at a civil engineering site or a construction site, which is the target area.

[0101] Also, the point cloud processing device 300 may be realized by an electronic device having a function as a computer executing a program. The program may be installed in the electronic device in advance, or may be distributed by downloading, a storage medium, etc., and installed by a user or the like.

[0102] Note that the terminal device 200 may have a function of distributing RTK (Real Time Kinematic) correction information and supply RTK correction information to construction machinery or the like using an RTK correction information distribution service configured by an external wireless device or a cloud server.

[0103] Also, the point cloud processing device 300 may transmit the created three-dimensional point cloud to an external device, a cloud server, or the like. Then, the external device or the cloud server may convert the data format of the three-dimensional point cloud and transmit it to another external device, a cloud server, or the like.

[0104] The cloud server may be composed not only of a single computer device but also of a plurality of systematized computer devices. The plurality of computer devices are systematized, for example, by a LAN (Local Area Network) or the like. Also, a plurality of computer devices arranged in a remote location may be systematized by a VPN (Virtual Private Network) or the like using the Internet or the like. The plurality of computer devices may include computer devices as a server group (cloud) available by cloud computing services.

[0105] <Second Embodiment> [Configuration of Point Cloud Processing System 1000] Next, a second embodiment of the present technology will be described. In the second embodiment, as shown in FIG. 14, a marker 400 is installed in the target area. The marker 400 serves as a reference for the third three-dimensional point cloud generated in the second embodiment.

[0106] Based on a plurality of images captured by the camera 105 provided in the mobile body 100 of the ground surface of the target area, the point cloud processing device 300 creates a third three-dimensional point cloud, which is a three-dimensional point cloud of the area around the marker 400 (marker surrounding area) in the target area. The marker surrounding area includes the marker 400 installed in the target area as shown in FIG. 14 and is an area smaller than the target area. In FIG. 14, the target area and the marker surrounding area are shown as rectangular areas, but they may be other shapes such as circular, trapezoidal, or free form. Note that there may be objects such as buildings and automobiles on the ground surface.

[0107] The marker 400 may be installed so as to be photographable by the camera 105 provided in the mobile body 100 and may have known three-dimensional coordinates.

[0108] As the marker 400, an air target marker with a GPS function capable of obtaining the three-dimensional coordinates of the marker 400 can be used. Also, a general air target marker without a GPS function can also be used as the marker 400. Regardless of the presence or absence of the GPS function, the appearance of the air target marker is the same as that shown in FIG. 13 in the first embodiment. However, when using an air target marker without a GPS function, it is necessary to measure the three-dimensional coordinates of the air target marker in advance using an external device or system. Examples of methods for measuring three-dimensional coordinates include SLAM (Simultaneous Localization and Mapping), conventional surveying methods, LiDAR (Light Detection And Ranging), and methods of referring to existing map data. Any method may be used as long as the three-dimensional coordinates of the marker 400 can be obtained.

[0109] In addition, objects that pre-exist in the target area such as buildings, and objects installed in the target area by the user can also be used as the marker 400. Since such objects do not have a GPS function, it is necessary to measure the three-dimensional coordinates in advance using an external device or system with SLAM, conventional surveying methods, LiDAR, map data, etc.

[0110] When the marker 400 is equipped with a GPS function and a communication function, the marker 400 transmits three-dimensional coordinate information to the point cloud processing device 300 by communication. Also, when the marker 400 does not have a GPS function and the three-dimensional coordinates of the air target marker are measured in advance by an external device or system, it is necessary to transmit the three-dimensional coordinate information of the marker 400 from that external device or system to the point cloud processing device 300.

[0111] It is desirable to install the marker 400 on a flat surface in the target area that has no inclination or has little inclination. The marker 400 can be installed anywhere within the target area.

[0112] When the marker 400 is set as a verification point, the accuracy of the position of the first three-dimensional point cloud can be verified using the third three-dimensional point cloud. Also, when the marker 400 is set as a GCP (Ground Control Point) or a calibration point, the position of the first three-dimensional point cloud can be corrected using the third three-dimensional point cloud. Therefore, the user needs to specify whether to set the marker 400 for verifying the accuracy of the position of the first three-dimensional point cloud or for correcting the position of the first three-dimensional point cloud, and set the marker 400 accordingly. Note that the installation method of the marker 400 is the same when it is used for accuracy verification of the position and when it is used for correction of the position.

[0113] The configuration and movement of the moving body 100, the imaging by the camera 105 provided in the moving body 100, and the transmission of a plurality of images and camera position information from the moving body 100 to the terminal device 200 are the same as those in the first embodiment. As a premise for the processing in the point cloud processing device 300, first, based on a pre-created movement path, the moving body 100 moves (flies) over the target area and imaging is performed by the camera 105.

[0114] The configuration of the terminal device 200 having the function as the point cloud processing device 300 is the same as that in the first embodiment.

[0115] [Configuration and Processing of Point Cloud Processing Device 300] With reference to FIG. 15, the configuration of the point cloud processing device 300 operating in the terminal device 200 and the processing in the point cloud processing device 300 will be described.

[0116] The configurations and processes of the information acquisition unit 301, the feature extraction unit 302, the feature matching unit 303, the sparse point cloud generation unit 304, and the output unit 309 are the same as those in the first embodiment.

[0117] The image selection unit 305 selects an image for generating a three-dimensional point cloud based on a plurality of images captured by the camera 105 and supplies it to the first point cloud generation unit 306 and the third point cloud generation unit 310.

[0118] The image selection unit 305 selects an image (target area image) for generating the first three-dimensional point cloud of the target area in the same manner as in the first embodiment, and supplies it to the first point cloud generation unit 306.

[0119] Also, the image selection unit 305 selects an image (marker peripheral area image) for generating the third three-dimensional point cloud of the area around the marker, and supplies it to the third point cloud generation unit 310. The third three-dimensional point cloud is the three-dimensional point cloud of the area around the marker. The marker peripheral area image for generating the third three-dimensional point cloud is an image in which the marker 400 exists as a subject among a plurality of images captured by the moving body 100.

[0120] The image selection unit 305 selects a marker peripheral area image from a plurality of images using the three-dimensional coordinates of the marker 400. The position where the image was taken can be specified based on the camera position information acquired in synchronization with the shooting. Also, the three-dimensional coordinates of the marker 400 can be obtained by using the GPS function provided in the marker 400 or by measuring in advance using an external device or system. Therefore, the image in which the marker 400 exists as a subject can be specified based on the three-dimensional coordinates of the marker 400 and the camera position information. The image selection unit 305 selects the image in which the marker 400 exists as a subject as the marker peripheral area image. Note that the marker peripheral area image may be one or a plurality.

[0121] Note that the image selection unit 305 can also detect the marker 400 from a plurality of images by using an AI for marker detection or a known subject detection method without using the three-dimensional coordinates of the marker 400, and select the image in which the marker 400 exists as a subject as the marker peripheral area image based on the detection result.

[0122] The three-dimensional coordinates of the marker 400 and the detection result of the marker 400 in a plurality of images correspond to the information regarding the marker 400 in the claims.

[0123] The first point cloud generation unit 306 generates a first three-dimensional point cloud, which is a three-dimensional point cloud of the target area, through the same process as in the first embodiment. The first point cloud generation unit 306 supplies the first three-dimensional point cloud to the output unit 309.

[0124] The third point cloud generation unit 310 generates a depth image of the marker surrounding area based on the marker surrounding area image supplied from the image selection unit 305, and uses the depth image to generate a third three-dimensional point cloud, which is a dense three-dimensional point cloud of the marker surrounding area. By the image selection unit 305 supplying the marker surrounding area image to the third point cloud generation unit 310, the third point cloud generation unit 310 can generate the third three-dimensional point cloud of the marker surrounding area where the marker 400 exists. The third point cloud generation unit 310 supplies the third three-dimensional point cloud to the output unit 309.

[0125] In this way, in the second embodiment, in addition to the first three-dimensional point cloud of the target area, a third three-dimensional point cloud of the marker surrounding area where the marker 400 exists is generated. The first point cloud generation unit 306 and the third point cloud generation unit 310 can generate three-dimensional point clouds by SfM. The third three-dimensional point cloud is of higher quality than the first three-dimensional point cloud. High quality means that there is little variation among the plurality of points constituting the three-dimensional point cloud (low dispersion of the point cloud), high accuracy (high average positional accuracy of the point cloud), high density of points, etc. The third three-dimensional point cloud does not necessarily need to satisfy all of these, and may satisfy any one of them, or all of them, or any combination of them. The first three-dimensional point cloud and the third three-dimensional point cloud are generated in the same coordinate system.

[0126] In the second embodiment, similar to the first embodiment, when the first point cloud generation unit 306 and the third point cloud generation unit 310 use the same depth image generation algorithm, the quality of the point cloud can be adjusted according to the resolution of the depth image and the parameter settings during depth calculation.

[0127] In FIG. 14, the terminal device 200 is installed within the target area. However, it is not essential to generate the second three-dimensional point cloud for a partial area that is the area including the terminal device 200. Also, the terminal device 200 may be installed outside the target area.

[0128] Here, the relationship between the three-dimensional point cloud of the target area in the plan view direction (XY direction) and the top surface position of the marker 400 will be described.

[0129] As shown in FIG. 16A, the marker 400 is installed on the ground surface of the target area. When generating the three-dimensional point cloud for the target area where the marker 400 exists, as shown in FIG. 16B, the three-dimensional point cloud includes, in addition to the three-dimensional point cloud of the ground surface, the three-dimensional point cloud corresponding to the top surface of the marker 400.

[0130] When there is no error in the three-dimensional point cloud (including the case where the error is slightly below a predetermined amount), the relationship between the top surface center position of the marker 400 that can be specified from the three-dimensional coordinates of the marker 400 and the three-dimensional point cloud corresponding to the top surface of the marker 400 is as shown in FIG. 16B. Since there is no error, the top surface center position of the marker 400 coincides with the approximate center of the three-dimensional point cloud of the top surface of the marker 400. It is desirable that the three-dimensional point cloud is in such a state without error.

[0131] On the other hand, when there is an error in the XY direction in the three-dimensional point cloud, as shown in FIG. 17B, the top surface center position of the marker 400 and the position of the three-dimensional point cloud of the marker 400 in the XY direction are displaced.

[0132] Next, with reference to FIG. 18, a method for verifying the accuracy of the position of the first three-dimensional point cloud in the XY direction will be described. Verifying the position accuracy means verifying how much error there is between the first three-dimensional point cloud for the target area and the three-dimensional position of the actual target area.

[0133] First, clip the third three-dimensional point cloud in the marker surrounding area shown in FIG. 18A within a predetermined range. The predetermined range is, for example, a range with a side length of s centered on the top surface center position of the marker 400 that can be specified from the three-dimensional coordinates of the marker 400. The third three-dimensional point cloud includes the point cloud corresponding to the top surface of the marker 400 and the point cloud of the ground surface. However, by the clipping, as shown in FIG. 18B, among the points constituting the third three-dimensional point cloud, the points outside the predetermined range are excluded from the processing target. Note that the Z direction may be clipped with a predetermined width based on the physical height information of the marker 400. The clipping in the Z direction is also effective in removing outliers noise in the point cloud in the Z direction.

[0134] Next, as shown in FIG. 18C, calculate a predetermined value in the XY direction of the clipped third three-dimensional point cloud. The predetermined value is the average or median of the XY coordinates of a plurality of points constituting the clipped third three-dimensional point cloud, the center of a circle obtained as a result of fitting a circle to the projected points after projecting the third three-dimensional point cloud onto the XY plane, the centroid or average position of a point sequence surrounding the top surface center position of the marker 400, and the like. Also, among the plurality of points constituting the third three-dimensional point cloud, the position of the point closest to the top surface center position of the marker 400 that can be specified from the three-dimensional coordinates of the marker 400 may be used as the predetermined value.

[0135] Since the first three-dimensional point cloud and the third three-dimensional point cloud are generated in the same coordinate system, as shown in FIG. 18D, the error between the predetermined value in the XY direction of the third three-dimensional point cloud and the top surface center position of the marker 400 can be used as the error between the first three-dimensional point cloud and the marker 400. Therefore, by checking the error between the predetermined value in the XY direction of this third three-dimensional point cloud and the top surface center position of the marker 400, the accuracy verification of the position of the first three-dimensional point cloud can be performed. It can be said that the smaller the error between the predetermined value in the XY direction of the third three-dimensional point cloud and the top surface center position of the marker 400, the higher the accuracy of the position of the first three-dimensional point cloud.

[0136] By using the error between the predetermined value in the XY direction of the third three-dimensional point cloud obtained in this way and the top surface center position of the marker 400 as the correction amount, the position in the XY direction of the first three-dimensional point cloud can be corrected to make the first three-dimensional point cloud highly accurate.

[0137] When verifying the accuracy of the position of the first three-dimensional point cloud, if the density of the third three-dimensional point cloud around the marker 400 is low, the resolution in the XY direction becomes rough and the verification accuracy becomes low. By generating the third three-dimensional point cloud with a high density in the area around the marker, it is possible to verify and correct the accuracy of the position of the first three-dimensional point cloud with high precision.

[0138] In the second embodiment, the output unit 309 outputs the first three-dimensional point cloud, the third three-dimensional point cloud, the three-dimensional coordinates of the marker 400, etc. to an external device or a cloud server via a network or the like. Then, the accuracy verification and correction of the position of the first three-dimensional point cloud can be performed by the above-described method in the external device or the cloud server or the like.

[0139] Note that the output unit 309 may output the first three-dimensional point cloud, the third three-dimensional point cloud, the three-dimensional coordinates of the marker 400, etc. to the storage unit 208 of the terminal device 200 or a storage medium such as a USB flash memory or an SD memory card.

[0140] The processing of the second embodiment is performed as described above. According to the second embodiment, when the marker 400 is used as a verification point, regardless of the generation density, it is possible to verify the accuracy of the position of the first three-dimensional point cloud, which is the three-dimensional point cloud of the entire target area, with high precision without significantly increasing the calculation resources. Further, when the marker 400 is used as a GCP or a calibration point, the position accuracy of the first three-dimensional point cloud for the target area can be improved by performing correction.

[0141] Note that the verification accuracy of the position varies depending on the density of the three-dimensional point cloud. This point will be described with reference to FIG. 19. FIGS. 19A to 19C are schematic diagrams showing the first three-dimensional point cloud and the third three-dimensional point cloud. The points outside the broken line frame are the first three-dimensional point cloud on the ground surface, and the points inside the broken line frame are the third three-dimensional point cloud of the marker 400. Further, in FIGS. 19A to 19C, partial enlarged views of the third three-dimensional point cloud are shown.

[0142] FIG. 19A shows an example where the densities of the first three-dimensional point group and the third three-dimensional point group are the same or nearly the same. FIG. 19B shows an example where the densities of the first three-dimensional point group and the third three-dimensional point group are the same or nearly the same and the density is higher than that in FIG. 19A. FIG. 19C shows an example where the density of the third three-dimensional point group is higher than that of the first three-dimensional point group.

[0143] In FIG. 19, the "ideal position for measuring the error" is the coordinate position corresponding to the actual top surface center in the three-dimensional point group coordinate system of the marker 400. The "top surface center position (three-dimensional coordinates) of the marker 400" is the top surface center position of the marker 400 specified from the three-dimensional coordinates. The "position for actually measuring the error" is the predetermined value described above. The predetermined value is the position used for actually measuring the position error, and as described above, it is the average of the XY coordinates of a plurality of points constituting the third three-dimensional point group, the median, the centroid or the average position of the point sequence, the position of the point closest to the top surface center position of the marker 400, and the like.

[0144] As shown in Fig. 19A, when both the first three-dimensional point group and the third three-dimensional point group are of low density, the computational resources for verifying the position accuracy are reduced, but the verification accuracy also decreases. When both the first three-dimensional point group and the third three-dimensional point group are of high density as shown in Fig. 19B rather than of low density as shown in Fig. 19A, the distance between the position (predetermined value) for measuring the actual error and the center position of the top surface of the marker 400 becomes shorter, and the verification accuracy of the position of the first three-dimensional point group becomes higher. However, if an attempt is made to generate both the first three-dimensional point group and the third three-dimensional point group as high-density ones, the processing load in generation increases, and the computational resources also increase. As a result, a large amount of computational resources are required, and the time required for processing also becomes longer. Therefore, as shown in Fig. 19C, by generating the third three-dimensional point group at a high density and generating the first three-dimensional point group at a lower density than the third three-dimensional point group, the processing load of three-dimensional point group generation can be reduced to the same level as when both the first three-dimensional point group and the third three-dimensional point group are of low density. Also, the verification accuracy of the position of the first three-dimensional point group can be made as high as when both the first three-dimensional point group and the third three-dimensional point group are of high density. That is, if the third three-dimensional point group is generated at a high density, even if the first three-dimensional point group is generated at a low density, the verification accuracy can be increased with fewer computational resources. Therefore, regardless of the generation density of the first three-dimensional point group, highly accurate position accuracy verification becomes possible. This is the same in the first embodiment. Note that the content described with reference to Fig. 19 is the same in terms of the correction accuracy of the position of the three-dimensional point group.

[0145] In FIG. 14, there is one marker 400 installed in the target area, but a plurality of markers 400 may be installed in the target area. When installing a plurality of markers 400, the point cloud processing device 300 needs to be provided with a plurality of point cloud generation units to generate the third three-dimensional point cloud of the area around each marker 400. By using the third three-dimensional point cloud of each of the plurality of areas around the markers, the verification accuracy and correction accuracy of the position of the first three-dimensional point cloud for the target area can be improved. The plurality of markers 400 can be installed anywhere within the target area. For example, as shown in FIG. 20, when the target area is rectangular, it is preferable to install a total of five markers 400 at the four corners and approximately the center of the target area. In that case, the point cloud processing device 300 generates the third three-dimensional point cloud of each of the five areas around the markers.

[0146] Here, the GUI (Graphical User Interface) in the case where a marker 400 is installed as a verification point and the accuracy of the position of the first three-dimensional point cloud is verified using the third three-dimensional point cloud will be described.

[0147] As described above, in order to verify the accuracy of the position of the first three-dimensional point cloud using the third three-dimensional point cloud, a position (referred to as the error measurement position) that is actually used to measure the position error between the first three-dimensional point cloud and the third three-dimensional point cloud is required. As described above, the error measurement position is, for example, the average of the XY coordinates of a plurality of points constituting the third three-dimensional point cloud, the median, the centroid or average position of the point sequence, the position of the point closest to the top surface center position of the marker 400, etc. Also, the user can actually specify the error measurement position by input. The GUI is used for the user to input the error measurement position.

[0148] The GUI shall be displayed on the display device 500. The display device 500 is, for example, a display device such as a liquid crystal display or an organic EL (Electroluminescence). The display device 500 may be provided in the terminal device 200, may be connected to the terminal device 200, may be provided in the device that performs the accuracy verification process, or may be connected to the device that performs the accuracy verification process. Further, the display device 500 may be a device dedicated to GUI display. The display device 500 may have a display processing function for the GUI, or the terminal device 200 provided with the display device 500 or connected to the display device 500, or the device that performs the accuracy verification process may have a display processing function for the GUI.

[0149] As shown in FIG. 21A, in the first display mode of the GUI, when the user makes an input instructing a transition to a mode for verifying the accuracy of the first three-dimensional point group in a state where the first three-dimensional point group is displayed in 2D in the XY direction, the GUI transitions to the second display mode.

[0150] As shown in FIG. 21B, in the second display mode, the third three-dimensional point group is displayed in 2D in the XY direction. With this second display mode, as shown in FIG. 21C, the center position of the marker in the third three-dimensional point group, that is, the position for error measurement, becomes easier to visually recognize, and the user can easily specify the position for error measurement by input.

[0151] Note that not limited to the state where the first three-dimensional point group is displayed in 2D, even if the device that displays the third three-dimensional point group is in any state, when the user makes an input instructing a transition to a mode for verifying the accuracy of the first three-dimensional point group, the third three-dimensional point group may be displayed.

[0152] The input for specifying the position for error measurement may be a touch input to the display device 500 when the display device 500 is a touch panel, may be an input by a cursor superimposed on the third three-dimensional point group on the display device 500, may be a gaze input, or may be any input method that can specify a position.

[0153] Rather than selecting any point that constitutes the third three-dimensional point group as the position for error measurement, the user specifies the coordinates in the third three-dimensional point group. Therefore, it is not necessary for the user to specify a point, and it may be between points. The coordinates are XY coordinates.

[0154] In addition, the third three-dimensional point group may be enlarged and displayed in the GUI so that the user can accurately specify the center of the third three-dimensional point group as the position for error measurement. The enlarged display may be by specifying a specific magnification numerically, or when the display device 500 is a touch panel, the magnification may be changed according to a pinch-in operation by the user. Also, the display range of the third three-dimensional point group may be adjustable according to the user's input.

[0155] As described above, in the second embodiment, the output unit 309 outputs the first three-dimensional point group, the third three-dimensional point group, the three-dimensional coordinates of the marker 400, etc. to an external device or a cloud server via a network or the like, so that the accuracy verification of the first three-dimensional point group can be performed externally. Therefore, the position for error measurement input by the user can also be output to an external device or a cloud server together with that information, so that the accuracy verification of the first three-dimensional point group can be performed by an external device or the like. Note that the point group processing device 300 may be provided with a function of performing the accuracy verification of the first three-dimensional point group.

[0156] When the position for error measurement is input as shown in FIG. 22A, the accuracy verification of the first three-dimensional point group is performed using it. To perform the accuracy verification of the first three-dimensional point group, as shown in FIG. 22B, the 3D coordinates P = (X P , Y P , Z P ) of the position for error measurement are required. Since the XY coordinates (X P , Y P ) are specified by the user's input for the third three-dimensional point group displayed in 2D, it is necessary to calculate the Z coordinate (Z P ) for accuracy verification.

[0157] The Z coordinate can be calculated, for example, by taking the average or median of the Z coordinates of N (e.g., N = 4) neighboring points around the position for error measurement input by the user among the plurality of points constituting the third three-dimensional point group, or by linear interpolation of the Z coordinate according to the XY distance from the neighboring points.

[0158] As shown in FIG. 23A, when the user tries to specify the center of the first three-dimensional point group as the position for error measurement while viewing the display of the first three-dimensional point group, since the density of the point group is sparse, the range formed by the points around the center (indicated by the dashed line in FIG. 23A) becomes large, resulting in a deviation between the input position and the center of the first three-dimensional point group.

[0159] On the other hand, according to the GUI of the present technology, as shown in FIG. 23B, since the third three-dimensional point group with a higher density than the first three-dimensional point group is displayed, the range formed by the points around the center (indicated by the dashed line in FIG. 23B) becomes small, and the specification accuracy of the position for error measurement can be improved. Therefore, the user can accurately perform the input of specifying the center of the first three-dimensional point group as the position for error measurement. Thereby, the accuracy verification of the first three-dimensional point group can be performed more accurately.

[0160] In the accuracy verification of the first three-dimensional point group, the position error between the first three-dimensional point group and the top surface center position of the marker 400 is calculated based on the 3D coordinates P of the position for error measurement on the first three-dimensional point group.

[0161] <Modification Example> Although the embodiments of the present technology have been specifically described above, the present technology is not limited to the above-described embodiments, and various modifications based on the technical idea of the present technology are possible.

[0162] FIG. 24 shows a first modification of the point cloud processing apparatus 300. In the first modification, the point cloud processing apparatus 300 includes a correction amount calculation unit 311 instead of the point cloud correction unit 308. The correction amount calculation unit 311 calculates the correction amount in the same manner as the point cloud correction unit 308 in the embodiment, and supplies the correction amount to the first point cloud generation unit 306. Then, the first point cloud generation unit 306 generates the first three-dimensional point cloud while adjusting either one or both of the position in the Z direction and the position in the XY direction based on the correction amount.

[0163] Alternatively, the first point cloud generation unit 306 may reflect the correction amount calculated by the correction amount calculation unit 311 in the camera position information, and generate the first three-dimensional point cloud based on the corrected camera position information.

[0164] Alternatively, the first point cloud generation unit 306 may reflect the correction amount in the depth image generated as intermediate data in the process of generating the three-dimensional point cloud, and generate the first three-dimensional point cloud.

[0165] FIG. 25 shows a second modification of the point cloud processing apparatus 300. The point cloud processing apparatus 300 may be configured not to include the point cloud correction unit 308 as in the second modification. In that case, the output unit 309 outputs the first three-dimensional point cloud and the second three-dimensional point cloud to an external device or a system, etc., and the external device or the system, etc. calculates the correction amount and corrects the first three-dimensional point cloud.

[0166] FIG. 26 shows a third modification of the point cloud processing apparatus 300. The point cloud processing apparatus 300 may include a point cloud synthesis unit 312 as in the third modification. The first three-dimensional point cloud is input from the first point cloud generation unit 306 to the point cloud synthesis unit 312, and the second three-dimensional point cloud is input from the second point cloud generation unit 307. The point cloud synthesis unit 310 overlaps and synthesizes these multiple three-dimensional point clouds to generate a synthesized three-dimensional point cloud. The synthesized three-dimensional point cloud is useful when presenting multiple three-dimensional point clouds to the user by display. Note that the point cloud processing apparatus 300 of the second embodiment including the first point cloud generation unit 306 and the third point cloud generation unit 310 may include the point cloud synthesis unit 312. Further, the point cloud processing apparatus 300 may include both the point cloud correction unit 308 and the point cloud synthesis unit 312, or may include only one of them.

[0167] FIG. 27 shows a fourth modification of the point cloud processing apparatus 300. As in the fourth modification, the point cloud processing apparatus 300 may include a first point cloud generation unit 306 that generates a first three-dimensional point cloud for a target area, a second point cloud generation unit 307 that generates a second three-dimensional point cloud for a partial area, and a third point cloud generation unit 310 that generates a third three-dimensional point cloud for an area around a marker. The processing of the second point cloud generation unit 307 is the same as that in the first embodiment. The processing of the third point cloud generation unit 310 is the same as that in the second embodiment. That is, the first embodiment and the second embodiment may be combined.

[0168] According to the fourth modification, it is possible to verify and correct the accuracy of the position in the Z direction of the first three-dimensional point cloud using the second three-dimensional point cloud, and to verify and correct the accuracy of the position in the XY direction of the first three-dimensional point cloud using the third three-dimensional point cloud.

[0169] In the fourth modification, it is necessary to install the terminal device 200 and the marker 400 in the target area. The image selection unit 305 selects a partial area image from a plurality of images based on the three-dimensional coordinates of the terminal device 200 and outputs it to the second point cloud generation unit 307. Further, the image selection unit 305 selects a marker surrounding area image from a plurality of images based on the three-dimensional coordinates of the marker 400 and outputs it to the third point cloud generation unit 310.

[0170] Note that the point cloud processing apparatus 300 of the fourth modification may include a point cloud synthesis unit 312.

[0171] The point cloud processing apparatus 300 of the second embodiment may include a point cloud verification unit that verifies the accuracy of the position of the first three-dimensional point cloud by the method described in the embodiment, and a point cloud correction unit that corrects the first three-dimensional point cloud. In that case, the output unit 309 can output the verification result and the corrected first three-dimensional point cloud to an external device or the like.

[0172] In addition, the point cloud processing device 300 according to the second embodiment includes a correction amount calculation unit 311 that calculates a correction amount for correcting the first three-dimensional point cloud by the method described in the embodiment, and may output the correction amount as information to an external device or the like. In that case, the first three-dimensional point cloud can be corrected by such an external device or the like. Further, the third and fourth modified examples may include any one or a plurality of the point cloud verification unit, the point cloud correction unit, and the correction amount calculation unit 311 described above.

[0173] In the embodiment, it is assumed that the target area is an outdoor area where the ground surface can be photographed from above. However, any location may be used as long as it is a space where the moving body 100 can move and photograph even indoors. In that case, a positioning method such as using an MBS (Metropolitan Beacon System) or a BLE (Bluetooth Low Energy) beacon instead of GNSS may be used to acquire the three-dimensional coordinates of the terminal device indoors.

[0174] In addition to the unmanned aerial vehicle, the moving body 100 may be a manned aircraft, a glider, a helicopter, a balloon, an airship, a rocket, or the like, or may be a device that can move along a rail installed at a high position such as the ceiling inside a building. Further, the moving body 100 is not limited to those that move in the air, and may be an automobile, a motorcycle, a bicycle, a personal mobility device, an airplane, a ship, a robot, a construction machine, an agricultural machine, or the like.

[0175] This technology can also adopt the following configuration. (1) A first point cloud generation unit that generates a first three-dimensional point cloud of a target area using a plurality of images captured by a camera included in a moving body; A second point cloud generation unit that generates a second three-dimensional point cloud of a partial area within the target area using an image selected based on information about a terminal device installed within the target area among the plurality of images; A point cloud processing device comprising: (2) The partial area includes the terminal device and is smaller than the target area. The point cloud processing device according to (1). (3) The second point cloud generation unit generates the second three-dimensional point cloud using an image in which the terminal device exists as a subject, which is selected from the plurality of images based on the three-dimensional coordinates of the terminal device, which are information about the terminal device, in the point cloud processing device according to (1) or (2). (4) The second point cloud generation unit generates the second three-dimensional point cloud using an image in which the terminal device exists as a subject, which is selected from the plurality of images based on the detection result of the terminal device in the image, which is information about the terminal device, in the point cloud processing device according to any one of (1) to (3). (5) The point cloud processing device according to any one of (1) to (4), comprising a point cloud correction unit that calculates a correction amount for correcting the first three-dimensional point cloud based on the second three-dimensional point cloud. (6) The point cloud correction unit calculates, as the correction amount, a difference between a predetermined value in the Z direction of the second three-dimensional point cloud and a value in the Z direction of the three-dimensional coordinates in the point cloud processing device according to (5). (7) The point cloud correction unit calculates, as the correction amount, a difference between a predetermined value in the XY direction of the second three-dimensional point cloud and a value in the XY direction of the three-dimensional coordinates in the point cloud processing device according to (5) or (6). (8) The point cloud correction unit corrects the first three-dimensional point cloud using the correction amount in the point cloud processing device according to any one of (5) to (7). (9) The three-dimensional coordinates are acquired by the terminal device based on information received using a satellite positioning system in the point cloud processing device according to (3). (10) The terminal device is installed at a position where the information in the satellite positioning system can be received in the point cloud processing device according to (9). (11) The second three-dimensional point cloud is of higher quality than the first three-dimensional point cloud in the point cloud processing device according to any one of (1) to (10). (12) The terminal device is the point cloud processing device according to any one of (1) to (11), which has a specific pattern on its top surface. (13) The point cloud processing device according to any one of (1) to (12), which performs processing on the terminal device installed in the target area. (14) The point cloud processing device according to any one of (1) to (13), which includes a synthesis processing unit that synthesizes the first three-dimensional point cloud and the second three-dimensional point cloud. (15) A mobile body equipped with a camera, A first point cloud generation unit that generates a first three-dimensional point cloud of a target area using a plurality of images captured by the camera of the mobile body, and a second point cloud generation unit that generates a second three-dimensional point cloud of a partial area within the target area using an image selected based on information about a terminal device installed in the target area among the plurality of images. A point cloud processing system comprising: (16) A first point cloud generation unit that generates a first three-dimensional point cloud of a target area using a plurality of images captured by a camera of a mobile body, A third point cloud generation unit that generates a third three-dimensional point cloud of an area around a marker within the target area using an image selected based on information about the marker installed in the target area among the plurality of images, A point cloud processing device comprising: (17) The point cloud processing device according to (16), wherein the area around the marker includes the marker and is smaller than the target area. (18) The point cloud processing device according to (16) or (17), wherein the third point cloud generation unit generates the third three-dimensional point cloud using an image selected from the plurality of images based on the three-dimensional coordinates of the marker, which is information about the marker, and in which the marker exists as a subject. (19) The point cloud processing device according to any one of (16) to (18), wherein the marker is an air target with a position information detection function. (20) The marker is the point cloud processing device according to any one of (16) to (19), which is an air target marker with known coordinates. (21) The marker is the point cloud processing device according to any one of (16) to (20), which is installed as a verification point for verifying the accuracy of the first three-dimensional point cloud. (22) The marker is the point cloud processing device according to any one of (16) to (21), which is installed as a GCP (Ground Control Point) or calibration point for correcting the first three-dimensional point cloud. (23) The point cloud processing device according to any one of (16) to (22), comprising a second point cloud generation unit that generates a second three-dimensional point cloud of a partial region within the target region using an image selected based on information about a terminal device installed within the target region among the plurality of images. (24) The point cloud processing device according to (16), wherein the third three-dimensional point cloud is displayed on a display unit for verifying the accuracy of the first three-dimensional point cloud. (25) The point cloud processing device according to (24), wherein in a state where the third three-dimensional point cloud is displayed, it is possible to specify a position used for measuring a position error between the first three-dimensional point cloud and the third three-dimensional point cloud by a user. (26) The point cloud processing device according to (25), wherein the position is the center of the marker in the third three-dimensional point cloud. (27) The point cloud processing device according to (25) or (26), wherein the Z coordinate of the position is calculated based on the XY coordinates of the position, and the position error between the first three-dimensional point cloud and the third three-dimensional point cloud is measured based on the XY coordinates and the Z coordinate. (28) The point cloud processing device according to any one of (24) to (27), wherein the third three-dimensional point cloud is displayed on a display unit in response to an input instructing verification of the accuracy of the first three-dimensional point cloud from a user. (29) A moving body equipped with a camera, A point cloud processing apparatus including: a first point cloud generation unit that generates a first three-dimensional point cloud of a target area using a plurality of images captured by the camera included in the moving body; and a third point cloud generation unit that generates a third three-dimensional point cloud of an area around a marker within the target area using an image selected based on information about a marker installed within the target area among the plurality of images. A point cloud processing system comprising the above.

Explanation of Signs

[0176] 100 ··· Moving body 105 ··· Camera 200 ··· Terminal device 300 ··· Point cloud processing apparatus 306 ··· First point cloud generation unit 307 ··· Second point cloud generation unit 310 ··· Third point cloud generation unit 308 ··· Point cloud correction unit 400 ··· Marker 1000 ··· Point cloud processing system

Claims

1. a first point cloud generation unit that generates a first three-dimensional point cloud of a target area using a plurality of images captured by a camera provided on the moving object; a second point cloud generation unit that generates a second three-dimensional point cloud of a partial area within the target area using an image selected from the plurality of images based on information about a terminal device installed within the target area; a point cloud correction unit that calculates a correction amount for correcting the first three-dimensional point cloud based on the second three-dimensional point cloud; A point cloud processing device comprising:

2. The point cloud correction unit calculates, as the correction amount, a difference between a predetermined value in the Z direction of the second three-dimensional point cloud and a value in the Z direction of the three-dimensional coordinates. The point cloud processing device according to claim 1 .

3. The point cloud correction unit calculates, as the correction amount, a difference between a predetermined value in the XY direction of the second three-dimensional point cloud and a value in the XY direction of the three-dimensional coordinates. The point cloud processing device according to claim 1 .

4. The point cloud correction unit corrects the first three-dimensional point cloud using the correction amount. The point cloud processing device according to claim 1 .

5. The three-dimensional coordinates of the terminal device are acquired by the terminal device based on information received using a satellite positioning system. The point cloud processing device according to claim 1 .

6. The terminal device is installed in a position where it can receive the information in the satellite positioning system. The point cloud processing device according to claim 5 .

7. a first point cloud generation unit that generates a first three-dimensional point cloud of a target area using a plurality of images captured by a camera provided on the moving object; a third point cloud generation unit that generates a third three-dimensional point cloud of a marker surrounding area within the target area using an image selected from the plurality of images based on information about a marker placed within the target area; A point cloud processing device comprising:

8. The marker surrounding region includes the marker and is smaller than the target region. The point cloud processing device according to claim 7 .

9. The third point cloud generation unit generates the third three-dimensional point cloud using an image in which the marker exists as a subject, selected from the plurality of images based on the three-dimensional coordinates of the marker, which is information about the marker. The point cloud processing device according to claim 7 .

10. The marker is an anti-aircraft beacon equipped with a position information detection function. The point cloud processing device according to claim 7 .

11. The marker is an airborne indicator with known coordinates. The point cloud processing device according to claim 7 .

12. The markers are set as verification points for verifying the accuracy of the first three-dimensional point cloud. The point cloud processing device according to claim 7 .

13. The markers are installed as GCPs (Ground Control Points) or orientation points for correcting the first three-dimensional point cloud. The point cloud processing device according to claim 7 .

14. a second point cloud generation unit that generates a second three-dimensional point cloud of a partial area within the target area using an image selected from the plurality of images based on information about a terminal device installed within the target area; The point cloud processing device according to claim 7 .

15. The third three-dimensional point cloud is displayed on a display device to verify the accuracy of the first three-dimensional point cloud. The point cloud processing device according to claim 7 .

16. While the third three-dimensional point cloud is being displayed, the user can specify a position to be used for measuring the positional error between the first three-dimensional point cloud and the third three-dimensional point cloud. The point cloud processing device according to claim 15 .

17. The location is the center of the marker in the third three-dimensional point cloud. The point cloud processing device according to claim 16 .

18. A Z coordinate of the position is calculated based on the XY coordinates of the position, and a position error between the first three-dimensional point cloud and the third three-dimensional point cloud is measured based on the XY coordinates and the Z coordinates. The point cloud processing device according to claim 16 .

19. The third three-dimensional point cloud is displayed on the display device in response to an input from a user instructing accuracy verification of the first three-dimensional point cloud. The point cloud processing device according to claim 15 .

20. a mobile object equipped with a camera; a point cloud processing device including: a first point cloud generation unit that generates a first three-dimensional point cloud of a target area using a plurality of images captured by the camera equipped on the moving body; and a third point cloud generation unit that generates a third three-dimensional point cloud of a marker surrounding area in the target area using an image selected from the plurality of images based on information about a marker installed in the target area; A point cloud processing system consisting of:

Images