3D information generation system and program

Abstract

To improve measurement accuracy of three-dimensional information indicating a three-dimensional shape of an object.SOLUTION: A three-dimensional information generation system includes: an image-capturing section that captures an image of an object; a distance measuring section that measures a distance from a predetermined viewpoint to the object, which is acquired by irradiating the object with light from the predetermined viewpoint and receiving the light reflected from the object; and an operation section. A relative position between the image-capturing section and the distance measuring section is fixed. The operation section generates three-dimensional point group information indicating a position of a three-dimensional point group based on the distance measured by the distance measuring section, estimates a position and orientation of an own device at a certain time on the basis of a change in positions of feature points detected from the image captured by the image-capturing section between images captured at different time, and generates three-dimensional information indicating a three-dimensional shape of the object on the basis of the position of the three-dimensional point group obtained by correcting, on the basis of the position and orientation of the own device estimated at the certain time, the position of the three-dimensional point group indicated by the three-dimensional point group information generated on the basis of the distance measured by the distance measuring section at the certain time.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to a three-dimensional information generation system and a program. [Background technology]

[0002] Inspecting waterway tunnels at hydroelectric power plants requires information about the inside of the tunnel. This information includes 3D information to check the condition of the tunnel's walls and the cross-sectional shape of the tunnel's interior. In a typical waterway tunnel inspection, the water inside the tunnel is drained, and then a person enters the tunnel to visually inspect it. If an abnormality is found in the wall, the cross-sectional shape is measured on-site. It is efficient and desirable to acquire information about the inside of a tunnel while moving. Methods for acquiring information about the condition of the tunnel's interior walls while moving include capturing images of the walls with a camera. However, there was no technology available for acquiring 3D information about the cross-sectional shape of the tunnel's interior while moving.

[0003] If the cross-sectional shape inside the waterway tunnel continues to change (deformation), it could lose its functionality as a waterway tunnel and cause problems with power generation. For this reason, the waterway is drained once every few years, and inspectors walk through the waterway tunnel to conduct visual inspections and deformation surveys. However, visual inspections and deformation surveys when the water is drained take time, which can lead to issues such as reduced profits due to prolonged power generation outages. Furthermore, in some cases, it is difficult to comprehensively inspect long waterway tunnels spanning tens of kilometers by human eyes in a short period of time.

[0004] As a solution to the above, there are known techniques for obtaining internal information about the interior of a tunnel, such as a technique that uses images with distance information (Patent Document 1), a ground-mounted laser scanner in which a laser scanner is mounted on a ground-mounted vehicle, a technique that uses SLAM (Simultaneous Localization and Mapping) using a stereo camera (Patent Document 2), and a technique that uses self-position estimation using multiple sensors (Patent Document 3).

[0005] In the technology using images with distance information described in Patent Document 1, a device (water surface drone) that can flow unmanned along the wall of a water tunnel is equipped with a laser that can obtain inspection distances and a camera that can obtain images, and position information in the longitudinal direction of the tunnel is obtained while moving within the water tunnel.

[0006] For example, some laser scanners mounted on ground vehicles acquire three-dimensional information while manually moving the measuring device. This technology acquires three-dimensional information indicating the three-dimensional shape using a laser scanner mounted on the vehicle. While this technology simultaneously acquires the three-dimensional information and estimates the vehicle's own position using the acquired three-dimensional information, the accuracy of the self-position estimation based on the three-dimensional information alone is insufficient. Therefore, this technology creates a map while correcting the self-position estimated using the three-dimensional information based on the following information: This technology acquires the vehicle's position information by correcting coordinates measured by a total station (TS) or by using a global positioning system (GPS). For example, this technology acquires the vehicle's attitude information (position information and orientation information) using an inertial measurement unit (IMU) mounted on the vehicle.

[0007] The technology using SLAM described in Patent Document 2 simultaneously acquires image information and three-dimensional information. This technology adds three-dimensional information to feature points in the image information and creates a map while estimating the vehicle's own position.

[0008] In the technology using self-location estimation using multiple sensors described in Patent Document 3, the self-location is corrected by combining the self-location obtained by SLAM with the self-location obtained by multiple other odometry sensors. In this technology, each self-location estimation result is input to a weighting coefficient learning processing unit to correct the self-location. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] Patent Publication No. 2021-110695 [Patent Document 2] Japanese Patent Application Laid-Open No. 2018-194417 [Patent Document 3] Japanese Patent Publication No. 2020-160594 [Patent Document 4] Japanese Patent Publication No. 2020-002756 Summary of the Invention [Problem to be solved by the invention]

[0010] However, with the technology described in Patent Document 1, the laser scanner mounted on a ground vehicle, the technology described in Patent Document 2, and the technology described in Patent Document 3, measurements fail when using only a LiDAR (Light Detection and Ranging) sensor in spaces such as tunnels where the shape features change little in the direction of travel. Furthermore, these technologies have already established methods for estimating self-position by integrating a LiDAR sensor with another odometry, but when estimating self-position by integrating each piece of information, the error increases in a tunnel depending on the weighting coefficient of the LiDAR sensor, which is a factor in the error.

[0011] The technology described in Patent Document 1, which uses images with distance information, cannot obtain the cross-sectional shape of a waterway tunnel. Furthermore, although this technology uses multiple cameras to obtain images from different directions, the positional relationship between the images is unclear. When using a laser scanner mounted on a ground vehicle to correct 3D information, if multiple TSs installed at predetermined locations are used, the TSs to be used must be switched for each location. Furthermore, with this technology, the laser scanner must be stopped every few seconds to switch TSs, which is too time-consuming and impractical. The technology using SLAM described in Patent Document 2 is a mapping technology that is based on an existing map, and therefore cannot estimate a new position or create a new map. The technology using self-position estimation with multiple sensors described in Patent Document 3 estimates self-position by integrating image information and 3D information, which can cause errors in tunnels where the shape features change little in the direction of travel. In addition, this technology requires prior learning, making it difficult to train odometry that can be applied in tunnels. There is a demand for improving the accuracy of measuring three-dimensional information that indicates the three-dimensional shape of an object.

[0012] The present invention has been made in view of the above points, and provides a three-dimensional information generation system and program that can improve the measurement accuracy of three-dimensional information that indicates the three-dimensional shape of an object. [Means for solving the problem]

[0013] The present invention has been made to solve the above-mentioned problems, and one aspect of the present invention is an image capturing apparatus including an image capturing unit that captures an image of an object, a distance measuring unit that measures the distance from a predetermined viewpoint to the object by irradiating the object with light from the predetermined viewpoint and receiving light reflected from the object, and a calculation unit, wherein the relative positions of the image capturing unit and the distance measuring unit are fixed, and the calculation unit generates three-dimensional point cloud information indicating the positions of three-dimensional point clouds based on the distance measured by the distance measuring unit, and calculates three-dimensional point cloud information based on changes in positions of feature points detected from the images captured by the image capturing unit between the images captured at different times. Based on SLAM This is a 3D information generation system that estimates the position and orientation of its own device at a certain time, corrects the position of the 3D point cloud indicated by the 3D point cloud information generated based on the distance measured by the ranging unit at the certain time based on the position and orientation of the own device estimated at the certain time, and generates 3D information indicating the 3D shape of the object based on the corrected position of the 3D point cloud.

[0014] In addition, one aspect of the present invention is a three-dimensional information generation system, wherein the calculation unit calculates a feature point based on a change in position between the images captured at a first time and a second time, the feature point being detected from the images captured by the imaging unit. Based on SLAMThe position and orientation of the device at the second time are estimated, and the second time is a time immediately after the first time among a plurality of times at which the imaging unit captures an image of the object.

[0015] In one aspect of the present invention, a three-dimensional information generation system includes an imaging unit that captures an image of an object, and a distance measurement unit that measures the distance from a predetermined viewpoint to the object by irradiating the object with light from the predetermined viewpoint and receiving the light reflected from the object, the three-dimensional information generation unit including: a three-dimensional point cloud information generation step that generates three-dimensional point cloud information indicating the positions of three-dimensional point clouds based on the distances measured by the distance measurement unit; and a three-dimensional point cloud information generation step that generates three-dimensional point cloud information indicating the positions of three-dimensional point clouds based on changes in positions of feature points detected from the images captured by the imaging unit between the images captured at different times. Based on SLAM This is a program for executing a self-position estimation step of estimating the position and orientation of the device at a certain time, a correction step of correcting the position of the three-dimensional point cloud indicated by the three-dimensional point cloud information generated based on the distance measured by the ranging unit at the certain time based on the position and orientation of the device estimated at the certain time, and a three-dimensional information generation step of generating three-dimensional information indicating the three-dimensional shape of the object based on the corrected position of the three-dimensional point cloud. [Effects of the Invention]

[0016] According to the present invention, it is possible to improve the measurement accuracy of three-dimensional information that indicates the three-dimensional shape of an object. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a diagram illustrating an example of an appearance of an inspection device according to an embodiment of the present invention. [Figure 2] FIG. 1 is a diagram showing an example of an image of the inside of a waterway tunnel according to an embodiment of the present invention. [Figure 3] FIG. 10 is a diagram showing an example of an image of the inner wall of a waterway tunnel. [Figure 4] FIG. 1 is a diagram showing an example of the three-dimensional shape of the inner wall of a waterway tunnel. [Figure 5]1 is a diagram illustrating an example of a configuration of an inspection device according to an embodiment of the present invention. [Figure 6] 1 is a diagram illustrating an example of a hardware configuration of an inspection device according to an embodiment of the present invention. [Figure 7] 1 is a diagram illustrating an example of a functional configuration of a three-dimensional information generating device according to an embodiment of the present invention. [Figure 8] FIG. 4 is a diagram illustrating an example of a three-dimensional information generation process according to an embodiment of the present invention. [Figure 9] FIG. 4 is a diagram showing an example of feature points detected in an image captured by an imaging section according to an embodiment of the present invention. [Figure 10] FIG. 10 is a diagram illustrating an example of correction processing according to an embodiment of the present invention. [Figure 11] FIG. 2 is a diagram showing an example of a three-dimensional image according to an embodiment of the present invention. [Figure 12] FIG. 2 is a diagram showing an example of a three-dimensional image according to an embodiment of the present invention. [Figure 13] FIG. 10 is a diagram showing an example of a floating body inspection device according to a modified example of an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0018] (Embodiment) Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. Fig. 1 is a diagram showing an example of the appearance of an inspection device 1 according to this embodiment. The inspection device 1 is a device for inspecting the inside of a waterway tunnel. As an example, the inspection device 1 is carried by an operator and moves within the waterway tunnel.

[0019] The inspection device 1 acquires image information of the wall surface inside the aqueduct tunnel and three-dimensional information showing the three-dimensional shape of the wall surface while moving inside the aqueduct tunnel. The inspection device 1 acquires the three-dimensional information using a LiDAR (Light Detection and Ranging) sensor.

[0020] A waterway tunnel is a space where there are few changes in shape characteristics in the direction of travel. In a space where there are few changes in shape characteristics in the direction of travel, measurement fails when acquiring 3D information using only a LiDAR sensor as in the past. The reason for this is that there are few changes in the shape's irregularities (feature points) required for LiDAR SLAM.

[0021] Here, the characteristics of the interior of the waterway tunnel will be described with reference to Figures 2 to 4. Figure 2 is a diagram showing an example of an image captured of the interior of the waterway tunnel according to this embodiment. As shown in Figure 2, a wealth of image information is included, such as an image P1 of the joints and an image P2 of water stains.

[0022] Figure 3 shows an image of the inner wall of a water tunnel. The inner wall shown in Figure 3 is, as an example, the inner wall of a side. As shown in Figure 3, the inner wall of the water tunnel contains a wealth of image features, such as patterns caused by dirt on the concrete surface, such as water stains. Image features are features based on color contrast. These features are suitable for extraction as feature points that indicate a three-dimensional shape.

[0023] On the other hand, Figure 4 shows the three-dimensional shape of the inner wall of a waterway tunnel. The inner wall shown in Figure 4 is the inner wall of the ceiling, as an example. As shown in Figure 4, the three-dimensional shape of the inner wall of a waterway tunnel has few geometric features such as unevenness.

[0024] When the inspection device 1 acquires 3D information, it takes note of the fact that the inner walls of waterway tunnels contain a wealth of image features and uses SLAM (Simultaneous Localization and Mapping), which combines image information and 3D information. Visual SLAM deals with image information but has a low point cloud density. On the other hand, LiDAR SLAM deals with shape information but has a high point cloud density. The inspection device 1 improves the accuracy of the 3D information by combining the image information handled by Visual SLAM with 3D information representing the shape information handled by LiDAR SLAM.

[0025] The inspection device 1 acquires three-dimensional information indicating the three-dimensional shape of the target using LiDAR SLAM. The inspection device 1 estimates its own position based on image information acquired in synchronization with the three-dimensional information (Visual SLAM). The inspection device 1 corrects the acquired three-dimensional information based on the estimated self-position.

[0026] [Configuration of inspection equipment] 5 is a diagram showing an example of the configuration of the inspection device 1 according to this embodiment. The inspection device 1 includes a camera 2, a LiDAR sensor 3, lighting 4, a PC (Personal Computer) 5, a waterproof case 6, and a frame 7.

[0027] As an example, camera 2 is a 360-degree camera. Camera 2 captures color images of the inner wall of the aqueduct tunnel. Camera 2 captures the color images of the inner wall of the aqueduct tunnel as a video. In other words, camera 2 continues to capture the color images at a predetermined cycle. The angle of view of camera 2 may be less than 360 degrees, but it is preferable that it has a predetermined angle of view or more and be able to capture a wide range at the same time. Camera 2 may capture monochrome images, but it is preferable that it captures color images in order to obtain a wealth of image information.

[0028] The LiDAR sensor 3 is made up of two sensors, a LiDAR sensor 31 and a LiDAR sensor 32. The LiDAR sensor 31 and the LiDAR sensor 32 have equivalent functions, and therefore the LiDAR sensor 31 and the LiDAR sensor 32 will be described as representatives of the LiDAR sensor 3.

[0029] The LiDAR sensor 3 measures the distance from itself to each point on the inner wall surface of the waterway tunnel. The LiDAR sensor 3 is a TOF distance measurement sensor. That is, the LiDAR sensor 3 measures the distance from a predetermined viewpoint to an object by irradiating the object with light from the predetermined viewpoint and receiving the light reflected from the object. In this embodiment, the object is each point on the inner wall surface of the waterway tunnel.

[0030] The LiDAR sensor 3 includes a laser light source for irradiating light onto an object, a scanner mechanism and optical system for scanning the object, a light receiving element for receiving light reflected from the object, and a self-positioning mechanism for acquiring the self-position of the LiDAR sensor 3. The self-positioning mechanism is a global positioning system (GPS) or an inertial measurement unit (IMU), etc.

[0031] The distance measured by the LiDAR sensor 3 is used to generate a three-dimensional point cloud. In this embodiment, an example will be described in which the inspection device 1 is equipped with two LiDAR sensors 3, LiDAR sensor 31 and LiDAR sensor 32, in order to widen the range of the interior wall surface that is simultaneously measured. However, the inspection device 1 may be equipped with only one of the LiDAR sensor 31 and the LiDAR sensor 32. Furthermore, the inspection device 1 may be equipped with three or more LiDAR sensors 3.

[0032] The lighting 4 illuminates the object to be imaged by the camera 2. In other words, the lighting 4 illuminates the inner wall of the water tunnel. The lighting 4 illuminates the inner wall, thereby increasing the brightness of each pixel in the color image of the inner wall captured by the camera 2. The higher the brightness of each pixel in the color image captured by the camera 2, the more image information (features contained in the image) will be contained in the color image.

[0033] The lighting 4 preferably illuminates a wide range around the inspection device 1. In this embodiment, the lighting 4 is, for example, composed of three lighting units that emit illumination light in different directions (front, diagonally forward right, and diagonally forward left). In addition, if sufficient illumination is available around the inspection device 1, such as when lighting is installed in advance inside the waterway tunnel, the lighting 4 may be omitted from the configuration of the inspection device 1.

[0034] The PC 5 performs various calculations. As an example, the PC 5 is housed in a waterproof case 6, which is not shown in FIG. 5. The calculations performed by the PC 5 include generating a 3D point cloud based on the distance measured by the LiDAR sensor 3, estimating the vehicle's own position based on a color image captured by the camera 2, correcting the 3D point cloud based on the estimated vehicle's own position, and generating 3D information (information indicating the 3D shape of the inner wall of the water tunnel) based on the corrected 3D point cloud. Details of these calculations will be described later.

[0035] Each device included in the inspection device 1 is mounted on the frame 7. That is, the camera 2, the LiDAR sensor 3, the lighting 4, and the PC 5 housed in the waterproof case 6 are mounted on the frame 7. Here, the relative positions of the devices mounted on the frame 7 are fixed. Therefore, the relative positions of the camera 2 and the LiDAR sensor 3 are fixed. The relative positions include the distance between the camera 2 and the LiDAR sensor 3 and the angle between the orientations of the camera 2 and the LiDAR sensor 3.

[0036] The inspection device 1 moves inside the waterway tunnel by being held and carried by a worker at a predetermined portion of the frame 7. As the inspection device 1 moves, it acquires three-dimensional information about the inner wall of the waterway tunnel. The speed at which the inspection device 1 moves is, for example, about walking speed.

[0037] 6 is a diagram showing an example of the hardware configuration of the inspection device 1 according to this embodiment. The LiDAR sensors 31 and 32 are connected to the PC 5 by signal lines via a USB hub 8. The camera 2 is connected to the PC 5 by a signal line.

[0038] The LiDAR sensor 31 and the LiDAR sensor 32 are connected by a synchronization cable C1. This synchronizes the time at which the distance measurements are performed between the LiDAR sensor 31 and the LiDAR sensor 32.

[0039] Furthermore, a battery 91 and a battery 92 are connected to the LiDAR sensor 31 and the LiDAR sensor 32, respectively. Power is supplied to the LiDAR sensor 31 and the LiDAR sensor 32 from the battery 91 and the battery 92, respectively. The battery 91 and the battery 92 are mobile batteries. The PC 5 and the camera 2 are each supplied with power by their own internal batteries (not shown).

[0040] [Functional configuration of the three-dimensional information generation device 10] Here, the functional configuration of the inspection device 1 will be described as the functional configuration of a three-dimensional information generation device 10 with reference to FIG. Fig. 7 is a diagram showing an example of the functional configuration of a three-dimensional information generation device 10 according to this embodiment. The three-dimensional information generation device 10 shown in Fig. 7 corresponds to the inspection device 1 shown in Fig. 5. The three-dimensional information generation device 10 includes an imaging unit 11, a distance measurement unit 12, a calculation unit 13, and a storage unit 14.

[0041] The imaging unit 11 captures an image of the object (the inner wall of the waterway tunnel). The imaging unit 11 includes the camera 2 shown in FIG. The distance measuring unit 12 measures the distance from a predetermined viewpoint to the target object based on the TOF principle. The distance measuring unit 12 includes the LiDAR sensor 3 shown in FIG.

[0042] The calculation unit 13 performs various calculations. The calculation unit 13 includes a three-dimensional point cloud generation unit 130, a self-position estimation unit 131, a correction unit 132, and a three-dimensional information generation unit 133. Each functional unit included in the calculation unit 13 is realized by a CPU (Central Processing Unit) reading a program from a ROM (Read Only Memory) and executing the process.

[0043] The three-dimensional point cloud generating unit 130 generates three-dimensional point cloud information A1. The three-dimensional point cloud information A1 indicates the positions of the three-dimensional point cloud based on the distances measured by the distance measuring unit 12. In the three-dimensional point cloud information A1, the positions of the three-dimensional point cloud are indicated by the three-dimensional coordinates of each point included in the three-dimensional point cloud.

[0044] The self-position estimation unit 131 detects feature points from the image captured by the imaging unit 11, and estimates the position (self-position) of the own device (three-dimensional information generation device 10) based on SLAM.

[0045] The correction unit 132 corrects the position of the three-dimensional point cloud generated by the three-dimensional point cloud generation unit 130. The three-dimensional information generator 133 generates three-dimensional information based on the positions of the three-dimensional point group. The three-dimensional information indicates the three-dimensional shape of the object.

[0046] The storage unit 14 stores various types of information. The information stored in the storage unit 14 includes three-dimensional point cloud information A1 and image information A2. The storage unit 14 is configured using a storage device such as a semiconductor storage device. The storage unit 14 may also be configured using a magnetic hard disk drive as long as it has a vibration-proof structure.

[0047] [3D information generation processing] Here, with reference to FIG. 8, the three-dimensional information generation process in which the three-dimensional information generation device 10 generates three-dimensional information will be described. FIG. 8 is a diagram showing an example of the three-dimensional information generation process according to this embodiment. The three-dimensional information generation process shown in FIG. 8 is started when the PC 5 (the calculation unit 13) receives an operation to start imaging the target object (the inner wall of the waterway tunnel). This operation is performed by an operator who is the user of the three-dimensional information generation device 10 from an operation unit (not shown in FIG. 5). Furthermore, at least the processes of step S10, step S20, and step S30 are executed while the three-dimensional information generation device 10 is moving.

[0048] Step S10: The distance measuring unit 12 acquires distance information. The distance information is information indicating the distance from a predetermined viewpoint to an object. Here, the distance measuring unit 12 measures the distance from the predetermined viewpoint to the object by irradiating the object with light from the predetermined viewpoint and receiving the light reflected from the object. The distance measuring unit 12 acquires the distance information as a measurement result. The distance measuring unit 12 supplies the acquired distance information to the 3D point cloud generating unit 130. Here, the distance measuring unit 12 includes the time when the distance was measured (measurement time) in the distance information along with the measurement result.

[0049] Step S20: The three-dimensional point cloud generation unit 130 generates three-dimensional point cloud information A1 based on the distance information acquired from the distance measurement unit 12. Each point constituting the three-dimensional point cloud whose position is indicated by the three-dimensional point cloud information A1 corresponds to each point on the surface of the object onto which the distance measurement unit 12 irradiates light. In the three-dimensional point cloud information A1, the three-dimensional coordinates of each point included in the three-dimensional point cloud are indicated by, for example, three-dimensional Cartesian coordinates. Here, the three-dimensional point cloud generation unit 130 generates the three-dimensional point cloud information A1 by pairing the position of the three-dimensional point cloud with the measurement time indicated by the distance information. The three-dimensional point cloud generation unit 130 stores the generated three-dimensional point cloud information A1 in the storage unit 14.

[0050] The three-dimensional point cloud information A1 generated based on the distance information acquired from the distance measurement unit 12 indicates the positions of the three-dimensional point cloud generated based on the distance measured by the LiDAR sensor 3. That is, in this embodiment, the three-dimensional point cloud information A1 is information indicating the positions of the three-dimensional point cloud generated based on the result of measuring the distance from a predetermined viewpoint to an object, which is acquired by irradiating the object with light from the predetermined viewpoint and receiving the light reflected from the object.

[0051] Point clouds (depth images) obtained from Visual SLAM or stereo images are generally susceptible to the effects of lighting and blurring of images captured by a camera, and tend to be wavy. As a result, point clouds obtained from Visual SLAM or stereo images tend to have large errors in the 3D information indicating the shape of the object represented by the point cloud. In this embodiment, the accuracy of the 3D information indicating the shape of the object is improved by using a 3D point cloud generated based on the distance measured by the LiDAR sensor 3 as the 3D point cloud.

[0052] Step S30: The imaging unit 11 captures an image of the object (the inner wall of the waterway tunnel). The imaging unit 11 generates image information A2 by combining the captured image with the image capture time. The imaging unit 11 stores the generated image information A2 in the 3D point cloud generation unit 130.

[0053] Step S40: The calculation unit 13 determines whether or not the image capturing of the object has been completed. When the calculation unit 13 receives an operation indicating that the image capturing has been completed, the calculation unit 13 determines that the image capturing has been completed. When the calculation unit 13 determines that the image capturing has been completed (step S40; YES), the calculation unit 13 executes the process of step S50. On the other hand, when the calculation unit 13 determines that the image capturing has not been completed (step S40; NO), the calculation unit 13 executes the process of step S10 again.

[0054] Therefore, the processes of steps S10, S20, and S30 are repeatedly executed while the three-dimensional information generation device 10 moves until the imaging is completed. The repetition period is, for example, the period in which the imaging unit 11 captures one image.

[0055] Step S50: The self-position estimation unit 131 estimates its own position based on SLAM. The self-position estimation unit 131 reads image information A2 from the storage unit 14. The self-position estimation unit 131 detects feature points of the object (the inner wall of the waterway tunnel) from the read image information A2. Here, the self-position estimation unit 131 detects feature points from the image for each image capture time.

[0056] Fig. 9 shows an example of feature points P4 detected in image P3 captured by imaging unit 11. Imaging unit 11 captures an image using camera 2, which is a 360-degree camera. In image P3 shown in Fig. 9, the inner wall of a water tunnel surrounding three-dimensional information generating device 10 is captured over a 360-degree angle of view. In Fig. 9, multiple feature points P4 are each indicated by a square mark.

[0057] Returning to FIG. 8, the description of the three-dimensional information generation process will be continued. The self-position estimation unit 131 estimates the self-position of the 3D information generation device 10 based on the detected feature points and SLAM. The self-position estimation unit 131 estimates the position of the 3D information generation device 10 at a certain imaging time as its self-position based on changes in the positions of the feature points between different imaging times. Here, the self-position estimated by the self-position estimation unit 131 includes the relative position and relative orientation of the device itself. The relative position and relative orientation are the relative position and relative orientation of the position of a feature point extracted from an image captured at a certain imaging time to the position of a feature point extracted from an image captured at a different imaging time. The relative position and relative orientation are indicated by vector information.

[0058] A certain imaging time is, for example, time t+1, and another imaging time is, for example, time t. Here, time t+1 is the time immediately after time t among multiple imaging times at which the imaging unit 11 images the object. As described above, the correction unit 132 estimates the position and orientation of the device at time t+1 based on the change in position of feature points detected from the images captured by the imaging unit 11 between the images captured at time t and time t+1. The position and orientation of the device are collectively referred to as the attitude of the device or its own position.

[0059] As an example, the self-location estimation unit 131 uses the open-source Open-VSLAM as SLAM. Open-VSLAM is a Visual SLAM based on ORB (Oriented Fast and Rotated Brief) feature points. Note that the self-location estimation unit 131 may use a Visual SLAM other than Open-VSLAM.

[0060] Step S60: The correction unit 132 corrects the three-dimensional point group information A1 based on the image information A2. The correction process by the correction unit 132 will now be described with reference to FIG. 10. FIG. 10 is a diagram showing an example of the correction process according to this embodiment. The steps shown in FIG. 10 are executed in step S60 shown in FIG. 8.

[0061] Step S110: The correction unit 132 synchronizes the data acquisition time between the image information A2 and the three-dimensional point cloud information A1. The correction unit 132 associates the image capture time of the image included in the image information A2 with the measurement time included in the three-dimensional point cloud information A1.

[0062] Step S120: The correction unit 132 starts the process of correcting the position of the three-dimensional point cloud for each data acquisition time.

[0063] Step S130: The correction unit 132 corrects the position of the 3D point cloud based on the estimation result of the self-position by the self-position estimation unit 131. At time t and time t+1, the correction unit 132 corrects the position of the 3D point cloud indicated by the 3D point cloud information A1 at time t+1 using the estimation result of the self-position by the self-position estimation unit 131 based on the image information A2. As described above, the estimation result of the self-position is vector information indicating the relative position and relative orientation of the 3D information generation device 10 that changed between time t and time t+1.

[0064] As described above, the correction unit 132 corrects the position of the three-dimensional point cloud indicated by the three-dimensional point cloud information A1 generated based on the distance measured by the distance measuring unit 12 at a certain time, based on the position of the device itself (the three-dimensional information generating device 10) estimated at that certain time.

[0065] Step S140: The correction unit 132 corrects the positions of the three-dimensional point groups indicated by the three-dimensional point group information A1. Here, the correction unit 132 corrects the positions of the three-dimensional point groups indicated by the three-dimensional point group information A1 corrected at time t and time t+1 so that the positions of the three-dimensional point groups match (also referred to as matching correction). Correcting the positions of the three-dimensional point groups is also referred to as aligning the positions of the three-dimensional point groups.

[0066] The correction unit 132 corrects the positions of the three-dimensional point groups based on, for example, ICP (Iterative Closest Point). Note that the correction unit 132 may correct the positions of the three-dimensional point groups based on a registration algorithm other than ICP.

[0067] Step S150: The correction unit 132 ends the process of correcting the position of the three-dimensional point cloud for each data acquisition time. This completes the correction process by the correction unit 132.

[0068] In the present embodiment, an example has been described in which the self-position estimation unit 131 estimates its own position using, as relative position information, the relative position and relative orientation from the position of a feature point at time t among multiple image capture times at which the image capture unit 11 captures an image of the target object to the position of the feature point at time t+1, which is the time immediately after time t, but the present invention is not limited to this. For example, the self-position estimation unit 131 may estimate the position and orientation of the device itself based on the relative position and relative orientation from the position of the feature point at an initial time (initial position) to the position of the feature point at a certain time.

[0069] When the position and orientation of the device itself are estimated based on the relative position and relative orientation from the initial position, it is thought that a larger amount of error accumulates in the estimated self-position compared to when the position and orientation are estimated based on the relative position and relative orientation between a certain time and the time immediately following that certain time. If a large amount of error accumulates, the correction (alignment) process becomes difficult and the correction (alignment) error becomes larger in the process of correcting the positions of the 3D point clouds in step S140 (for example, correction by ICP).

[0070] As described above, the self-position estimation unit 131 estimates the position of the device itself (the three-dimensional information generating device 10) at a certain time based on the change in the position of feature points detected from images captured by the imaging unit 11 between the images captured at different times.

[0071] Returning to FIG. 8, the description of the three-dimensional information generation process will be continued. Step S70: The three-dimensional information generator 133 generates three-dimensional information based on the corrected three-dimensional point cloud positions. The corrected three-dimensional point cloud positions here are positions corrected by correction based on the self-position estimated based on the image information A2 in step S130 and correction between the three-dimensional point cloud positions in step S140. In the three-dimensional information, the shape of the target object is indicated by the corrected three-dimensional point cloud positions.

[0072] The three-dimensional information generation unit 133 outputs the generated three-dimensional information. For example, the three-dimensional information generation unit 133 outputs the three-dimensional information to a display device. The display device displays a three-dimensional image showing the shape of the object (the inner wall of the waterway tunnel) based on the three-dimensional information. As a result, the shape of the object (the inner wall of the waterway tunnel) is visualized as a three-dimensional image. The three-dimensional information generating unit 133 may output the three-dimensional information to the storage unit 14 or an external storage device for storage.

[0073] The three-dimensional information generating unit 133 may include, in the three-dimensional image, position information and orientation information for any point on the shape of the object. The position information and orientation information are included in the three-dimensional image based on three-dimensional point cloud data. The position information and orientation information indicate the position and direction of a point using three-dimensional coordinates. This allows the user of the three-dimensional information generating device 10 to obtain position information and orientation information based on the three-dimensional point cloud data by simply selecting a point on the three-dimensional image displayed by the display device. With this, the calculation unit 13 ends the three-dimensional information generation process.

[0074] [3D image] 11 and 12, a specific example of a three-dimensional image generated by the three-dimensional information generating device 10 will be described. FIG. 11 is a diagram showing an example of a three-dimensional image P5 according to this embodiment. The three-dimensional image P5 is a three-dimensional image of the inner wall of a water tunnel generated by the three-dimensional information generating device 10. In the three-dimensional image P5, the shape of the inner wall is shown by point cloud data.

[0075] 12 is a diagram showing an example of a three-dimensional image P6 according to this embodiment. The three-dimensional image P6 is a three-dimensional image (bird's-eye view) of the entire aqueduct tunnel generated by moving the three-dimensional information generating device 10 inside the aqueduct tunnel. In the three-dimensional image P6, the shape of the inner wall of the entire aqueduct tunnel is shown using point cloud data.

[0076] In the present embodiment, an example has been described in which the calculation unit 13 includes each functional unit for performing the 3D information generation process, but this is not limiting. Any one or more of the functional units for performing the 3D information generation process may be included in a calculation device separate from the 3D information generation device 10. That is, any one or more of the 3D point cloud generation unit 130, self-position estimation unit 131, correction unit 132, and 3D information generation unit 133 may be included in the calculation device. In this case, the 3D information generation device 10 includes a communication unit for communicating with the calculation device. The calculation device is, for example, a terminal device or a server. When one or more of the functional units for performing the three-dimensional information generation process are provided in one or more arithmetic devices separate from the three-dimensional information generation device 10, the one or more arithmetic devices and the three-dimensional information generation device 10 constitute a three-dimensional information generation system.

[0077] (Variation) In the present embodiment, an example in which the inspection device 1 (three-dimensional information generating device 10) is carried by an operator has been described, but the present invention is not limited to this. The inspection device may be placed on a floating body and moved within the waterway tunnel. Here, referring to Fig. 13, a modified example of this embodiment will be described, in which the inspection device is mounted on a floating body. Fig. 20 is a diagram showing an example of a floating body inspection device 20 according to this modified example. The floating body inspection device 20 comprises an inspection device 21, a floating body 22, a parachute 23, and a tail 24. The configuration of the floating body inspection device 20 is based on the configuration described in Patent Document 4, for example.

[0078] The inspection device 21 has the same configuration as the inspection device 1 (FIG. 1) of this embodiment, and therefore a description thereof will be omitted. Note that the shape of the frame may be changed from the frame 7 shown in FIG. The floating body 22 floats on the water surface of the waterway tunnel. The parachute 23 receives a propulsive force from the water in the waterway tunnel. The floating body inspection device 20 moves through the waterway tunnel using this propulsive force. The tail 24 is a rope-like member connected to the pillar 25 on the side opposite to the parachute 23. The parachute 23 and tail 24 are connected to the float 22 via the pillar 25. In the floating body inspection device 20, the propulsive force received from the water is greatest for the tail 24, the float 22, and the parachute 23 in that order. Therefore, the parachute 23 pulls the float 22 downstream, and the tail 24 pulls the float 22 upstream. This maintains the posture of the float 22 against the water current.

[0079] The floating inspection device 20 can inspect the inner wall of a waterway tunnel by moving through the waterway tunnel while flowing through it, without having to drain the water from the tunnel. Because the floating inspection device 20 moves while floating on the water using the float 22, the influence of vibrations of the floating inspection device 20 itself can be reduced when capturing images of the inner wall of the waterway tunnel and measuring the distance to the inner wall.

[0080] The inspection device may be mounted on a mobile device and moved within the waterway tunnel. The mobile device may be self-propelled or may be towed or pushed by an operator. The mobile device may be a vehicle that travels automatically or may be driven by an operator. The mobile device may be, for example, a vehicle.

[0081] As described above, the three-dimensional information generation system according to this embodiment (three-dimensional information generation device 10 in this embodiment) includes the imaging unit 11, the distance measurement unit 12, and the calculation unit 13. The imaging unit 11 captures an image of an object (in this embodiment, the inner wall of a water tunnel). The distance measurement unit 12 measures the distance from a specified viewpoint to an object (in this embodiment, the inner wall of the waterway tunnel) by irradiating light onto the object (in this embodiment, the inner wall of the waterway tunnel) from the specified viewpoint and receiving the light reflected from the object (in this embodiment, the inner wall of the waterway tunnel). The relative positions of the imaging unit 11 and the distance measuring unit 12 are fixed. The calculation unit 13 generates three-dimensional point cloud information A1 indicating the position of the three-dimensional point cloud based on the distance measured by the ranging unit 12 (three-dimensional point cloud generation unit 130), estimates the position and orientation of the device itself (three-dimensional information generating device 10) at a certain time based on the change in position of feature points detected from images captured by the imaging unit 11 between images captured at different times (self-position estimation unit 131), corrects the position of the three-dimensional point cloud indicated by the three-dimensional point cloud information A1 generated based on the distance measured by the ranging unit 12 at the certain time based on the position and orientation of the device itself (three-dimensional information generating device 10) estimated at the certain time (correction unit 132), and generates three-dimensional information indicating the three-dimensional shape of the object (in this embodiment, the inner wall of the waterway tunnel) based on the position of the corrected three-dimensional point cloud (three-dimensional information generation unit 133).

[0082] With this configuration, the 3D information generation device 10 according to this embodiment can correct the position of the 3D point cloud based on the position and orientation of the device itself estimated based on image information, thereby improving the measurement accuracy of 3D information indicating the 3D shape of the object. The 3D information generation device 10 according to this embodiment improves the accuracy of the 3D information by combining 3D information (LiDAR SLAM) indicating the 3D shape of the object with image information (Visual SLAM).

[0083] The 3D information generating device 10 according to this embodiment is suitable for use in inspecting the inner walls of a waterway tunnel, as described in this embodiment. It is possible to inspect a waterway tunnel while moving, regardless of whether the waterway tunnel is drained or running. Inspection using the 3D information generating device 10 makes it possible to acquire a 3D image with location information and orientation information. By selecting a point on the 3D image, it is possible to acquire coordinate information for that point based on the 3D point cloud data. Furthermore, inspection using the 3D information generating device 10 makes it possible to acquire information showing the 3D cross-sectional shape of the waterway tunnel at any point.

[0084] Furthermore, in the three-dimensional information generating device 10 according to this embodiment, the calculation unit 13 estimates (self-position estimation unit 131) the position of the device itself (the three-dimensional information generating device 10) at a second time based on the change in position of feature points detected from the image captured by the imaging unit 11 between the images captured at a first time (time t) and a second time (time t+1), and the second time (time t+1) is the time immediately after the first time (time t) among the multiple times at which the imaging unit 11 captures an image of the target object (in this embodiment, the inner wall of the waterway tunnel).

[0085] With this configuration, the three-dimensional information generating device 10 according to this embodiment can estimate the position and orientation of the device itself based on the relative position and relative orientation between a certain time and the time immediately following that certain time, thereby improving the accuracy of position estimation compared to when the position and orientation of the device itself is estimated based on the relative position and relative orientation from the initial position.

[0086] In this embodiment, an example in which the inspection device 1 is used to inspect a waterway tunnel has been described, but the present invention is not limited to this. The inspection device 1 may also be used to inspect a road tunnel. The inspection device 1 may also be used to inspect places without a ceiling. Places without a ceiling include river banks, road curbs, etc.

[0087] Note that a portion of the 3D information generation device 10 in the above-described embodiment, such as the calculation unit 13, may be implemented by a computer. In this case, a program for implementing this control function may be recorded on a computer-readable recording medium, and the program may be loaded into a computer system and executed. Note that the term "computer system" refers to a computer system built into the 3D information generation device 10, including hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into a computer system. Furthermore, the term "computer-readable recording medium" may also include media that dynamically store programs for a short period of time, such as communication lines used when transmitting programs via networks such as the Internet or telephone lines, or media that store programs for a fixed period of time, such as volatile memory within a computer system that serves as a server or client. The program may be a program that implements part of the above-described functions, or may be a program that can implement the above-described functions in combination with a program already stored in the computer system. Furthermore, part or all of the three-dimensional information generation device 10 in the above-described embodiment may be realized as an integrated circuit such as an LSI (Large Scale Integration). Each functional block of the three-dimensional information generation device 10 may be individually implemented as a processor, or part or all of the blocks may be integrated into a processor. The integrated circuit implementation method is not limited to LSI, and may be implemented using a dedicated circuit or a general-purpose processor. Furthermore, if an integrated circuit implementation technology that can replace LSI emerges due to advances in semiconductor technology, an integrated circuit based on that technology may be used.

[0088] One embodiment of the present invention has been described in detail above with reference to the drawings, but the specific configuration is not limited to that described above, and various design changes and the like are possible within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0089] 10...3D information generating device, 11...imaging unit, 12...distance measuring unit, 13...calculation unit, 130...3D point cloud generating unit, 131...self-position estimating unit, 132...correction unit, 133...3D information generating unit, 1...inspection device, 2...camera, 3...LiDAR sensor, 5...PC

Claims

1. an imaging unit that captures an image of an object; a distance measuring unit that measures a distance from a predetermined viewpoint to the object by irradiating the object with light from the predetermined viewpoint and receiving light reflected from the object; Calculation unit and Equipped with The relative positions of the imaging unit and the distance measuring unit are fixed, The calculation unit generating three-dimensional point cloud information indicating the positions of the three-dimensional point cloud based on the distance measured by the distance measuring unit; estimating the position and orientation of the device at a certain time based on SLAM based on changes in positions of feature points detected from the images captured by the imaging unit between the images captured at different times; correcting the position of the three-dimensional point cloud indicated by the three-dimensional point cloud information generated based on the distance measured by the distance measuring unit at the certain time based on the position and orientation of the own device estimated at the certain time; Generate three-dimensional information indicating the three-dimensional shape of the object based on the corrected positions of the three-dimensional point cloud. 3D information generation system.

2. the calculation unit estimates a position and orientation of the device at the second time based on SLAM based on a change in position of a feature point detected from the image captured by the imaging unit between the images captured at a first time and a second time, The second time is a time immediately after the first time among a plurality of times at which the imaging unit captures an image of the object. The three-dimensional information generation system according to claim 1 .

3. an imaging unit that captures an image of an object; a distance measuring unit that measures the distance from a predetermined viewpoint to the object by irradiating the object with light from the predetermined viewpoint and receiving the light reflected from the object; A computer of a three-dimensional information generation system comprising: a three-dimensional point cloud information generating step of generating three-dimensional point cloud information indicating positions of three-dimensional point clouds based on the distances measured by the distance measuring unit; a self-position estimation step of estimating a position and orientation of the device at a certain time based on SLAM based on changes in positions of feature points detected from the images captured by the imaging unit between the images captured at different times; a correction step of correcting a position of the three-dimensional point cloud indicated by the three-dimensional point cloud information generated based on the distance measured by the distance measuring unit at the certain time, based on the position and orientation of the own device estimated at the certain time; a three-dimensional information generating step of generating three-dimensional information indicating a three-dimensional shape of the object based on the corrected positions of the three-dimensional point cloud; A program to execute.

Images