Information processing device, moving body, information processing method, and computer program

Abstract

To provide an information processing device capable of highly accurately generating the three-dimensional map of an environment.SOLUTION: The information processing device for estimating the three-dimensional position attitude of a moving body includes a two-dimensional point group input part for inputting the two-dimensional point group data of the environment from a first sensor for measuring the two-dimensional point group data, a captured image input part for inputting the captured image of the environment captured by an imaging device, a two-dimensional position attitude estimation part for estimating a two-dimensional position attitude representing the position attitude in the horizontal direction of the first sensor on the basis of the two-dimensional point group data, and a three-dimensional position attitude estimation part for estimating the three-dimensional position attitude of the moving body on the basis of the two-dimensional position attitude of the first sensor and the position attitude in a height direction obtained from the captured image.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an information processing device, a mobile object, an information processing method, a computer program, and the like. [Background technology]

[0002] There is a known technology for generating a 3D map of an environment based on image information, distance information, etc. acquired by a sensor. In Non-Patent Document 1, a 3D map of an environment is generated using the position and orientation of a camera estimated based on an image captured by a camera on a moving object and measurement data from a 3D distance sensor. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] H. Oleynikova, Z. Taylor, M. Fehr, R. Siegwart, and J. Nieto, "Voxblox: Incremental 3D Euclidean signed distance fields for on-board MAV planning," in IEEE / RSJ Int. Conf. on Intelligent Robots and Systems, 2017, pp. 1366-1373. Summary of the Invention [Problem to be solved by the invention]

[0004] However, the method of Non-Patent Document 1 has a problem in that if the accuracy of the estimated position and orientation is low, the accuracy of the 3D map generated using the estimated position and orientation also decreases.

[0005] In view of the above, an object of the present invention is to provide an information processing device capable of generating a 3D map of an environment with high accuracy. [Means for solving the problem]

[0006] An information processing device that estimates a three-dimensional position and orientation of a moving object, a two-dimensional point cloud input unit that inputs two-dimensional point cloud data of the environment from a first sensor that measures two-dimensional point cloud data; a captured image input unit for inputting a captured image of an environment captured by the imaging device; a two-dimensional position and orientation estimation unit that estimates a two-dimensional position and orientation that represents a position and orientation in a horizontal direction of the first sensor based on the two-dimensional point cloud data; a three-dimensional position and orientation estimation unit that estimates a three-dimensional position and orientation of the moving object based on the two-dimensional position and orientation of the first sensor and a position and orientation in a height direction obtained from the captured image; The present invention is characterized by having the following. Effect of the Invention

[0007] According to the present invention, it is possible to provide an information processing device capable of generating a 3D map of an environment with high accuracy. [Brief description of the drawings]

[0008] [Figure 1] 1 is a schematic diagram showing an example of use of a moving object in the first embodiment. FIG. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device 100 in FIG. [Diagram 3] 2 is a functional block diagram showing an example of a data flow configuration of the information processing device according to the first embodiment. FIG. [Figure 4] 4 is a flowchart showing an example of an information processing method executed by the information processing device 100 according to the first embodiment. [Diagram 5] 5 is a flowchart showing an example of a processing procedure for estimating a three-dimensional position and orientation in step S104 of FIG. 4. [Figure 6] FIG. 11 is a functional block diagram showing an example of the data flow configuration of an information processing device according to the second embodiment. [Figure 7] FIG. 11 is a functional block diagram showing an example of the data flow configuration of an information processing device according to a third embodiment. [Figure 8]FIG. 13 is a diagram showing an example of a GUI whose display is controlled by an information processing device according to a fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiment. In each drawing, the same members or elements are given the same reference numerals, and duplicated descriptions are omitted or simplified.

[0010] <Embodiment 1> In the first embodiment, a case will be described in which a moving object travels on a flat floor surface, and an imaging device mounted on the moving object is a stereo camera that captures stereo images.

[0011] 1 is a schematic diagram showing an example of use of a moving body in embodiment 1. The moving body 1, which is equipped with an information processing device 100, a first sensor 11, an imaging device 12, and a second sensor 13, autonomously travels based on a three-dimensional map created in advance. The information processing device 100 has a function of estimating the three-dimensional position and orientation of the moving body 1.

[0012] A user 101 uses a portable PC terminal such as a tablet 102 to instruct the moving object 1 to create a map of the environment. The user 101 can also use the tablet 102 to check the created map of the environment and instruct the moving object 1 to start autonomous traveling.

[0013] In addition, the moving body 1 has a drive control unit such as an ECU (Electronic Control Unit) (not shown) that performs autonomous driving based on the three-dimensional position and orientation of the moving body estimated by a three-dimensional position and orientation estimation unit described below and the three-dimensional map updated by a three-dimensional map update unit.

[0014] The mobile object is, for example, an Autonomous Mobile Robot (AMR) such as a cleaning robot, an Automated Guided Vehicle (AGV), an autonomous vehicle, a drone, etc. In this embodiment, a case in which the information processing device is applied to an AMR will be described.

[0015] Fig. 2 is a diagram showing an example of a hardware configuration of the information processing device 100 shown in Fig. 1. Reference numeral 21 denotes a CPU, which controls various devices connected to a system bus 29 and executes programs. Reference numeral 22 denotes a ROM, which stores a BIOS program and a boot program.

[0016] Reference numeral 23 denotes a RAM, which is used as a main storage device for the CPU 21. Reference numeral 24 denotes an external memory, which stores computer programs to be processed by the CPU 21 as a computer. An input unit 25 performs processing related to input of information, etc., via a keyboard, mouse, etc.

[0017] The display unit 26 generates display control signals for the calculation results, maps, images, etc. of the information processing device 100 in accordance with instructions from 21, and displays them on a display device such as a tablet 102 via the communication I / F. The display device may be of any type, such as a liquid crystal display device, a projector, or an LED indicator.

[0018] Reference numeral 27 denotes a communication interface, which communicates information via a network, and the communication interface may be Ethernet, or any type of interface such as USB, serial communication, wireless communication, etc. Reference numeral 28 denotes an I / O, which inputs data from a sensor.

[0019] In this embodiment, an example will be described in which the information processing device 100 is mounted on a mobile object. However, a part or the whole of the information processing device 100 can also be provided on a server connected via a network.

[0020] Fig. 3 is a functional block diagram showing an example of the functional configuration of the information processing device in embodiment 1. Note that some of the functional blocks shown in Fig. 3 are realized by causing a CPU serving as a computer included in the information processing device to execute a computer program stored in a memory serving as a storage medium.

[0021] However, some or all of them may be realized by hardware. A dedicated circuit (ASIC) or a processor (reconfigurable processor, DSP) may be used as the hardware. Furthermore, the functional blocks shown in Fig. 3 do not have to be built in the same housing, and may be configured by separate devices connected to each other via signal paths. The above explanation regarding Fig. 3 also applies to Figs. 6 and 7.

[0022] Reference numeral 111 denotes a two-dimensional point cloud input unit that inputs two-dimensional point cloud data of the environment to the information processing device 100 via the first sensor 11. That is, the two-dimensional point cloud input unit 111 inputs the two-dimensional point cloud data of the environment from the first sensor 11 that measures the two-dimensional point cloud data.

[0023] The first sensor 11 in this embodiment is a 2D LiDAR (Light Detection and Ranging). The 2D LiDAR acquires two-dimensional point cloud data by scanning a laser beam parallel to the floor surface on which the moving object 1 travels, and inputs the acquired two-dimensional point cloud data to the two-dimensional point cloud input unit 111.

[0024] Reference numeral 112 denotes a captured image input unit that inputs a captured image of the environment surrounding the moving object 1 to the information processing device 100 via the imaging device 12. That is, the captured image input unit 112 inputs a captured image of the environment captured by the imaging device 12.

[0025] In the first embodiment, the imaging device 12 is a stereo camera. The stereo camera captures a stereo image made up of two monochrome images, and inputs the captured stereo image to the captured image input unit 112.

[0026] A three-dimensional point cloud input unit 113 inputs three-dimensional point cloud data of the environment to the information processing device 100 from a second sensor 13 that measures three-dimensional point cloud data. In this embodiment, the second sensor 13 is a ToF (Time of Flight) sensor. The ToF sensor acquires three-dimensional point cloud data by scanning the sensor in the horizontal and vertical directions. The acquired three-dimensional point cloud data is input to the three-dimensional point cloud input unit 113.

[0027] Reference numeral 114 denotes a two-dimensional position and orientation estimation unit that estimates the two-dimensional position and orientation (position and / or orientation on a two-dimensional plane) of the first sensor 11 based on the two-dimensional point cloud data input by the two-dimensional point cloud input unit 111.

[0028] That is, the two-dimensional position and orientation estimation unit 114 estimates a two-dimensional position and orientation that represents the position and orientation of the first sensor in the horizontal direction based on the two-dimensional point cloud data. As described above, in this embodiment, the two-dimensional plane is the floor surface on which the moving body 1 as an AMR travels, and the two-dimensional position and orientation estimation unit 114 estimates the two-dimensional position and orientation of the moving body 1 on the floor surface.

[0029] A three-dimensional position and orientation estimation unit 115 estimates a three-dimensional position and orientation (position and / or orientation in three-dimensional space) of the second sensor 13 based on the two-dimensional position and orientation estimated by the two-dimensional position and orientation estimation unit 114 and a captured image captured by the imaging device. That is, the three-dimensional position and orientation estimation unit 115 estimates the three-dimensional position and orientation of the second sensor 13, i.e., the three-dimensional position and orientation of the moving body, based on the two-dimensional position and orientation of the first sensor and the position and orientation in the height direction obtained from the captured image.

[0030] Reference numeral 116 denotes a 3D map update unit that updates 3D map data indicating the arrangement of objects in the environment stored in storage unit 118, based on the 3D point cloud data input by 3D point cloud input unit 113 and the 3D position and orientation estimated by 3D position and orientation estimation unit 115. That is, the 3D map update unit 116 updates the 3D map in storage unit 118, based on the 3D position and orientation estimated by the 3D position and orientation estimation unit, and the 3D point cloud data.

[0031] Reference numeral 117 denotes a display control unit that displays the creation status and creation results of the 3D map on the tablet 102 to the user 101 in Fig. 1. Reference numeral 118 denotes a storage unit that stores the 3D map updated by the 3D map update unit 116. In other words, the storage unit 118 stores a 3D map that indicates the arrangement of objects in the environment.

[0032] It is assumed that the first sensor 11, the imaging device 12, and the second sensor 13 are fixed on the moving body 1, and that the specifications of the respective positional relationships, the sensor orientations, the measurement and imaging ranges, the resolutions, etc. are known. In other words, the two-dimensional position and orientation of the first sensor 11, the three-dimensional position and orientation of the imaging device 12, and the three-dimensional position and orientation of the second sensor 13 measured at the same timing can be mutually converted.

[0033] In this embodiment, the three-dimensional point group data refers to a point cloud, and each point has three parameters of three-dimensional coordinates (X, Y, Z). The Z axis of the world coordinate system is defined as the direction perpendicular to the floor surface, and the X and Y axes of the world coordinate system are defined as the directions parallel to the floor surface. The three-dimensional position and orientation has three parameters of the position of the moving object 1, i.e., the three-dimensional coordinates (X, Y, Z), and three parameters of the orientation (φ, θ, ψ).

[0034] The direction φ is the roll angle, θ is the pitch angle, and ψ is the yaw angle (azimuth angle, the direction of the X-axis of the moving body in the world coordinate system). In this embodiment, the moving body 1 runs on the floor surface and does not move in the Z-axis direction (height direction), so φ=θ=0° is fixed.

[0035] The two-dimensional point cloud data has two parameters, two-dimensional coordinates (X, Y) based on the three-dimensional point cloud data, excluding the Z-axis data. The horizontal direction in the world coordinate system is the X-axis and Y-axis. The two-dimensional position and orientation is expressed by the position of the moving object 1, i.e., the two-dimensional coordinates (X, Y), and the direction ψ (azimuth angle).

[0036] 4 is a flowchart showing an example of an information processing method executed by the information processing device 100 in embodiment 1, which illustrates an information processing method for estimating a three-dimensional position and orientation of a moving object. Note that the CPU as a computer in the information processing device 100 executes a computer program stored in a memory, thereby sequentially performing the operations of the steps in the flowchart in FIG.

[0037] In the following description, the processing procedures according to the flowcharts are not limited to those shown in the drawings, and any combination of procedures, combining of multiple processes, or subdivision of processes is possible as long as the results of the embodiment are achieved. Also, each process can be extracted individually and function as a single functional element, and can be used in combination with processes other than those shown.

[0038] The flow in Fig. 4 starts when the user 101 in Fig. 1 issues an instruction to start traveling the moving object 1. In step S101, the information processing device 100 is initialized. That is, a program is read from the external memory 24 in Fig. 2, and the information processing device 100 is made operable.

[0039] Also, the two-dimensional map data used in step S103 and the three-dimensional point cloud data used in step S114, which will be described later, are loaded from the external memory 24 to the RAM 23. Furthermore, the three-dimensional map data managed by the three-dimensional map update unit 116 is initialized.

[0040] In step S102, the two-dimensional point cloud input unit 111 inputs two-dimensional point cloud data from the first sensor 11. Step S102 functions as a two-dimensional point cloud input step of inputting two-dimensional point cloud data of the environment from the first sensor that measures the two-dimensional point cloud data.

[0041] In step S103, the 2D position and orientation estimation unit 114 estimates its own 2D position and orientation by aligning the 2D point cloud data input from the 2D point cloud input unit 111 with the 2D map loaded in step S101. Here, step S103 functions as a 2D position and orientation estimation step that estimates the 2D position and orientation of the first sensor based on the 2D point cloud data.

[0042] A specific registration method may be a known method such as the Iterative Closest Point (ICP) technique. Alternatively, a Simultaneous Localization and Mapping (SLAM) technique may be used, which simultaneously performs position and orientation estimation and map creation.

[0043] In step S104, the 3D position and orientation estimation unit 115 estimates the 3D position and orientation based on the 3D point group data, the captured image, and the 2D position and orientation data. A specific estimation method will be described later.

[0044] In step S105, based on the 3D point cloud data and the 3D position and orientation, the 3D map update unit 116 updates the 3D map stored in the storage unit 118. In this embodiment, the 3D map is represented by a 3D point cloud.

[0045] Specifically, the update process involves aligning the point cloud of the 3D map held by the holding unit 118 with the 3D point cloud input from the 3D point cloud input unit 113 using the 3D position and orientation, and then comparing them. If the distance between the closest points is equal to or greater than a predetermined value, the points are added to the 3D map to update it. The update process is not limited to this method, and other methods may be used.

[0046] In step S106, the 3D map update unit 116 judges whether or not to end the update operation of the 3D map in Fig. 4. Specifically, if an end instruction from the user 101 is received by the tablet 102, the flow in Fig. 4 ends. If not, the process returns to step S102 and continues.

[0047] Fig. 5 is a flowchart showing an example of a processing procedure for estimating a 3D position and orientation in step S104 in Fig. 4. Note that the CPU as a computer in the information processing device 100 executes a computer program stored in a memory, thereby sequentially performing the operations of the steps in the flowchart in Fig. 5.

[0048] In step S111, the three-dimensional position and orientation estimation unit 115 acquires the two-dimensional position and orientation of the first sensor 11 estimated by the two-dimensional position and orientation estimation unit 114, and sets this as the initial value of the three-dimensional position and orientation of the second sensor 13.

[0049] As described above, the position, orientation, and specifications of each sensor on the moving body are known, and the XY coordinates of the three-dimensional position and orientation of the second sensor 13 can be calculated based on the two-dimensional position and orientation of the first sensor 11. Since the two-dimensional position and orientation does not have a value in the Z axis (height) direction, only the position (X, Y) and direction ψ (azimuth angle) on the two-dimensional plane are set as initial values.

[0050] In step S112, the captured image input unit 112 performs processing to input captured images from the stereo camera imaging device 12. Here, step S112 functions as a captured image input step for inputting captured images of the environment captured by the imaging device.

[0051] In step S113, the 3D position and orientation estimation unit 115 generates 3D point cloud data by extracting corresponding feature points from the captured images (i.e., stereo images in this embodiment). A known technique of passive stereo matching may be used as a method for generating 3D point cloud data from stereo images.

[0052] In step S114, the 3D position and orientation estimation unit 115 estimates the 3D position and orientation of the second sensor 13 by aligning the 3D point group data generated from the captured image in step S113 with the 3D point group loaded from the storage unit 118 in step S101. Here, step S114 functions as a 3D position and orientation estimation step that estimates the 3D position and orientation of the moving object based on the 2D position and orientation of the first sensor and the captured image.

[0053] Specifically, by taking advantage of the fact that the positional relationship between the imaging device 12 and the second sensor 13 on the moving body 1, as well as the orientation and specifications of the sensors, the position and orientation of the imaging device 12 are estimated by alignment and converted into the position and orientation of the second sensor 13.

[0054] That is, the 3D position / orientation estimation unit 115 estimates the position and orientation of the imaging device 12 based on the stereo images, and also estimates the 3D position / orientation of the second sensor 13 based on the position and orientation of the second sensor 13 relative to the imaging device 12.

[0055] A known method such as ICP technology may be used for the alignment. However, during alignment, the two-dimensional position and orientation determined in step S111 are used as initial values. Specifically, the position (X, Y) on the two-dimensional plane is fixed by the value of the two-dimensional position and orientation, and the orientation ψ is set as the initial value for alignment.

[0056] In this embodiment, by fixing a position on a two-dimensional plane determined using measurement data from a 2D LiDAR, which has higher measurement accuracy for a two-dimensional point cloud than a method of generally estimating a two-dimensional position and orientation from a captured image, it is possible to estimate a three-dimensional position and orientation with higher accuracy.

[0057] As a secondary effect, by limiting the degree of freedom of the parameters to be aligned in this way, the calculations are simplified and processing can be performed faster than when all parameters are aligned.

[0058] As described above, in the first embodiment, the two-dimensional position and orientation are estimated based on the two-dimensional point cloud data, and the three-dimensional position and orientation are estimated by estimating the height and orientation based on the captured image using the two-dimensional position and orientation as initial values.

[0059] When estimating the 3D position and orientation, if a 3D distance sensor is used, the estimation accuracy of the 3D position and orientation may be lower depending on the accuracy of the sensor. Also, if the 3D position and orientation is estimated using images captured by a stereo camera or the like, the estimation accuracy of the 3D position and orientation may be lower due to estimation errors when estimating the 3D position and orientation from a 2D image.

[0060] However, by using accurate 2D position and orientation as in this embodiment, it is possible to estimate 3D position and orientation with higher accuracy. In the above description, a monochrome stereo camera is used as the imaging device, but a color stereo camera or a monocular camera may be used. In the case of a monocular camera, images from multiple viewpoints are captured by moving the camera according to the movement of the moving object, and 3D point cloud data is generated by extracting corresponding points between the images.

[0061] A known method can be used to reconstruct a three-dimensional structure (generate three-dimensional point cloud data) based on such multiple captured images. For example, Structure from Motion (SfM), Visual SLAM, or Visual Odometry technology can be used. The monocular camera can be an RGB camera, a monochrome camera, or an infrared camera.

[0062] In the above description, a ToF sensor is used as the second sensor 13, but another sensor may be used as long as it can acquire three-dimensional point cloud data. For example, an image sensor using a passive stereo method, an active stereo method using structured light projection, an image plane phase difference sensor, or a coded aperture method may be used to measure the distance to an object and generate three-dimensional point cloud data.

[0063] Alternatively, 3D point cloud data may be generated using an estimation model that estimates the distance of an object in an image based on one or more 2D images. The estimation model may be trained using deep learning technology or the like.

[0064] In the above description, the two-dimensional position and orientation are estimated based on two-dimensional point cloud data, but as long as two-dimensional plane information can be acquired, the estimation may be based on other types of information. For example, a two-dimensional map (Occupancy Grid Map) may be used.

[0065] For this purpose, a sensor that outputs a two-dimensional map may be provided, or hardware or software for generating a two-dimensional map from a distance sensor may be provided. Position and orientation estimation may be performed while creating a two-dimensional map using SLAM technology.

[0066] In addition, in the above explanation, the position and orientation of the moving object are expressed by parameters of three degrees of freedom, X, Y, and θ. However, as long as the position and orientation can be expressed, they may be held in a different manner, such as a 3 × 3 rigid body transformation matrix.

[0067] Also, it is possible to store six degrees of freedom including the height position (Z) and the rotation parameters of PITCH and ROLL, or to store them in the form of a 4 x 4 matrix. By storing six degrees of freedom in this way, it becomes possible to calculate the height position and orientation when the moving body moves on a slope, etc.

[0068] In the above description, the three-dimensional position and orientation are estimated based on the three-dimensional point group data generated from stereo images by a passive stereo matching method, but the three-dimensional position and orientation may be estimated by another method.

[0069] For example, a Visual SLAM technique may be used that extracts feature points based on multiple captured 2D images to estimate the 3D position and orientation, or a direct method may be used that uses the Visual SLAM technique to minimize the error in brightness values ​​between multiple 2D images to estimate the 3D position and orientation.

[0070] In addition, Visual Inertial SLAM technology or Visual Inertial Odometry technology using an IMU (Inertial Measurement Unit) may be used.

[0071] In the above explanation, the data structure of the three-dimensional point map is a three-dimensional point group, i.e., a point cloud. However, other data structures may be used as long as they can represent the presence or absence of structures in three-dimensional space.

[0072] For example, it may be a voxel representation that indicates whether or not an object is filled in the voxel space, or a data structure such as a 3D OCCUPANCY GRID MAP or a TSDF (TRUNCATED SIGNED DISTANCE FUNCTION), which is a spatial field representation that can represent the surface of a structure (object).

[0073] When a 3D map is stored as a TSDF, it may be converted into a format that is easy for the user to visualize, such as a 3D point cloud or mesh model, and output, depending on the user's request.

[0074] In addition, in the above description, when estimating the three-dimensional position and orientation, only the two-dimensional position of the two-dimensional position and orientation is fixed, and the orientation (azimuth angle) is estimated based on the captured image using the orientation of the two-dimensional position and orientation as the initial value.

[0075] That is, the 3D position and orientation estimation unit 115 estimates the 3D position and orientation by setting the position parameters on the 2D plane estimated by the 2D position and orientation estimation unit 114 as fixed values. However, the orientation may also be fixed. That is, the 3D position and orientation estimation unit 115 may estimate the 3D position and orientation by setting at least some of the position parameters and orientation parameters on the 2D plane estimated by the 2D position and orientation estimation unit 114 as fixed values.

[0076] That is, the two-dimensional position and orientation of the three-dimensional position and orientation may be fixed, and only the height (Z axis) may be estimated. Contrary to the first embodiment, only the orientation may be fixed by the estimated value of the two-dimensional position and orientation, and the two-dimensional position of the two-dimensional position and orientation may be used as an initial value to estimate based on the captured image.

[0077] That is, the 3D position and orientation estimation unit 115 may estimate the 3D position and orientation using the position parameters on the 2D plane estimated by the 2D position and orientation estimation unit 114 as fixed values ​​and the orientation as initial values.

[0078] In addition, in the above explanation, the roll angle and pitch angle are fixed at 0 because the target is a moving body running on a floor surface, but the roll angle and pitch angle may be estimated when running on an uneven floor.

[0079] In the above description, the estimation of the 3D position and orientation and the updating of the 3D map are performed by one information processing device 100. However, the configuration of the device is not limited to this, and the updating of the 3D map may be performed by a device other than the information processing device 100, for example, an external server. In this case, the information processing device 100 does not include the 3D point cloud input unit 113, the 3D map update unit 116, and the storage unit 118.

[0080] In the above description, the information processing device 100 includes a display control unit 117 that displays the creation status and creation results of the 3D map. However, the configuration of the device is not limited to this, and the display control unit 117 may be located outside the information processing device 100. In this case, the information processing device 100 and the display device including the display control unit 117 transmit and receive information via a data communication unit.

[0081] <Embodiment 2> In the first embodiment, an example is described in which a captured image and a 3D point cloud are acquired by a stereo camera and a ToF sensor, respectively, and a 3D map of the environment is generated with high accuracy based on the 2D position and orientation and the captured image. In contrast, in the second embodiment, an example is described in which measurement data from a 3D distance sensor is also used to generate the captured image.

[0082] Fig. 6 is a functional block diagram showing a configuration example of a data flow of an information processing device in embodiment 2. The difference from Fig. 3 in embodiment 1 is that there is no imaging device 12, and measurement data of a second sensor 13 is input to an imaged image input unit 112 and a three-dimensional point cloud input unit 113.

[0083] In this embodiment, the first sensor 11 is a 2D LiDAR, and the second sensor 13 is a ToF sensor. Since the processing flow is the same as in the first embodiment, only the processing that has differences will be specifically described with reference to FIG.

[0084] 5, the captured image input unit 112 inputs a captured image from the second sensor 13. The second sensor 13 is a ToF sensor, and the measurement data is a depth image in which each element on a two-dimensional plane has a value in the depth direction.

[0085] That is, the second sensor 13 is a three-dimensional distance sensor that generates a depth image, and inputs this depth image as a captured image to the captured image input unit 112. Also, the three-dimensional point cloud input unit 113 inputs three-dimensional point cloud data based on the output of the three-dimensional distance sensor.

[0086] In step S113, the 3D position and orientation estimation unit 115 converts the captured image, i.e., the depth image, into 3D point cloud data. Note that the processing procedure may be such that the 3D point cloud data is input as the captured image in step S112, and nothing is done in step S113. In step S114, the 3D position and orientation estimation unit 115 estimates the 3D position and orientation based on the 3D point cloud data, as in the first embodiment.

[0087] As described above, in the second embodiment, a 3D map is generated by using measurement data from a ToF sensor for both capturing captured images and capturing 3D point cloud data. In this way, a 3D map can be generated without a separate imaging device as in the first embodiment. In other words, costs and size can be reduced.

[0088] <Embodiment 3> In the first embodiment, an example is described in which a captured image and a 3D point cloud are acquired by a stereo camera and a ToF sensor, respectively, and a 3D map of an environment is generated with high accuracy based on the 2D position and orientation and the captured image. In the third embodiment, an example is described in which image data captured by an imaging device is also used to generate 3D point cloud data.

[0089] Fig. 7 is a functional block diagram showing a configuration example of a data flow of an information processing device in embodiment 3. The difference from Fig. 3 in embodiment 1 is that there is no second sensor 13, and that the captured image of the imaging device 12 is input to the captured image input unit 112 and the three-dimensional point cloud input unit 113.

[0090] In the third embodiment, the first sensor 11 is a 2D LiDAR, and the imaging device 12 is an RGBD camera. The RGBD camera is a camera that can measure both a color two-dimensional image (RGB) and a depth image (D). The measured two-dimensional image and depth image are input to the captured image input unit 112, and the distance image is input to the three-dimensional point cloud input unit 113.

[0091] That is, in the third embodiment, the imaging device includes an RGBD camera that generates a two-dimensional image and a depth image. The captured image input unit inputs a captured image based on the output of the RGBD camera, and the three-dimensional point cloud input unit inputs three-dimensional point cloud data based on the output of the RGBD camera.

[0092] Since the processing flow is the same as in the first embodiment, only the processing that differs will be specifically described with reference to FIG.

[0093] 5, the captured image input unit 112 performs a process of inputting a captured image from the imaging device 12. The imaging device 12 is an RGBD sensor, and inputs a measured color two-dimensional image and a depth image as a captured image.

[0094] In step S113, the 3D position / orientation estimation unit 115 generates 3D point cloud data from the range image. At this time, color information corresponding to each point of the generated 3D point cloud data is obtained from the 2D image and associated with the data.

[0095] In step S114, as in the first embodiment, the 3D position and orientation estimation unit 115 estimates the 3D position and orientation by aligning the 3D point cloud data generated in step S113 with the 3D point cloud data prepared in advance using ICP technology or the like.

[0096] In this embodiment, alignment is performed using both the distance between points and color information between points. For this reason, color information of each point is associated with and stored in the 3D point cloud data prepared in advance. In this way, alignment can be performed taking into account both the distance and color information of each point, which can improve accuracy compared to alignment based only on the distance between points. can.

[0097] As described above, in the third embodiment, a three-dimensional map is generated by using measurement data from an RGBD camera for both the captured image and the three-dimensional point cloud data. In this way, the three-dimensional position and orientation can be estimated taking color information into account, improving the estimation accuracy of the three-dimensional position and orientation. Also, a three-dimensional map can be generated without separately providing a second sensor as in the first embodiment. In other words, the cost and size can be reduced.

[0098] In addition to the above-described embodiment, the display control unit 117 in FIG. 3 may be used to remotely control the moving body 1 or to display a GUI (Graphical User Interface) presenting a created 3D map on, for example, the tablet 102.

[0099] 8 is a diagram showing an example of a GUI whose display is controlled by the information processing device in embodiment 4. G10 is a display screen of the tablet 102, and displays a GUI for remotely controlling a moving object, a three-dimensional model of the real space, the position and attitude of the moving object, operation information of the moving object, a GUI for operating the moving object, and the like, which are described in embodiments 1 to 3.

[0100] Various buttons and check boxes can be operated by the user's touch operation through the touch panel of the tablet 102. G101 is an emergency stop button, and the user can stop the movement of a moving object by touching this button with his / her finger.

[0101] G103 is an example of a controller for a moving object. The user can move the circle inside the controller up, down, left, and right by touching it, and the moving object 1 can move forward, backward, left, and right according to the input.

[0102] G104 is an example of the internal state of the moving object 1. It shows the moving object 1 moving automatically at a speed of 0.5M / s. G105 is a button for resetting the autopilot and operation settings.

[0103] G106 is a GUI for generating 3D models of real space and setting display information. By touching the checkboxes, you can select whether to generate a 3D model and whether to display each element, such as 3D point cloud, 3D surface information, 2D reference surface information, and 3D reference surface information.

[0104] G107 indicates the current position and the destination of the moving body as operation information of the moving body 1. G110 is a window that presents the three-dimensional model described in this embodiment, and presents the three-dimensional point group G111 set in G106 described above.

[0105] G112 is an obstacle such as a wall, and G113 is a three-dimensional model of the moving object 1 based on the position and orientation of the moving object. Note that the user can control the viewpoint of the three-dimensional model by touching G110 and moving his / her finger up, down, left, or right.

[0106] Thus, in the fourth embodiment, the display control unit 117 displays at least one of the three-dimensional point cloud data, the three-dimensional position and orientation (position and orientation in the three-dimensional space), and the three-dimensional map. Therefore, by presenting a three-dimensional model of the real space, the user can intuitively and remotely control the moving object while recognizing the space around the moving object.

[0107] In the above embodiment, the present invention is applied to an autonomous moving body. However, the moving body in the present embodiment is not limited to an autonomous moving body such as an AGV (Automated Guided Vehicle) or an AMR (Autonomous Mobile Robot).

[0108] In addition, the moving object may be used as a driving assistance device even if it does not move completely autonomously. In addition, the moving object may be any moving device that moves, such as a car, a train, a ship, an airplane, a robot, or a drone.

[0109] Furthermore, at least a part of the information processing system of the embodiment may or may not be mounted on a mobile object. Furthermore, the present invention can also be applied to a case where a mobile object is remotely controlled.

[0110] Although the present invention has been described in detail above based on the preferred embodiments, the present invention is not limited to the above embodiments, and various modifications and combinations of the above embodiments are possible based on the spirit of the present invention, and are not excluded from the scope of the present invention. Note that the present invention includes combinations of the above embodiments, and also includes the following combinations.

[0111] (Configuration 1) An information processing device that estimates a three-dimensional position and orientation of a moving object, comprising: a two-dimensional point cloud input unit that inputs two-dimensional point cloud data of an environment from a first sensor that measures two-dimensional point cloud data; an information processing device comprising: a captured image input unit that inputs a captured image of an environment captured by an imaging device; a 2D position and orientation estimation unit that estimates a 2D position and orientation that represents a position and orientation in a horizontal direction of the first sensor based on the 2D point cloud data; and a 3D position and orientation estimation unit that estimates a 3D position and orientation of the moving object based on the 2D position and orientation of the first sensor and a position and orientation in a height direction obtained from the captured image.

[0112] (Configuration 2) An information processing device according to Configuration 1, comprising: a 3D point cloud input unit for inputting 3D point cloud data; a storage unit for storing a 3D map showing an arrangement of objects in the environment; and a 3D map update unit for updating the 3D map in the storage unit based on the 3D position and orientation estimated by the 3D position and orientation estimation unit and the 3D point cloud data.

[0113] (Configuration 3) An information processing device according to configuration 2, characterized in that the captured image input unit inputs the captured image based on the output of a 3D distance sensor that generates a depth image, and the 3D point cloud input unit inputs the 3D point cloud data based on the output of the 3D distance sensor.

[0114] (Configuration 4) The information processing device described in Configuration 2, characterized in that the imaging device includes an RGBD camera that generates a two-dimensional image and a depth image, the captured image input unit inputs the captured image based on the output of the RGBD camera, and the 3D point cloud input unit inputs the 3D point cloud data based on the output of the RGBD camera.

[0115] (Configuration 5) The information processing device according to any one of configurations 2 to 4, further comprising a display control unit that displays at least one of the 3D point cloud data, the 3D position and orientation, and the 3D map.

[0116] (Configuration 6) The information processing device according to configuration 2, wherein the three-dimensional point cloud input unit inputs the three-dimensional point cloud data of the environment from a second sensor that measures three-dimensional point cloud data.

[0117] (Configuration 7) The imaging device includes a stereo camera for capturing a stereo image, The information processing device according to configuration 6, wherein the 3D position and orientation estimation unit estimates a position and orientation of the imaging device based on the stereo images, and estimates a 3D position and orientation of the second sensor based on a position and orientation of the second sensor relative to the imaging device.

[0118] (Configuration 8) An information processing device according to any one of configurations 1 to 7, characterized in that the 3D position and orientation estimation unit estimates the 3D position and orientation using at least a portion of the position parameters and orientation parameters on a two-dimensional plane estimated by the 2D position and orientation estimation unit as fixed values.

[0119] (Configuration 9) The information processing device described in Configuration 8, wherein the 3D position and orientation estimation unit estimates the 3D position and orientation using at least a portion of the position parameters and orientation parameters on the 2D plane estimated by the 2D position and orientation estimation unit as fixed values.

[0120] (Configuration 10) The information processing device described in Configuration 8 or 9, wherein the 3D position and orientation estimation unit estimates the 3D position and orientation using position parameters on the 2D plane estimated by the 2D position and orientation estimation unit as fixed values ​​and an orientation as an initial value.

[0121] (Configuration 11) The information processing device according to any one of configurations 1 to 10, wherein the first sensor includes a 2D LiDAR capable of two-dimensional measurement.

[0122] (Configuration 12) A moving body, characterized by having a drive control unit that performs autonomous driving based on the three-dimensional position and attitude of the moving body estimated by the three-dimensional position and attitude estimation unit of the information processing device described in any one of configurations 1 to 11.

[0123] (Method) An information processing method for estimating a three-dimensional position and orientation of a moving body, comprising: a two-dimensional point cloud input step of inputting two-dimensional point cloud data of an environment from a first sensor that measures two-dimensional point cloud data; an captured image input step of inputting an captured image of the environment captured by an imaging device; a two-dimensional position and orientation estimation step of estimating the two-dimensional position and orientation of the first sensor based on the two-dimensional point cloud data; and a three-dimensional position and orientation estimation step of estimating the three-dimensional position and orientation of the moving body based on the two-dimensional position and orientation of the first sensor and the captured image.

[0124] (Program) A computer program for controlling each part of the information processing device according to any one of configurations 1 to 11 or the moving object according to configuration 12 by a computer.

[0125] In order to realize a part or all of the control in the above-mentioned embodiment, a computer program that realizes the functions of the above-mentioned embodiment may be supplied to an information processing device or the like via a network or various storage media. Then, a computer (or a CPU, MPU, or the like) in the information processing device or the like may read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. [Explanation of symbols]

[0126] 1: Moving object 11: First sensor 12: Imaging device 13: Second sensor 100: Information processing device

Claims

1. An information processing device that estimates a three-dimensional position and orientation of a moving object, a two-dimensional point cloud input unit that inputs two-dimensional point cloud data of the environment from a first sensor that measures two-dimensional point cloud data; a captured image input unit for inputting a captured image of an environment captured by the imaging device; a two-dimensional position and orientation estimation unit that estimates a two-dimensional position and orientation that represents a position and orientation in a horizontal direction of the first sensor based on the two-dimensional point cloud data; a three-dimensional position and orientation estimation unit that estimates a three-dimensional position and orientation of the moving object based on the two-dimensional position and orientation of the first sensor and a position and orientation in a height direction obtained from the captured image; 13. An information processing device comprising:

2. a three-dimensional point cloud input unit for inputting three-dimensional point cloud data; a storage unit for storing a three-dimensional map showing an arrangement of objects in the environment; a 3D map update unit that updates the 3D map in the storage unit based on the 3D position and orientation estimated by the 3D position and orientation estimation unit and the 3D point cloud data; 2. The information processing apparatus according to claim 1, further comprising:

3. The captured image input unit inputs the captured image based on an output of a three-dimensional distance sensor that generates a depth image, and 3. The information processing apparatus according to claim 2, wherein the three-dimensional point cloud input unit inputs the three-dimensional point cloud data based on an output from the three-dimensional distance sensor.

4. the imaging device includes an RGBD camera for generating a two-dimensional image and a depth image; The captured image input unit inputs the captured image based on an output of the RGBD camera, The information processing apparatus according to claim 2 , wherein the three-dimensional point cloud input unit inputs the three-dimensional point cloud data based on an output from the RGBD camera.

5. 3. The information processing apparatus according to claim 2, further comprising a display control unit that displays at least one of the three-dimensional point cloud data, the three-dimensional position and orientation, and the three-dimensional map.

6. The information processing apparatus according to claim 2 , wherein the three-dimensional point cloud input unit inputs the three-dimensional point cloud data of the environment from a second sensor that measures three-dimensional point cloud data.

7. the imaging device includes a stereo camera that captures a stereo image; 7. The information processing device according to claim 6, wherein the three-dimensional position and orientation estimation unit estimates a position and orientation of the imaging device based on the stereo images, and estimates a three-dimensional position and orientation of the second sensor based on a position and orientation of the second sensor relative to the imaging device.

8. The information processing apparatus according to claim 1 , wherein the three-dimensional position and orientation estimation unit estimates the three-dimensional position and orientation using at least a part of the position parameters and orientation parameters on the two-dimensional plane estimated by the two-dimensional position and orientation estimation unit as fixed values.

9. 9. The information processing apparatus according to claim 8, wherein the three-dimensional position and orientation estimation unit estimates the three-dimensional position and orientation using position parameters on the two-dimensional plane estimated by the two-dimensional position and orientation estimation unit as fixed values.

10. 9. The information processing device according to claim 8, wherein the three-dimensional position and orientation estimation unit estimates the three-dimensional position and orientation using position parameters on the two-dimensional plane estimated by the two-dimensional position and orientation estimation unit as fixed values ​​and an orientation as an initial value.

11. The information processing apparatus according to claim 1 , wherein the first sensor includes a 2D LiDAR capable of two-dimensional measurement.

12. A moving body comprising a drive control unit that performs autonomous driving based on the three-dimensional position and orientation of the moving body estimated by the three-dimensional position and orientation estimation unit of the information processing device according to any one of claims 1 to 11.

13. An information processing method for estimating a three-dimensional position and orientation of a moving object, comprising: a two-dimensional point cloud input step of inputting two-dimensional point cloud data of the environment from a first sensor that measures two-dimensional point cloud data; A captured image input step of inputting a captured image of an environment captured by the imaging device; a two-dimensional position and orientation estimation step of estimating a two-dimensional position and orientation representing a position and orientation in a horizontal direction of the first sensor based on the two-dimensional point cloud data; a three-dimensional position and orientation estimation step of estimating a three-dimensional position and orientation of the moving object based on the two-dimensional position and orientation of the first sensor and a position and orientation in a height direction obtained from the captured image; 13. An information processing method comprising:

14. A computer program for controlling each part of the information processing device according to any one of claims 1 to 11 or the moving body according to claim 12 by a computer.

Images