3D map estimation device and obstacle detection device
The 3D map estimation device improves accuracy by selecting and combining position and orientation estimates from multiple imaging devices, incorporating sensor information, and performing coordinate transformation to create a high-density 3D map, addressing challenges in existing systems.
Patent Information
- Application Number
- JP2021071762
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-04-21
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2041-04-21
AI Technical Summary
Existing 3D map estimation and obstacle detection systems face challenges in achieving high accuracy for position and orientation estimation, particularly when objects are far away or moving slowly, leading to the need for many frames and reduced accuracy.
A three-dimensional map estimation device that selects and combines position and orientation estimates from multiple imaging devices using a first and second selection unit, incorporating sensor information to enhance accuracy and stability, and performs coordinate transformation to create a high-density 3D map.
The system achieves stable and accurate position and orientation estimation, reducing instances where 3D map estimation is not possible, and enhances the accuracy of obstacle detection by utilizing a combination of imaging devices and sensors.
Smart Images

Figure 0007577600000001 
Figure 0007577600000002 
Figure 0007577600000003
Abstract
Description
[Technical field]
[0001] An embodiment of the present invention relates to a 3D map estimation device and an obstacle detection device. [Background technology]
[0002] There is a technology that estimates the position and orientation of a vehicle using a camera mounted on the vehicle and estimates a 3D map of the surrounding area. For example, Visual SLAM (Simultaneous Localization And Mapping) is used to estimate the position and orientation of the vehicle.
[0003] Accuracy of position and orientation estimation is important for estimating 3D maps. However, the accuracy of position and orientation estimation can be reduced depending on the movement of the vehicle equipped with the camera and the object captured by the camera. Also, if the object captured by the camera is far away, the movement of the object is small, so it may take many frames to estimate the position and orientation. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] Patent No. 4760831 Summary of the Invention [Problem to be solved by the invention]
[0005] An object of the present invention is to provide a 3D map estimation device and an obstacle detection device that are capable of estimating position and orientation with high accuracy. [Means for solving the problem]
[0006] The 3D map estimation device according to the embodiment includes a first selection unit, a first position and orientation estimation unit, A second selection unit; andThe vehicle includes a second position and orientation estimation unit and a three-dimensional map estimation unit. The first selection unit selects one or more imaging devices from among a plurality of imaging devices mounted on the vehicle and capturing a plurality of images. The first position and orientation estimation unit performs position and orientation estimation for each imaging device based on the captured images of the imaging devices selected by the first selection unit, and outputs a first position and orientation estimation result for each imaging device. The second selection unit calculates a similarity between a plurality of first position and orientation estimation results, and selects one or more necessary first position and orientation estimation results based on the similarity. The second position and orientation estimation unit outputs a second position and orientation estimation result from one or more first position and orientation estimation results. The 3D map estimation unit estimates a 3D map of the surroundings of the moving object from the second position and orientation estimation result and the image. [Brief description of the drawings]
[0007] [Figure 1] 1 is a block diagram showing functions of a 3D map estimation device 100 and an obstacle detection device 200 according to a first embodiment. [Diagram 2] FIG. 2 is a top view illustrating the arrangement of a camera 10 attached to a vehicle 1. [Diagram 3] 3 is a flowchart showing functions of a 3D map estimation device 100 and an obstacle detection device 200 according to the first embodiment. [Figure 4] FIG. 1 is a diagram showing a method for estimating a position and orientation at time t1 using Visual SLAM. [Diagram 5] 5(a) is a diagram showing an example of an environment including an obstacle 2. (b) is a diagram showing an example of a travel path of the vehicle 1 when the vehicle 1 is caused to travel in the environment shown in FIG. [Figure 6] 1 is a diagram showing an implementation example of a 3D map estimation device 100 and an obstacle detection device 200 according to a first embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0008] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In this description, common parts are denoted by common reference numerals throughout the drawings. Note that the present invention is not limited to the embodiment.
[0009] [First embodiment] (Example of Functional Configuration of 3D Map Estimation Device 100 and Obstacle Detection Device 200) A 3D map estimation device 100 and an obstacle detection device 200 according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the functions of the 3D map estimation device 100 and the obstacle detection device 200 according to the first embodiment.
[0010] The 3D map estimation device 100 estimates a 3D map of the periphery of a moving body, such as an automobile, from cameras 10 mounted inside and outside the moving body, and sensors 20 other than the cameras. The obstacle detection device 200 detects an obstacle 2 based on the 3D map estimated by the 3D map estimation device 100. The 3D map estimation device 100 has a first selection unit 30 connected to each camera 10 and sensor 20, a first position and orientation estimation unit 40, a second selection unit 50, a second position and orientation estimation unit 60, a coordinate conversion unit 70, and a 3D map estimation unit 80. Furthermore, the obstacle detection device 200 has an obstacle detection unit 82 in addition to the 3D map estimation device 100.
[0011] In the following description, a case will be described in which a camera 10 and a sensor other than a camera 20 are mounted on an automobile (vehicle 1). It is possible to mount the camera 10 and the sensor other than a camera 20 on not only automobiles but also robots, drones, etc., estimate a three-dimensional map of the surroundings, and detect an obstacle 2.
[0012] In this specification, the position and orientation of the vehicle 1 refers to, for example, the position and orientation of the vehicle 1. The position may be the absolute position of the acquisition device, or may be a relative position with respect to a certain specific point.
[0013] In this specification, an obstacle 2 is an object whose height from a certain reference point is greater than a certain threshold. For example, this threshold is set based on the shortest distance from the road surface to the lowest part of the vehicle body in a direction perpendicular to the road surface, with the road surface as the reference point. Objects that do not exceed this threshold can be excluded from detection targets for obstacle 2, as they are objects that will not collide with vehicle 1 while vehicle 1 is traveling. Note that an object may be determined as an obstacle 2 not only based on its height, but also based on its size in the horizontal direction to the road surface.
[0014] 2, a plurality of cameras 10 are provided on the vehicle 1, for example, at the front, rear, left and right of the vehicle 1. The sensors 20 are, for example, ultrasonic sensors, laser distance sensors, speed sensors, angular velocity sensors, etc., and are provided inside and outside the vehicle 1, respectively.
[0015] Each camera 10 and each sensor 20 acquires an image and sensor information, respectively, at a certain time t.
[0016] The first selection unit 30 selects which of the multiple cameras 10 and sensors 20 to use to estimate the position and attitude of the vehicle 1. The first selection unit 30 selects the cameras 10 and sensors 20 that will improve the accuracy of the position and attitude estimation by a selection method described later.
[0017] The first position and orientation estimation unit 40 estimates the position and orientation of the vehicle 1 based on the image of the camera 10 selected by the first selection unit 30 and the sensor information acquired by the sensor 20, and obtains a first position and orientation estimation result. The first position and orientation estimation unit 40 estimates the position and orientation of the vehicle 1 based on the image of the camera 10 selected by the first selection unit 30 and the information of the sensor 20, for example, using a Visual SLAM technique. In this embodiment, the first position and orientation estimation unit 40 is provided corresponding to each camera 10 and sensor 20. This is to further improve the accuracy of the second position and orientation estimation result in the 3D map estimation described later. In addition, in FIG. 1, the first position and orientation estimation unit 40 is described as being divided into each block, but the first position and orientation estimation unit 40 may be configured as one block, and the position and orientation estimation corresponding to each camera 10 may be performed individually based on the input signal from the first selection unit 30.
[0018] The second selection unit 50 obtains similarities between the first position and orientation estimation results from the multiple first position and orientation estimation results obtained by each first position and orientation estimation unit 40. The second selection unit 50 selects one or more necessary first position and orientation estimation results based on the similarities. For example, the second selection unit 50 excludes a first position and orientation estimation result whose similarity is relatively low, and selects one or more other first position and orientation estimation results.
[0019] The second position and attitude estimation unit 60 obtains a second position and attitude estimation result of the vehicle 1 by a weighted sum of one or more first position and attitude estimation results selected by the second selection unit 50.
[0020] The coordinate conversion unit 70 converts the movement of a certain camera 10 into the movement of another camera 10. As an example in which correction is necessary, an image acquired by a camera 10 not selected by the first selection unit 30 and the second selection unit 50 may be used for 3D map estimation, which will be described later. Since the first position and orientation estimation result or the second position and orientation estimation result cannot be obtained from an image acquired by a camera 10 not selected by the first selection unit 30 and the second selection unit 50, it is necessary to obtain the change in the position and orientation of the camera 10 from the change in the position and orientation of the other cameras 10. Here, when the vehicle 1 turns, the center of the turn is located near the center of the rear wheel axle of the vehicle 1. Since the distances from the turning center of the cameras 10 are different, the magnitude of the movement due to the turn is different. Therefore, for example, it is necessary to convert the change in the position and orientation of the camera 10 provided in front of the vehicle 1 into the change in the position and orientation of the camera 10 provided behind the vehicle 1. The change in the position and orientation of the camera 10 can be expressed by a matrix representing rotation and a vector representing translation (parallel movement). The coordinate conversion unit 70 performs linear conversion on the rotation matrix and translation vector of a certain camera 10, and can obtain the rotation matrix and translation vector of another camera 10.
[0021] The three-dimensional map estimation unit 80 estimates a three-dimensional map of the periphery of the vehicle 1 using the second position and orientation estimation result of the vehicle 1 and the feature points between the images captured by the camera 10 at different times. The three-dimensional map estimation unit 80 calculates the distance between the vehicle 1 and the feature points using the principle of triangulation from the movement of the camera 10 and the positional relationship of the same feature points in the images captured at different times, and estimates the three-dimensional map. As described above, the three-dimensional map is generated using the feature points of the image of the camera 10, but if there are few or no feature points in the image of the camera 10, the density of the three-dimensional map decreases. Therefore, the image captured by the camera 10 that was not selected by the first selection unit 30 is also input to the three-dimensional map estimation unit 80 to increase the number of feature points to be used. This allows a high-density three-dimensional map to be estimated. Similarly, the three-dimensional map may be estimated using the image that was not selected (excluded) by the second selection unit 50 and was used to calculate the first position and orientation estimation result. The 3D map estimation unit 80 may input external sensor information capable of acquiring distance information, such as external sensor information acquired by an ultrasonic sensor or a laser distance sensor, and estimate a 3D map of the surroundings of the vehicle 1.
[0022] The obstacle detection unit 82 detects an obstacle 2 based on driving information such as the traveling direction and speed of the vehicle 1 and the 3D map calculated by the 3D map estimation unit 80. This detection result is output to, for example, an alarm device (not shown) to alert the driver of the presence or absence of an obstacle 2.
[0023] (Examples of Processing by the 3D Map Estimation Device 100 and the Obstacle Detection Device 200) The process flow of the 3D map estimation device 100 and the obstacle detection device 200 will be described with reference to Fig. 3. Fig. 3 is a flowchart showing the process flow of the 3D map estimation device 100 and the obstacle detection device 200 according to the first embodiment.
[0024] <Step S101> In step S101, the first selection unit 30 selects the camera 10 and the sensor 20 to be used for position and attitude estimation of the vehicle 1 so as to improve the accuracy of the position and attitude estimation.
[0025] The first selection unit 30, for example, compares an image captured at t2 (second timing) with an image captured at time t1 (first timing) after time t2, and selects one or more cameras 10 and sensors 20 that have an appropriate amount of change in movement of an object in the image.
[0026] As an example, the camera 10 and the sensor 20 are selected that have an appropriate amount of change in object movement between images taken at different times based on the positional relationship between the vehicle 1 and surrounding obstacles. The first selection unit 30 selects the camera 10 whose distance between the vehicle 1 and an obstacle 2 around the vehicle 1 acquired from a 3D map based on the image acquired at time t2 and the sensor information is between a first threshold value and a second threshold value that are set in advance. In other words, the camera 10 is selected in which the distance between the obstacle 2 and the vehicle 1 is neither too large nor too small.
[0027] Also, the camera 10 and the sensor 20 may be selected based on the speed (vehicle speed) of the vehicle 1. In FIG. 2, for example, when the vehicle 1 turns left, the amount of movement change between images of the obstacle 2a captured by the camera 10 on the left side of the vehicle 1 at different times becomes small. On the other hand, the amount of movement change between images of the obstacle 2b captured by the cameras 10 in front and behind the vehicle 1 at different times becomes large. In this case, a threshold value may be set in advance for the vehicle speed. When the vehicle speed is equal to or less than the threshold value, the first selection unit 30 selects the cameras 10 mounted on the left and right sides of the vehicle 1 when traveling straight, and selects the cameras 10 mounted on the front and rear sides of the vehicle 1 when turning. On the other hand, when the vehicle speed is equal to or more than the threshold value, the cameras 10 mounted on the front and rear sides of the vehicle 1 are selected.
[0028] Furthermore, other moving objects such as pedestrians may be detected by pattern recognition, and a camera 10 with fewer other moving objects may be selected. This is because Visual SLAM, which will be described later, performs position and orientation estimation and 3D map estimation on the premise that objects in the image of the camera 10 are stationary, and therefore the fewer moving objects within the field of view, the higher the accuracy.
[0029] In addition, when estimating a 3D map using an image captured by the camera 10 at time t3 for the first time, there is no 3D map based on the image captured at time t3 and the sensor information, so the first selection unit 30 selects the camera 10 based on the vehicle speed and angular velocity at time t2 described above, and the presence or absence of other moving objects.
[0030] <Step S102> In step S102, based on the image of the camera 10 and the sensor information of the sensor 20 selected in step S101, each first position and orientation estimation unit 40 performs position and orientation estimation for the corresponding camera 10 and sensor 20.
[0031] Among the methods for estimating the position and orientation at time t1, a method using Visual SLAM will be described with reference to FIG. 4. First, the first position and orientation estimation unit 40 acquires images at time t2 and time t1 from the camera 10 selected by the first selection unit 30. Next, two or more feature point groups are extracted for each of the images at time t2 and time t1. Then, the feature point groups extracted at time t1 and t2 are matched to each other to obtain a correspondence relationship between the feature points at t1 and t2. For matching between the feature point groups, for example, a similarity value calculated using NCC (Normalized Cross-Correlation) or the like for the image pattern around the feature points is used. From the movement of the matched feature point pairs, a change in the position and orientation of the vehicle 1 from time t2 to t1 is estimated.
[0032] The first position and attitude estimation result is composed of the translational motion (three dimensions) and rotational motion (three dimensions) of the vehicle 1 from time t2 to t1, and a six-dimensional first position and attitude estimation result is obtained.
[0033] <Step S103> In step S103, the first position and orientation estimation unit 40 outputs to the second selection unit 50 the six-dimensional first position and orientation estimation result estimated in step S102.
[0034] <Step S104> In step S104, the second selection unit 50 obtains a similarity between the first position and orientation estimation results of each camera 10 based on the output result in step S103. Then, based on the obtained similarity, the second selection unit 50 selects one or more necessary first position and orientation estimation results. For example, the second selection unit 50 excludes a first position and orientation estimation result that is relatively out of the ordinary (an outlier), and selects one or more other first position and orientation estimation results.
[0035] The similarity is determined, for example, by comparing translation vectors representing the translational motion of each camera 10 or by taking the inner product of the vectors. A similarity between rotational motions may be defined and used.
[0036] <Step S105> In step S105, the one or more first position and orientation estimation results selected by the second selection unit 50 in step S104 are output to the second position and orientation estimation unit 60.
[0037] <Step S106> In step S106, the second position and attitude estimation unit 60 obtains a second position and attitude estimation result from the first position and attitude estimation result output in step S105, and outputs it to the coordinate conversion unit 70. In this way, a change in the position and attitude of the vehicle 1 from time t2 to t1 is estimated.
[0038] The second position and orientation estimation result is a weighted sum of each of the first position and orientation estimation results. The weighting coefficient applied to each of the first position and orientation estimation results may be determined according to the reliability, for example, calculated by the second position and orientation estimation unit 60 for each of the first position and orientation estimation results. One example of the reliability is the degree of variation in the distribution of feature points in a three-dimensional space. When the reliability is high, the degree of variation in the distribution of feature points is large. When the variation in the distribution of feature points in a three-dimensional space is small, the distance between the camera 10 and the feature points in a part where the feature points are located can be accurately calculated in step S108 described later, but the distance between the camera 10 and the feature points in other parts cannot be accurately calculated, so that the accuracy of the three-dimensional map estimation decreases. Therefore, the reliability of a first position and orientation estimation result in which the distribution of feature points in a three-dimensional space has a large variation is increased. In addition, there is also a method of increasing the reliability as the number of feature points pairs that match is increased, or increasing the reliability as the average value of the similarity of the feature points pairs that match is increased.
[0039] <Step S107> In step S107, the coordinate transformation unit 70 calculates a change in the position and orientation of each of the cameras 10 by performing coordinate transformation on the second position and orientation estimation result.
[0040] <Step S108> In step S108, the three-dimensional coordinates of the feature points are calculated from the change in the position and attitude of each camera 10 calculated in step S107 and the correspondence between the feature points at times t2 and t1. Then, the three-dimensional point group with the calculated coordinate values is output as a three-dimensional map of the surroundings of the vehicle 1.
[0041] A method for estimating a 3D map of the surroundings of camera 10 using Visual SLAM will be described with reference to Fig. 4. Using the corresponding feature point a in the images at times t2 and t1 extracted in step S102 and the change in position and orientation of camera 10 output in step S107, the 3D position of feature point a is estimated based on the principles of triangulation.
[0042] Furthermore, the 3D map estimation unit 80 determines the correspondence between feature points for the cameras 10 that were not selected by the first selection unit 30 and the second selection unit 50, and estimates the 3D positions of the feature points using the changes in position and orientation of each camera 10 calculated in step S107.
[0043] <Step S109> In step S109, the one or more 3D maps estimated in step S108 are output to the obstacle detection unit 82.
[0044] In addition, a threshold value is set for the distance between the vehicle 1 and the feature point. If the distance between the feature point and the vehicle 1 is equal to or less than the threshold value, the feature point is determined to be an obstacle 2.
[0045] Fig. 5 shows the results of an experiment in which the vehicle 1 was driven in an environment with an obstacle 2. Fig. 5(a) shows an example of an environment with an obstacle 2, and Fig. 5(b) shows an example of the driving trajectory of the vehicle 1 when the vehicle 1 was driven along the route indicated by the dashed line A in the environment shown in Fig. 5(a).
[0046] As shown in FIG. 5(a), the vehicle 1 is driven in an environment in which a plurality of obstacles 2 and one vehicle 1 are provided. Steps S101 to S106 are executed at each time, and the second position and orientation estimation results output at each time are plotted on a graph, resulting in a driving trajectory of the vehicle 1 as shown in FIG. 5(b). In FIG. 5, the traveling direction of the vehicle 1 is defined as the X direction and the Y direction, and the height direction of the vehicle 1 is defined as the Z direction. Note that the second position and orientation estimation results are output in three dimensions, but in the example of the driving trajectory of the vehicle 1 shown in FIG. 5(b), the second position and orientation estimation results in the Z direction are omitted. Also, in FIG. 5(b), the driving trajectory of the vehicle 1 is shown with the starting position of the vehicle 1 as the origin 0.
[0047] (Effects of the First Embodiment) The 3D map estimation device 100 and the obstacle detection device 200 according to the first embodiment are provided with a first selection unit 30 that selects the camera 10 and the sensor 20 to be used for estimating the position and orientation of the vehicle 1. Furthermore, the 3D map estimation device 100 and the obstacle detection device 200 according to the first embodiment use sensor information from the sensor 20 in addition to the image acquired by the camera 10 to estimate the position and orientation of the vehicle 1. Furthermore, the 3D map estimation device 100 and the obstacle detection device 200 according to the first embodiment perform estimation using the image acquired by the camera 10 and the sensor information acquired by the sensor 20 that are not selected by the first selection unit 30 and the second selection unit 50 to estimate the 3D map. As a result, this embodiment has two effects.
[0048] First, the position and orientation of the vehicle 1 can be estimated stably and accurately. The first selection unit 30 selects one or more cameras 10 and sensors 20 that have an appropriate amount of change in movement of an object between images at different times. If the amount of change in movement of an object between images at different times is less than one pixel, the accuracy of the position and orientation estimation deteriorates. However, since the first selection unit 30 selects cameras 10 and sensors 20 that have an appropriate amount of change in movement of an object between images at different times, it is expected that the accuracy of the position and orientation estimation of the vehicle 1 will improve.
[0049] Secondly, it is possible to reduce the number of situations in which 3D map estimation is not possible. The 3D map estimation device 100 and the obstacle detection device 200 according to the first embodiment perform position and orientation estimation using sensor information acquired by the sensor 20 in addition to the camera 10. This makes it possible to perform position and orientation estimation even in cases where it is difficult to estimate the position and orientation using an image from the camera 10, such as immediately after the vehicle 1 starts moving. Since the 3D map estimation uses the second position and orientation estimation result, it is possible to reduce the number of situations in which 3D map estimation is not possible.
[0050] Furthermore, the 3D map estimation device 100 and the obstacle detection device 200 according to the first embodiment estimate the 3D map using the images and sensor information acquired by the camera 10 and the sensor 20 that were not selected by the first selection unit 30. Similarly, the images used for calculating the position and orientation estimation result that were not selected by the second selection unit 50 are also used. As already described, the 3D map is estimated using Visual SLAM from the second position and orientation estimation result and feature points on the image acquired by the camera 10. Therefore, when the estimation accuracy of the change in the position and orientation is low, the estimation accuracy of the 3D map is reduced. The 3D map estimation device 100 and the obstacle detection device 200 according to the first embodiment select the camera 10 and the sensor 20 that can accurately calculate the change in the position and orientation, and estimate the 3D map based on the selected camera 10, so that it is possible to expect a highly accurate 3D map estimation even for the camera 10 that was not selected.
[0051] (Implementation example of 3D map estimation device 100 and obstacle detection device 200) 6 is a block diagram showing an example of hardware implementation of the 3D map estimation device 100 and the obstacle detection device 200 according to the first embodiment. An implementation example of the 3D map estimation device 100 will be described, but the 3D map estimation device 100 can be replaced with the obstacle detection device 200 and similar implementation can also be performed in the obstacle detection device 200.
[0052] The 3D map estimation device 100 can be realized by including a processor 91, a main memory device 92, an auxiliary memory device 93, a network interface 94, and a device interface 95, which are connected via a bus 96. The 3D map estimation device 100 may be a computer device that can run independently, or may be an accelerator incorporated in or connected to a computer device that runs independently.
[0053] 6 includes one of each component, the 3D map estimation device 100 may include multiple of the same component. Also, although one 3D map estimation device 100 is shown, the software may be installed on multiple computer devices, and each of the multiple computer devices may execute different parts of the processing of the software.
[0054] The processor 91 is an electronic circuit that operates as a processing circuit including a control device and a calculation device of the 3D map estimation device 100. The processor 91 performs calculation processing based on data and programs input from each device in the internal configuration of the 3D map estimation device 100, and outputs the calculation results and control signals to each device. Specifically, the processor 91 controls each component constituting the 3D map estimation device 100 by executing an OS (Operating System) and applications of the 3D map estimation device 100. The processor 91 is not particularly limited as long as it can perform the above processing. The 3D map estimation device 100 and each of its components are realized by the processor 91.
[0055] The main memory 92 is a memory device that stores instructions executed by the processor 91 and various data, and information stored in the main memory 92 is directly read by the processor 91. The auxiliary memory 93 is a memory device other than the main memory 92. These memory devices refer to any electronic components capable of storing electronic information, and may be memory or storage. In addition, the memory may be a volatile memory or a non-volatile memory, and either may be used. A memory for saving various data in the three-dimensional map estimation device 100 may be realized by the main memory 92 or the auxiliary memory 93. For example, the storage unit may be implemented in the main memory 92 or the auxiliary memory 93. As another example, when an accelerator is further provided in the three-dimensional map estimation device 100, the storage unit may be implemented in a memory provided in the accelerator.
[0056] The network interface 94 is an interface for connecting to a communication network 98 wirelessly or by wire. The network interface 94 may be one that complies with existing communication standards. The network interface 94 may exchange information with an external device 99A that is connected for communication via the communication network 98.
[0057] The external device 99A includes, for example, a stereo camera, a motion capture device, an output destination device, an external sensor, an input source device, etc. The external device 99A may also be a device having some of the functions of the components of the 3D map estimation device 100. The 3D map estimation device 100 may transmit and receive some of the processing results of the 3D map estimation device 100 via a communication network 98 like a cloud service.
[0058] The device interface 95 is an interface such as a Universal Serial Bus (USB) that directly connects to an external device 99B. The external device 99B may be an external storage medium or a storage device. The storage unit may be realized by the external device 99B.
[0059] The external device 99B may be an output device. The output device may be, for example, a display device for displaying an image, or a device for outputting sound or the like. Examples of the output device include, but are not limited to, an LCD (Liquid Crystal Display), a CRT (Cathode Ray Tube), a PDP (Plasma Display Panel), and a speaker. The output device may also be a component of an automobile controlled via a CAN.
[0060] The external device 99B may be an input device. The input device includes devices such as a keyboard, a mouse, and a touch panel, and provides information input by these devices to the 3D map estimation device 100. A signal from the input device is output to the processor 91.
[0061] Thus, in all the above descriptions, at least a part of the 3D map estimation device 100 and the obstacle detection device 200 may be configured with hardware, or may be configured with software, and may be implemented by a CPU or the like through information processing of the software. In the case of being configured with software, the 3D map estimation device 100, the obstacle detection device 200, and a program that realizes at least a part of the functions thereof may be stored in a storage medium such as a flexible disk or a CD-ROM, and may be read and executed by a computer. The storage medium is not limited to a removable medium such as a magnetic disk or an optical disk, and may be a fixed storage medium such as a hard disk device or a memory. In other words, the information processing by the software may be specifically implemented using hardware resources. Furthermore, the processing by the software may be implemented in a circuit such as an FPGA, and executed by hardware.
[0062] For example, the computer can become the device of the above embodiment by reading out the dedicated software stored in a computer-readable storage medium. The type of storage medium is not particularly limited. Also, the computer can become the device of the above embodiment by installing the dedicated software downloaded via a communication network. In this way, the information processing by the software is specifically implemented using hardware resources.
[0063] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]
[0064] 1 vehicle 2, 2a, 2b Obstacles 3. Feature points 10. Camera 20 Sensors 30 First Selection Section 40 First position and orientation estimation unit 50 Second Selection Section 60 Second position and orientation estimation unit 70 Coordinate conversion section 80 3D map estimation unit 82 Obstacle detection unit 91 Processor 92 Main memory 93 Auxiliary storage device 94 Network Interface 95 Device Interfaces 96 Bus 98 Communication Networks 99A, 99B External device 100 3D map estimation device 200 Obstacle detection device
Claims
1. a first selection unit that selects one or more imaging devices from among a plurality of imaging devices mounted on the moving object and that capture a plurality of images; a first position and orientation estimation unit that performs position and orientation estimation for each of the image capturing devices based on a captured image of the image capturing device selected by the first selection unit, and outputs a first position and orientation estimation result for each of the image capturing devices; a second selection unit that calculates a similarity between the plurality of first position and orientation estimation results and selects one or more necessary first position and orientation estimation results based on the similarity; a second position and orientation estimation unit that outputs a second position and orientation estimation result from one or more of the first position and orientation estimation results; a three-dimensional map estimation unit that estimates a three-dimensional map of an area around the moving object from the second position and orientation estimation result and the image; A three-dimensional map estimation device comprising:
2. the second selection unit selects one or more and N-1 or less of the first position and orientation estimation results from a total of N (N is a natural number) of the first position and orientation estimation results; The three-dimensional map estimation device according to claim 1 .
3. The three-dimensional map estimation device according to claim 1 or 2, wherein the moving body is further equipped with a sensor, and sensor information acquired by the sensor is further input to the first selection unit.
4. The 3D map estimation device of any one of claims 1 to 3, wherein the first selection unit selects the imaging device based on information including the first position and orientation estimation result prior to a first timing for estimating a 3D map around the moving body, the movement direction of the moving body, and the movement speed of the moving body.
5. The three-dimensional map estimation device according to claim 1 , wherein the first selection unit selects the imaging device based on the three-dimensional map estimated before a first timing for estimating the three-dimensional map of the surroundings of the moving body.
6. The 3D map estimation device according to claim 1 , wherein the first selection unit selects the imaging device based on position information of other moving objects existing around the moving object.
7. The 3D map estimation device according to claim 1 , wherein the second position and orientation estimation unit outputs the second position and orientation estimation result by a weighted sum of a plurality of the first position and orientation estimation results inputted thereto.
8. The 3D map estimation device according to claim 7 , wherein a weighting coefficient of the weighted sum is determined by a reliability of the image capture device or a reliability of the first position and orientation estimation result.
9. The 3D map estimation device described in any one of claims 1 to 8, characterized in that the 3D map estimation unit estimates a 3D map of the image using feature points between the image and an image at a time different from the first position and orientation estimation result.
10. A 3D map estimation device as described in any one of claims 1 to 8, characterized in that the 3D map estimation unit estimates a 3D map using the second position and orientation estimation result and feature points of an image acquired by the imaging device that was not selected by the first selection unit.
11. The 3D map estimation device according to claim 1, characterized in that the 3D map estimation unit estimates a 3D map using the second position and orientation estimation result and feature points of an image used to calculate the first position and orientation estimation result that were not selected by the second selection unit.
12. A three-dimensional map estimation device as described in any one of claims 1 to 8, characterized in that the three-dimensional map estimation unit calculates the distance between a feature point of an image acquired by the imaging device and the moving body from external sensor information capable of acquiring distance information, and estimates a three-dimensional map of the surroundings of the moving body.
13. An obstacle detection device having an obstacle detection unit that detects obstacles using a three-dimensional map estimated by a three-dimensional map estimation device described in any one of claims 1 to 12.
14. An obstacle detection device having an obstacle detection unit that detects an obstacle when the distance between the feature point of a three-dimensional map estimated by a three-dimensional map estimation device described in any one of claims 9 to 12 and the moving body is below a threshold value.
15. A plurality of imaging devices, at least one of which is mounted on each of the front, rear, left and right sides of the moving body, for capturing a plurality of images; a first selection unit that selects one or more of the imaging devices; a first position and orientation estimation unit that performs position and orientation estimation for each of the image capturing devices based on a captured image of the image capturing device selected by the first selection unit, and outputs a first position and orientation estimation result for each of the image capturing devices; a coordinate conversion unit that converts a change in a position and orientation of one of the plurality of image capture devices into a change in a position and orientation of another of the plurality of image capture devices; a second position and orientation estimation unit that outputs a second position and orientation estimation result from one or more of the first position and orientation estimation results; a three-dimensional map estimation unit that estimates a three-dimensional map of an area around the moving object from the second position and orientation estimation result and the image; Equipped with the three-dimensional map estimation unit estimates a three-dimensional map by using the second position and orientation estimation result and feature points of an image acquired by the imaging device that are not selected by the first selection unit. 3D map estimation device.
16. a first selection unit that selects one or more imaging devices from among a plurality of imaging devices mounted on the moving object and that capture a plurality of images; a first position and orientation estimation unit that performs position and orientation estimation for each of the image capturing devices based on a captured image of the image capturing device selected by the first selection unit, and outputs a first position and orientation estimation result for each of the image capturing devices; a coordinate conversion unit that converts a change in position and orientation of a first image capture device among the plurality of image capture devices into a change in position and orientation of a second image capture device that does not overlap with a field of view of the first image capture device; a second position and orientation estimation unit that outputs a second position and orientation estimation result from one or more of the first position and orientation estimation results; a three-dimensional map estimation unit that estimates a three-dimensional map of an area around the moving object from the second position and orientation estimation result and the image; Equipped with the three-dimensional map estimation unit estimates a three-dimensional map by using the second position and orientation estimation result and feature points of an image acquired by the imaging device that are not selected by the first selection unit. 3D map estimation device.
Citation Information
Patent Citations
Calibration method for in-vehicle computer-based vision systems
JP2015501471A
Control device, method, and program
JP2016197083A
Vehicle clearance monitoring device and vehicle clearance monitoring method
JP4760831B2
Vehicle-mounted stereo camera system and calibration method therefor
WO2015015542A1
Information processing device, information processing method and program
WO2017057054A1
