Map creation device, map creation method, and map creation program

Abstract

This map creation device is provided with: an acquisition unit that acquires images captured by an imaging device; a selection unit that selects an acquired image as a key frame, each time the amount of movement of a vehicle exceeds a predetermined value; a feature point detection unit that detects a plurality of feature points from the images of the key frames; a feature point selection unit that excludes, from the association of feature points, feature points, among the detected feature points, for which a distance, on the images, between consecutive key frames is smaller than a predetermined value and a luminance value is greater than a predetermined value; a feature point association unit that associates the feature points between the images of the key frames on the basis of features of the feature points that have not been excluded; and a position calculation unit that calculates three-dimensional positions of the associated feature points.

Description

Map creation device, map creation method, and map creation program

[0001] The present disclosure relates to a map creation device, a map creation method, and a map creation program.

[0002] There are technologies for creating an environmental map using images captured by a camera. For example, Non-Patent Document 1 discloses a system that simultaneously identifies the position of a mobile robot and creates an environmental map from images captured by a camera.

[0003] Non-patent document 1: Raul Mur-Artal, JMM Montiel, Juan D. Tardos, "ORB-SLAM: a Versatile and Accurate Monocular SLAM System", IEEE Transactions on Robotics Volume 31 Issue 5, pp 1147-1163

[0004] When lighting equipment located far from a vehicle-mounted camera is captured in an image captured by the camera, part of the image may appear overexposed, resulting in false detection of feature points near the edge of the overexposed area. This false detection of feature points can lead to incorrect correspondence between the falsely detected feature points and actual feature points in the environment, making it impossible to correctly calculate the 3D positions of the actual feature points in the environment. Furthermore, when the falsely detected feature points are matched with each other and the 3D positions of multiple matched feature points are simultaneously optimized using bundle adjustment, the falsely detected feature points can pull the actual feature points, increasing the error in the 3D positions of the actual feature points.

[0005] The present disclosure has been made in consideration of the above points, and aims to provide a map creation device, a map creation method, and a map creation program that improve the accuracy of environmental maps created using images captured by a camera compared to conventional methods.

[0006] In order to achieve the above object, according to an aspect of the present disclosure, there is provided a map creation device including: an acquisition unit that acquires images captured by an imaging device mounted on a vehicle; a selection unit that selects an image acquired by the acquisition unit as a key frame each time an amount of movement of the vehicle exceeds a predetermined value; a feature point detection unit that detects a plurality of feature points from the image of each of the key frames; a feature point selection unit that excludes, from feature point correspondence, feature points that have a distance on the image between consecutive key frames that is smaller than a predetermined value and a brightness value that is greater than a predetermined value, among the feature points detected from the image of each of the key frames by the feature point detection unit; a feature point correspondence unit that matches feature points between the images of each of the key frames based on feature amounts of feature points that were not excluded by the feature point selection unit; and a position calculation unit that calculates the three-dimensional positions of the feature points matched by the feature point correspondence unit.

[0007] When the number of feature points not excluded by the feature point selection unit exceeds a predetermined value, the feature point correspondence unit may exclude feature points for which the distance on the image between successive key frames is smaller than a predetermined value, in descending order of brightness value, until the number of feature points reaches the predetermined value.

[0008] The position calculation unit may be configured to relatively decrease an optimization weight for a feature point in which a distance on the image between consecutive key frames is smaller than a predetermined value, when calculating the three-dimensional position of the feature point, as the brightness value of the feature point increases.

[0009] In order to achieve the above object, according to another aspect of the present disclosure, there is provided a map creation method in which a processor acquires images captured by an imaging device mounted on a vehicle, selects the acquired images as key frames each time the amount of movement of the vehicle exceeds a predetermined value, detects a plurality of feature points from the images of each of the key frames, excludes from feature point correspondence those feature points detected from the images of each of the key frames whose distance on the images between consecutive key frames is smaller than a predetermined value and whose brightness value is greater than a predetermined value, matches feature points between the images of each of the key frames based on the feature amounts of the feature points that were not excluded, and executes a process of calculating the three-dimensional positions of the matched feature points.

[0010] In order to achieve the above object, according to another aspect of the present disclosure, there is provided a map creation program that causes a computer to execute a process of acquiring images captured by an imaging device mounted on a vehicle, selecting the acquired images as key frames each time the amount of movement of the vehicle exceeds a predetermined value, detecting a plurality of feature points from the images of each of the key frames, excluding from the feature point correspondence those feature points detected from the images of each of the key frames whose distance on the images between consecutive key frames is smaller than a predetermined value and whose brightness value is greater than a predetermined value, matching the feature points between the images of each of the key frames based on the feature amounts of the feature points that were not excluded, and calculating the three-dimensional positions of the matched feature points.

[0011] According to the present disclosure, it is possible to provide a map creation device, a map creation method, and a map creation program that improve the accuracy of an environmental map created using images captured by a camera compared to conventional methods.

[0012] 1 is a diagram illustrating an overview of a map creation device according to an embodiment of the disclosed technology; FIG. 2 is a block diagram illustrating a hardware configuration of the map creation device; FIG. 3 is a block diagram illustrating an example of a functional configuration of the map creation device; FIG. 4 is a flowchart illustrating a flow of map creation processing by the map creation device; FIG. 5 is a diagram illustrating calculation of vehicle odometry; FIG. 6 is an example of a keyframe image; FIG. 7 is a flowchart illustrating a flow of map creation processing by the map creation device; and FIG. 8 is a diagram illustrating calculation of three-dimensional positions of feature points according to the principle of triangulation using changes in the position and orientation of a camera.

[0013] An example of an embodiment of the present disclosure will be described below with reference to the drawings. The same reference numerals are used throughout the drawings to designate identical or equivalent components and parts. The dimensional proportions of the drawings are exaggerated for illustrative purposes and may differ from the actual proportions.

[0014] FIG. 1 is a diagram illustrating an overview of a map creation device according to this embodiment. The map creation device 10 is a device that creates an environmental map using images captured by a camera 20 mounted on a vehicle (not shown). The camera 20 is an example of an imaging device, and sequentially sends images captured at predetermined intervals (e.g., every 0.1 seconds) to the map creation device 10. When creating an environmental map using the images captured by the camera 20, the map creation device 10 uses sensing results from a wheel speed sensor 30 that senses the rotational speed of the vehicle's wheels and a steering angle sensor 40 that senses the steering angle of the vehicle. The vehicle's moving speed is calculated from the sensing result of the wheel speed sensor 30, and the vehicle's turning radius is calculated from the sensing result of the steering angle sensor 40.

[0015] The map creation device 10 according to this embodiment can reduce the influence of erroneously detected feature points that may occur in conventional map creation methods when creating an environmental map using images captured by the camera 20. The map creation device 10 according to this embodiment is characterized by improving the accuracy of an environmental map created using images captured by a camera compared to conventional map creation methods by reducing the influence of erroneously detected feature points.

[0016] FIG. 2 is a block diagram showing the hardware configuration of the map creation device 10.

[0017] 2 , the map creation device 10 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input unit 15, a display unit 16, and a communication interface (I / F) 17. Each component is connected to each other via a bus 19 so as to be able to communicate with each other.

[0018] The CPU 11 is a central processing unit that executes various programs and controls each component. That is, the CPU 11 reads programs from the ROM 12 or the storage 14 and executes the programs using the RAM 13 as a work area. The CPU 11 controls the above components and performs various arithmetic processing in accordance with the programs recorded in the ROM 12 or the storage 14. In this embodiment, the ROM 12 or the storage 14 stores a map creation program that creates an environmental map using images captured by the camera 20.

[0019] The ROM 12 stores various programs and various data. The RAM 13 temporarily stores programs or data as a working area. The storage 14 is configured with a storage device such as a hard disk drive (HDD), a solid state drive (SSD), or a flash memory, and stores various programs including an operating system and various data.

[0020] The input unit 15 includes a pointing device such as a mouse and a keyboard, and is used to input various types of information.

[0021] The display unit 16 is, for example, a liquid crystal display, and displays various information. The display unit 16 may be a touch panel type and function as the input unit 15.

[0022] The communication interface 17 is an interface for communicating with other devices, and uses standards such as Ethernet (registered trademark), FDDI, and Wi-Fi (registered trademark).

[0023] When executing the map creation program, the map creation device 10 uses the hardware resources described above to realize various functions.

[0024] Next, the functional configuration of the map creation device 10 will be described.

[0025] FIG. 3 is a block diagram showing an example of the functional configuration of the map creation device 10.

[0026] 3, the map creation device 10 has, as its functional components, an acquisition unit 101, an odometry calculation unit 102, a selection unit 103, a feature point detection unit 104, a feature point selection unit 105, a feature point correspondence unit 106, and a position calculation unit 107. Each functional component is realized by the CPU 11 reading and executing a map creation program stored in the ROM 12 or the storage 14.

[0027] The acquisition unit 101 sequentially acquires images captured by the camera 20, sensing results from a wheel speed sensor 30 that senses the rotational speed of the vehicle's wheels, and sensing results from a steering angle sensor 40 that senses the steering angle of the vehicle.

[0028] The odometry calculation unit 102 calculates the amount of change in position of the vehicle in the vehicle coordinate system (ΔX ｖ , ΔY ｖ ) and yaw angle change Δθ ｖ Calculate the following.

[0029] The selection unit 103 selects a key frame image from among the images acquired from the camera 20. Specifically, the selection unit 103 first selects the image acquired from the camera 20 as the first key frame, and thereafter selects a new image acquired from the camera 20 as another key frame each time the amount of movement of the vehicle obtained from the sensing result of the wheel speed sensor 30 exceeds a predetermined value (e.g., 0.3 m).

[0030] The feature point detection unit 104 detects multiple feature points using an arbitrary algorithm for the image selected as a key frame by the selection unit 103. An example of an algorithm for detecting feature points from an image is FAST (Features from Accelerated Segment Test). FAST is an algorithm that can quickly detect corners in an image.

[0031] The feature point selection unit 105 excludes from the feature point correspondence those feature points detected by the feature point detection unit 104 from the images of each successive key frame, whose distance on the image between successive key frames is smaller than a predetermined value and whose brightness value is greater than a predetermined value.

[0032] The feature point correspondence unit 106 associates feature points between images of each key frame based on feature amounts described by brightness values ​​of areas surrounding feature points that were not excluded by the feature point selection unit 105 .

[0033] If the number of feature points not excluded by the feature point selection unit 105 exceeds a predetermined value, the feature point correspondence unit 106 may exclude feature points for which the distance on the image between successive key frames is smaller than a predetermined value, in descending order of brightness value, until the number of feature points reaches the predetermined value.

[0034] The position calculation unit 107 calculates the three-dimensional positions of the feature points associated by the feature point association unit 106. The position calculation unit 107 calculates the amount of change in position and orientation of the camera 20 between key frames from the positions on the images of the feature points associated by the feature point association unit 106, and calculates the three-dimensional positions of the feature points according to the principle of triangulation using the calculation results of the amount of change in position and orientation. The position calculation unit 107 then applies bundle adjustment to the images of all key frames, thereby simultaneously optimizing the amount of change in position and orientation of the camera 20 and the three-dimensional positions of the feature points.

[0035] For a feature point where the distance on the image between successive key frames is smaller than a predetermined value, the position calculation unit 107 may relatively decrease the optimization weight for the feature point when calculating the three-dimensional position of the feature point as the brightness value increases.

[0036] Next, the operation of the map creation device 10 will be described.

[0037] 4 is a flowchart showing the flow of map creation processing by the map creation device 10. The CPU 11 reads a map creation program from the ROM 12 or the storage 14, loads it into the RAM 13, and executes it, thereby performing the map creation processing.

[0038] In step S101, the CPU 11 periodically acquires images captured by the camera 20. The CPU 11 acquires images captured by the camera 20, for example, every 0.1 seconds.

[0039] Following step S101, the CPU 11 calculates the vehicle odometry in step S102. FIG. 5 is a diagram for explaining the calculation of the vehicle odometry. The CPU 11 calculates the amount of change in the position of the vehicle in the vehicle coordinate system (ΔX ｖ , ΔY ｖ ) and yaw angle change Δθ ｖ The vehicle odometry is calculated by calculating

[0040] Following step S102, in step S103, the CPU 11 selects a key frame image from the images acquired from the camera 20. Specifically, the CPU 11 first selects the image acquired from the camera 20 as the first key frame, and thereafter selects a new image acquired from the camera 20 as another key frame each time the amount of movement of the vehicle obtained from the sensing result of the wheel speed sensor 30 exceeds a predetermined value (e.g., 0.3 m).

[0041] Following step S103, in step S104, the CPU 11 detects a plurality of feature points from the image selected as the key frame in step S103 using an arbitrary algorithm. An example of an algorithm for detecting feature points from an image is FAST.

[0042] Following step S104, in step S105, the CPU 11 selects feature points to be associated in the image of each key frame. Specifically, the CPU 11 excludes, from the feature point association, feature points whose distance on the image between consecutive key frames is smaller than a predetermined value and whose brightness value is greater than a predetermined value, among the feature points detected in the image of each key frame by the feature point detection unit 104.

[0043] 6 shows an example of consecutive key frame images. In FIG. 6, the N-1th key frame image 200ｎ－１ and the Nth key frame image 200 ｎ As shown in FIG. 6, the image of the key frame may contain overexposed areas caused by illumination light or the like. The CPU 11 determines whether the luminance value b of the feature point is equal to or greater than a predetermined value b ｓ and the distance d on the image between successive key frames is greater than a predetermined value d ｓ The distance d on the image is the distance between the (N-1)th keyframe image 200 and the feature point d. ｎ－１ The coordinates of the feature point (x ｎ－１ , y ｎ－１ ) and the Nth key frame image 200 ｎ The coordinates of the feature point (x ｎ , y ｎ ) can be calculated as follows:

[0044]

[0045] The brightness value b of the feature points near the edge of the overexposed area is large, and when the lighting equipment causing the overexposed area is far from the camera, the distance d on the image is small. ｓ is set to a large value, for example, 0.8 (when the brightness value is expressed as 0 to 1), and the predetermined value d ｓ By setting a small value of 10 pixels, the CPU 11 can eliminate feature points that are erroneously detected due to overexposure.

[0046] Following step S105, in step S106, the CPU 11 associates the feature points selected in step S105 with each other using local feature amounts such as ORB. If the number of feature points not excluded in step S105 exceeds a predetermined value, the CPU 11 may exclude feature points whose distance on the image of each key frame is less than the predetermined value in descending order of brightness value until the number of feature points reaches the predetermined value. The feature amounts may be described based on brightness values ​​or color information (e.g., RGB).

[0047] Following step S106, in step S107, the CPU 11 calculates the three-dimensional positions of the feature points associated in step S106. Specifically, the CPU 11 calculates the amount of change in position and orientation of the camera 20 between key frames from the positions on the image of the feature points associated in step S106, and calculates the three-dimensional positions of the feature points according to the principle of triangulation using the amount of change in position and orientation of the camera 20.

[0048] FIG. 7 is a flowchart showing the flow of the map creation process by the map creation device 10, and is a flowchart showing the process of calculating the three-dimensional positions of feature points.

[0049] In step S111, the CPU 11 determines whether the key frame to be processed is the second key frame.

[0050] If it is determined in step S111 that the key frame to be processed is the second key frame (step S111; Yes), the CPU 11 calculates the amount of change in position and orientation of the camera 20 in step S112 using an 8-point algorithm and RANSAC (RANdom Sample Consensus).

[0051] On the other hand, if it is determined in step S111 that the key frame to be processed is not the second key frame (step S111; No), the CPU 11 calculates in step S113 the amount of change in position and orientation of the camera 20 that minimizes the sum of reprojection errors.

[0052] Following step S112 or step S113, in step S114, the CPU 11 calculates the three-dimensional positions of the feature points using the amount of change in position and orientation of the camera 20 according to the principle of triangulation.

[0053] 8 is a diagram for explaining calculation of the three-dimensional position of a feature point according to the principle of triangulation using the amount of change in the position and orientation of the camera 20. ｎ－１ and the feature points in the N-th keyframe image 200 ｎ The three-dimensional position of the feature point is calculated from the feature point in the image by using the amount of change in the position and orientation of the camera 20 according to the principle of triangulation.

[0054] Finally, the CPU 11 applies bundle adjustment to all key frames to simultaneously optimize the position and orientation change amount of the camera 20 and the three-dimensional positions of the feature points. For feature points whose distance on the image is smaller than the predetermined value, the CPU 11 may decrease the optimization weight as the brightness value increases. By decreasing the optimization weight as the brightness value increases, the influence of falsely detected feature points that could not be excluded in step S105 can be reduced.

[0055] As described above, by excluding falsely detected feature points, the map creation device 10 according to this embodiment eliminates the problem that arose in the creation of conventional environmental maps, in which falsely detected feature points are mistakenly associated with actual feature points in the environment, making it impossible to correctly calculate the three-dimensional positions of the actual feature points in the environment.

[0056] Furthermore, the map creation device 10 according to this embodiment associates erroneously detected feature points with each other, and when simultaneously optimizing the three-dimensional positions of the associated feature points by bundle adjustment, the larger the brightness value, the smaller the optimization weight is, thereby eliminating the problem that occurs in the creation of conventional environmental maps, where the erroneously detected feature points tend to influence the three-dimensional positions of actual feature points, resulting in large errors in the positions. As described above, the map creation device 10 according to this embodiment can improve the accuracy of environmental maps created using images captured by a camera compared to conventional map creation methods.

[0057] Although the embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. The above-described embodiments are illustrative and do not limit the technical scope of the present disclosure. It is clear that a person skilled in the art of the present disclosure can conceive of various modifications or alterations within the scope of the technical idea described in the claims, and it is understood that these modifications or alterations also naturally fall within the technical scope of the present disclosure.

[0058] Furthermore, the effects described in the above embodiments are explanatory or exemplary and are not limited to those described in the above embodiments. In other words, the technology according to the present disclosure may achieve other effects that are obvious to a person skilled in the art of the present disclosure from the description in the above embodiments, in addition to or instead of the effects described in the above embodiments.

[0059] In the above embodiments, the map creation process executed by the CPU after reading the software (program) may be executed by various processors other than the CPU. Examples of processors in this case include programmable logic devices (PLDs) (such as field-programmable gate arrays (FPGAs)) whose circuit configuration can be changed after manufacture, and dedicated electrical circuits, such as application-specific integrated circuits (ASICs), which are processors with circuit configurations designed specifically to execute specific processes. The map creation process may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (e.g., multiple FPGAs, or a combination of a CPU and an FPGA). The hardware structure of these various processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor devices.

[0060] In addition, in each of the above embodiments, the map creation processing program is described as being pre-stored (installed) in a ROM or storage, but this is not limiting. The program may be provided in a form recorded on a non-transitory recording medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The program may also be downloaded from an external device via a network.

[0061] The disclosure of Japanese Patent Application No. 2024-43581, filed on March 19, 2024, is incorporated herein by reference in its entirety.

Claims

1. A map creation device comprising: an acquisition unit that acquires images captured by an imaging device mounted on a vehicle; a selection unit that selects an image acquired by the acquisition unit as a key frame each time an amount of movement of the vehicle exceeds a predetermined value; a feature point detection unit that detects a plurality of feature points from the image of each of the key frames; a feature point selection unit that excludes, from feature point correspondence, feature points detected by the feature point detection unit from the image of each of the key frames, those feature points whose distance on the image between consecutive key frames is smaller than a predetermined value and whose brightness value is greater than a predetermined value; a feature point correspondence unit that matches feature points between the images of each of the key frames based on feature amounts of feature points that were not excluded by the feature point selection unit; and a position calculation unit that calculates the three-dimensional positions of the feature points matched by the feature point correspondence unit.

2. The map creation device according to claim 1, wherein, when the number of feature points not excluded by the feature point selection unit exceeds a predetermined value, the feature point correspondence unit excludes feature points whose distance on the images of consecutive key frames is less than a predetermined value in descending order of brightness value until the number of feature points reaches the predetermined value.

3. A map creation device as described in claim 1 or claim 2, wherein the position calculation unit, for feature points whose distance on the images of consecutive key frames is smaller than a predetermined value, assigns a relatively smaller optimization weight to the feature points when calculating the three-dimensional positions of the feature points as the brightness value increases.

4. A map creation method comprising: a processor performing processes of acquiring images captured by an imaging device mounted on a vehicle; selecting the acquired image as a key frame each time the amount of movement of the vehicle exceeds a predetermined value; detecting a plurality of feature points from the image of each of the key frames; excluding, from the feature point correspondence, feature points whose distance on the image between consecutive key frames is smaller than a predetermined value and whose brightness value is greater than a predetermined value; matching feature points between the images of each of the key frames based on the feature amounts of the feature points that were not excluded; and calculating the three-dimensional positions of the matched feature points.

5. A map creation program that causes a computer to execute the following processes: acquire images captured by an imaging device mounted on a vehicle; select the acquired image as a key frame each time the amount of movement of the vehicle exceeds a predetermined value; detect a plurality of feature points from the image of each of the key frames; exclude from feature point correspondence those feature points detected from the image of each of the key frames whose distance on the image between consecutive key frames is smaller than a predetermined value and whose brightness value is greater than a predetermined value; associate feature points between the images of each of the key frames based on the feature amounts of the feature points that were not excluded; and calculate the three-dimensional positions of the associated feature points.

Images