self-position estimation device

Abstract

Self-position estimation device (10, 300), with: an image acquisition unit (14, 210) that captures a multitude of driving images in a state in which a self-propelled vehicle (50) is driven along a predetermined driving route, and a multitude of reference images at a multitude of positions along the predetermined driving route; a capture unit (20, 320) that captures feature points in each of the plurality of driving images and feature points in each of the plurality of reference images, each of which is correlated with the reference images, wherein the feature points in each of the reference images captured by the capture unit (20) serve as environment feature points in an environment along the predetermined driving route; a storage unit (16) that stores map information comprising the feature points in each of the plurality of reference images as well as a position and attitude of the image acquisition unit (14, 210) at a time when each of the plurality of reference images is acquired by the image acquisition unit (14, 210); an estimation unit (24, 400) that selects a similar image to one of the multitude of driving images from the multitude of reference images in order to correlate the feature points in that one of the multitude of driving images and feature points in the similar image, wherein the estimation unit (24, 400) estimates a position and attitude of its own vehicle (50) on the predetermined driving route based on a correlation result; and a setting unit (12) that estimates a road surface gradient difference of the predetermined driving route, and sets a detection range (As) of correlation feature points in each of the plurality of driving images correlated with the surrounding feature points based on the estimated road surface gradient difference, wherein the setting unit (12) selects that similar image which is similar to one of the plurality of driving images from the plurality of reference images based on the correlation feature points detected in the detection range.

Description

TECHNICAL AREA

[0001] This disclosure relates essentially to a self-position estimation device, a self-position estimation method, and a self-position estimation program. STATE OF THE ART

[0002] According to a known technology, the position of a moving body in a three-dimensional space is estimated on the basis of a captured image taken by an image acquisition unit, such as a camera, which is mounted on the moving body, for example.

[0003] For example, JP 2017 - 138 664 A, referred to below as Publication 1, discloses a control device for automatic driving as a technology relating to estimating the own position (own position) of a vehicle.The automatic driving control device disclosed in document 1 comprises an automatic driving information registration unit, which generates automatic driving information for the automatic driving of the vehicle based on a registration image that serves as an image capturing the vehicle's surroundings in a registration mode in which the vehicle is driven by a driver, and an automatic driving control unit that drives the vehicle automatically based on the automatic driving information and an automatic driving image that serves as an image capturing the vehicle's surroundings in an automatic driving mode in which the vehicle is driven automatically.The automatic driving information registration unit comprises a feature point candidate extraction unit that extracts a multitude of feature point candidates found in the vehicle's surrounding environment, based on the registration image. The automatic driving information registration unit also includes an automatic driving information generation unit that selects any of the feature point candidates, determined to be a structure (a building) arranged in a fixed state around a destination for the vehicle, as feature points, and generates the automatic driving information, which serves as information about the position of the selected feature points relative to a predetermined output coordinate.The automatic driving control unit comprises a vehicle position calculation unit that calculates vehicle position information, which serves as information about the vehicle's position relative to the origin coordinate, based on the automatic driving image and the automatic driving information. The automatic driving control unit also includes an automatic driving execution control unit that automatically drives the vehicle to the destination based on the vehicle position information. According to the automatic driving control device disclosed in Document 1, the feature points on the structure are pre-extracted from the image obtained during the vehicle's journey, and the three-dimensional positions of the respective feature points are estimated and recorded on a map.Additionally, feature points on the structure are extracted from the image obtained during automatic driving and compared with the feature point on the structure registered in the map, thereby estimating the vehicle's own position.

[0004] Furthermore, DE 10 2007 032 544 A1 discloses a motion state detection device with a camera mounted on a moving body, such as a vehicle, to capture an image of a road surface on which the moving body is traveling, preferably a road surface in front of the vehicle in the direction of travel, diagonally from above, so that a bird's-eye view image is obtained, and a coordinate conversion means that treats the road surface as a plane and performs a coordinate conversion of the bird's-eye view image to obtain an overhead image that views the road surface directly from above. Additionally, US 2019 / 0 039 605 A1 describes capturing images in a mode where the driver is driving and in a mode where the vehicle is driving automatically along a predetermined route. Feature points in both sets of images are determined and stored.The similar images and feature points are used for automated driving.

[0005] A simultaneous localization and mapping (SLAM) system is known, which includes a mapping mode for generating an environment map and a localization mode for estimating a self-position on the environment map.

[0006] An ORB-SLAM is also known to be a feature point, such as the corners of a structure (circles in Fig. 19) captured by FAST in an image taken by a camera as an example, and applies the ORB features as a description of feature values ​​for the feature points. Such an ORB-SLAM is disclosed in “ORB-SLAM: a Versatile and Accurate Monocular SLAM System” IEEE Transactions on Robotics, Vol. 31, No. 5, 2015, pp. 1147–1163, by Raul Mur-Artal, JMM Montiel, and Juan D. Tardos (hereinafter referred to as Publication 2).

[0007] Each of the ORB features is described as a feature value in 32 bytes, which is defined by a reference 256 times to an order-of-magnitude relationship of a mean luminance between two 5 x 5 pixel areas, each of which (for example, a pair of 1a and 1b in Fig. 19 out of 256 pairs) of 31 x 31 pixels with the center of the obtained feature point, as in Fig. 20 is illustrated, and can be selected.

[0008] Some feature points extracted from the image may exhibit a significant difference in luminance relative to their surroundings. In this case, the feature values ​​of such points may change between day and night due to variations in ambient brightness. Because of such changes in feature values, correlating a feature point extracted from the image with a feature point recorded on the map may fail, potentially preventing an accurate estimation of the vehicle's position. According to Document 1, because changes in ambient brightness are not taken into account, estimating the vehicle's position can be difficult.

[0009] Therefore, it is an object of the invention to provide a self-position estimation device that accurately estimates the self-position of a vehicle even in a case where the brightness in environments varies. SUMMARY

[0010] The object of the invention is achieved by a self-position estimation device according to claim 1, alternatively by a self-position estimation device according to claim 6, and further alternatively by a self-position estimation device according to claim 7. Further features and advantageous embodiments are shown in the dependent claims.

[0011] According to one aspect of this disclosure, a self-position estimation device comprises an image acquisition unit that acquires a plurality of driving images in a state in which a self-positioning vehicle is driven along a predetermined driving route and a plurality of reference images at a plurality of positions along the predetermined driving route; a detection unit that detects feature points in each of the plurality of driving images and feature points in each of the plurality of reference images correlated with each of the reference images; a storage unit that stores map information comprising the feature points in each of the plurality of reference images and a position and attitude of the image acquisition unit at a time when each of the plurality of reference images is acquired by the image acquisition unit; and an estimation unit that selects a similar image to one of the plurality of driving images from the plurality of reference images.to correlate the feature points in one of the multitude of driving images and feature points in the similar image, wherein the estimation unit estimates a position and attitude of its own vehicle on the predetermined driving route based on a correlation result.

[0012] Therefore, the own position of the vehicle can be accurately estimated based on the correlation result of correlating the feature points in the driving image and the feature points in the similar image.

[0013] The self-position estimation device further comprises a calculation unit that calculates at least one feature value from each of the feature points in each of the plurality of driving images captured by the detection unit, and at least one feature value from each of the feature points in each of the plurality of reference images captured by the detection unit. The estimation unit selects the image most similar to one of the plurality of driving images from the plurality of reference images based on the feature values ​​calculated by the calculation unit.

[0014] Therefore, by calculating the feature value of each of the feature points and selecting the similar image based on the calculation result, the similar image can be selected precisely even in a case where brightness changes in environments.

[0015] The self-position estimation device further includes an addition unit that adds the feature value of each of the feature points in one of the multitude of driving images correlated with each of the feature points in the similar image by the estimation unit to the map information as the feature value of each of the feature points on one of the multitude of reference images, which serves as the similar image.

[0016] Accordingly, the feature value of the feature point on the driving image is added and stored in the map information, so that it is referenced when the vehicle's own position is estimated.

[0017] The estimation unit calculates corresponding distances between a plurality of feature values ​​registered for each of the feature points in the similar image and the feature value of each of the feature points in the driving image to correlate each of the feature points in the similar image and each of the feature points in the driving image, wherein the estimation unit correlates the feature point in the driving image and the feature point in the similar image in a case where a minimum value of calculated distances is less than or equal to a predetermined value.

[0018] Consequently, the distance between the feature values ​​can easily fall to or below the predetermined value. The feature points between the similar image and the driving image can also be correlated even if the feature value of the feature point in the driving image changes due to a change in the brightness of the surroundings.

[0019] The addition unit calculates distances between all pairs of feature values ​​of each of the feature points in the similar image, which are selected from the multitude of feature values ​​registered for each of the feature points, and the feature point to be added in a case where the number of the multitude of feature values ​​registered for each of the feature points in the similar image reaches an upper limit, and deletes one of the feature values ​​whose median of distances of one of the feature values ​​with respect to the other of the feature values ​​is smallest among the medians of the distances between all pairs of feature values.

[0020] Therefore, the feature values ​​of the feature points between the similar image and the driving image can still be accurately correlated with each other.

[0021] The map information includes the positions of feature points captured from each of the multiple reference images. The estimation unit estimates the position and attitude of the vehicle on the predetermined route by estimating the position and attitude of the image acquisition unit based on the positions of feature points in the driving image and the positions of feature points in the similar image, which are correlated with each other, in order to convert the estimated position and attitude of the image acquisition unit to a representative point of the vehicle.

[0022] Consequently, because the position and attitude of the image capture unit are converted to the representative point of the vehicle with less computational load, the position of the vehicle is estimated in a short time period while the vehicle is being driven, which can reduce the delay time of a vehicle operation control.

[0023] The estimation unit estimates the position and attitude of the image acquisition unit at which the sum of projection differences, which are the differences between the respective positions of the feature points in the driving image and the respective positions of projection points obtained by projecting the feature points onto the similar image, is minimal relative to the driving image based on the position and attitude of the image acquisition unit at the time the similar image is acquired.

[0024] Therefore, because the sum of projection differences is the smallest, the position and attitude of one's own vehicle can be accurately estimated.

[0025] The addition unit selectively registers, in addition to the map information, the feature value of the feature point whose projection difference is less than or equal to a predetermined value, from the feature points in the driving image correlated with the feature points in the similar image by the estimation unit.

[0026] Consequently, the position and attitude of one's own vehicle can be estimated more accurately with less false correlation.

[0027] The feature points in each of the reference images captured by the acquisition unit serve as environmental feature points in an environment along the predetermined route.

[0028] Therefore, because the feature points in the reference image serve as the environment feature points, the number of feature points in the image can be reduced, which decreases the feature point extraction time.

[0029] The self-position estimation device further comprises a setting unit that estimates a road surface gradient difference of the predetermined driving route and sets a detection range of correlation feature points on each of the plurality of driving images correlated with the surrounding feature points based on the estimated road surface gradient difference, wherein the setting unit selects the image similar to one of the plurality of driving images from the plurality of reference images based on the correlation feature points detected in the detection range.

[0030] Therefore, even if the distribution of feature points across the entire recorded image is biased or uneven due to a gradient of the road surface, the vehicle's own position can be estimated.

[0031] The map information includes feature values ​​that specify the respective characteristics of the environmental feature points. The estimation unit selects the similar image from the multitude of reference images, whereby the similar image includes the largest number of environmental feature points correlated with the correlation feature points on the driving image, wherein the estimation unit correlates the environmental feature points and the correlation feature points by comparing the feature values ​​of the correlation feature points and the feature values ​​of the environmental feature points.

[0032] Accordingly, the reference image is selected based on the feature values ​​of the environmental feature points, so that the environmental feature points and the correlation feature points can be correlated exactly or accurately with each other.

[0033] The image acquisition unit is fixed to a predetermined position on the vehicle to capture an image of a predetermined area in front of the unit. The adjustment unit selects the most similar image from a multitude of reference images, and estimates the road surface gradient difference from the difference between a road surface gradient based on the position and attitude of the image acquisition unit correlated with the selected similar image and a road surface gradient based on the position and attitude of the image acquisition unit estimated from the previously captured driving image.

[0034] Therefore, the road surface gradient can be estimated with less computational effort.

[0035] The adjustment unit moves the detection area up and down within the driving pattern based on the difference in road surface gradient. Consequently, even with a large difference in road surface gradient, the vehicle's position can be accurately estimated.

[0036] The setting unit specifies a position of the detection area (As) in the driving image on an upper side when there is an increase in a road surface gradient at a position of the own vehicle, compared to a road surface gradient of the predetermined area.

[0037] Consequently, because the detection range is limited based on the road surface gradient difference, the vehicle's own position can be estimated with less computational effort.

[0038] According to another aspect of this disclosure, a self-position estimator estimates the position of its own vehicle from an environment map that calculates feature values ​​from each of a plurality of feature points whose positions are known, wherein the self-position estimator comprises a feature point extraction unit that extracts a plurality of feature points from an image capturing the environments of the own vehicle, a feature value calculation unit that calculates a feature value from each of the plurality of feature points extracted by the feature point extraction unit, the feature value being based on a luminance of each of the plurality of feature points, a light source direction estimation unit that estimates a light source direction relative to an imaging direction at a time while the image is being captured based on sensor information, and a correction factor decision unit.which determines a feature value correction factor to correct the feature value of each of the multitude of feature points extracted by the feature point extraction unit, so that the feature value is brought into a state in which the feature point is extracted from it by the feature point extraction unit, is brought into a light source direction in which the environment map is generated by obtaining the feature value from each of the multitude of feature points, on a basis of the feature value for each of the multitude of feature points extracted by the feature point extraction unit, the estimated light source direction, and the previously obtained light source direction when the feature value is obtained from each of the multitude of feature points on the environment map, a feature value correction unit,which corrects the feature value of each of the multitude of feature points extracted by the feature point extraction unit based on the feature value of each of the multitude of feature points extracted by the feature point extraction unit and the feature value correction factor for each of the multitude of feature points, and an estimation unit that estimates a position of the own vehicle based on the corrected feature value for each of the multitude of feature points extracted by the feature point extraction unit and the feature value of each of the multitude of feature points on the environment map.

[0039] Therefore, even if the estimated direction of the light source differs from the direction of the light source at the time when the feature value is taken from each of the feature points on the environment map, the vehicle's own position can be estimated with less computational effort.

[0040] According to another aspect of this disclosure, a self-position estimator estimates the position of its own vehicle from an environment map that calculates a feature value from each of a plurality of feature points whose positions are known, wherein the self-position estimator comprises a feature point extraction unit that extracts a plurality of feature points from an image capturing the environments of the own vehicle, a feature value calculation unit that calculates a feature value from each of the plurality of feature points extracted by the feature point extraction unit, the feature value being based on a luminance of each of the plurality of feature points, a light source direction estimation unit that estimates a light source direction relative to an imaging direction at a time when the image is captured based on sensor information, and a correction factor decision unit.which corrects a feature value correction factor to correct the feature value for each of the plurality of feature points on the environment map, so that the feature value is brought into a state in which the feature point of it, which is extracted by the feature point extraction unit, is brought into a light source direction in which the environment map is generated, by obtaining the feature value for each of the plurality of feature points, based on the feature value of each of the plurality of feature points extracted by the feature point extraction unit, the estimated light source direction, and the previously obtained light source direction, when the feature value is obtained from each of the plurality of feature points on the environment map, a feature value correction unit,which corrects the feature value of each of the multitude of feature points on the environment map based on the feature value of each of the multitude of feature points on the environment map and the feature value correction factor for each of the multitude of feature points, and an estimation unit that estimates a position of the own vehicle based on the corrected feature value for each of the multitude of feature points on the environment map and the feature values ​​of each of the multitude of feature points extracted by the feature point extraction unit.

[0041] Therefore, even if the estimated direction of the light source differs from the direction of the light source at the time when the feature value is taken from each of the feature points on the environment map, the vehicle's own position can be estimated with less computational effort.

[0042] The self-position estimation device further includes a sunshine state estimation unit, which estimates the sunshine state of the vehicle based on brightness obtained from sensor information. The correction factor decision unit determines the feature value correction factor to correct the feature value of each of the plurality of feature points based on the feature value of each of the plurality of feature points, the estimated light source direction, and the previously obtained light source direction when the feature value is referenced to the environment map, in a case where the sunshine state obtained by the sunshine state estimation unit influences the feature value of each of the feature points due to a difference in the light source direction.

[0043] Therefore, the vehicle's own position can be estimated with less computational effort based on the sunshine conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The foregoing and additional features and characteristics of this disclosure will become more apparent from the following detailed description with reference to the attached drawings, wherein: Fig. 1A is a representation illustrating a vehicle on which a camera is mounted, according to a first and second embodiment disclosed herein; Fig. 1B is a block diagram illustrating an example of a construction of a self-position estimation device according to the first and second embodiments; Fig. 2A is a representation illustrating an example of feature points on a parking route according to the first and second embodiments; Fig. 2B is a representation that illustrates an example of a camera position and positions of feature points in a keyframe according to the first and second embodiments; Fig. 3 is a block diagram illustrating an example of a functional configuration of the self-position estimation device according to the first embodiment; Fig. 4 is a representation that illustrates a correlation between feature points and a driving pattern as well as feature points of a similar key image according to the first embodiment; Fig. 5 is a representation that illustrates a distance between each of the feature points of the driving image and each of the feature points on the similar key image according to the first embodiment; Fig. 6 is a representation that explains a projection difference according to the first embodiment; Fig. 7 is a representation that explains a feature value which is deleted in a case where the number of multiple feature values ​​for each of the feature points of the similar key image reaches an upper limit, according to the first embodiment; Fig. 8 is a flowchart illustrating an example of a processing sequence by a self-position estimation program according to the first embodiment; Fig. 9A is a representation that illustrates a difference of a camera pitch angle based on a road surface gradient according to the second embodiment; Fig. 9B is a representation that explains a feature point detection area according to the second embodiment; Fig. 9C is a representation that illustrates a relationship between an upper end and a lower end of the feature point detection area and the road surface gradient according to the second embodiment; Fig. 10A is a representation illustrating an output feature point acquisition area according to the second embodiment; Fig. Figure 10B is a representation illustrating a feature point capture area in a case where a road surface gradient difference is greater than zero; Fig. 11 is a flowchart illustrating the process of a processing operation by a self-position estimation program according to the second embodiment; Fig. 12 is a representation that explains a projection difference according to the second embodiment; Fig. 13 is a block diagram illustrating a schematic configuration of a self-position estimation device according to a third embodiment disclosed herein; Fig. 14 is a representation that illustrates an example of a feature value correction factor according to the third embodiment; Fig. 15 is a flowchart illustrating a self-position estimation processing routine according to the third embodiment; Fig. 16 is a flowchart illustrating a correction factor decision processing routine according to the third embodiment; Fig. 17 is a block diagram illustrating a schematic configuration of a self-position estimation device according to a modified example of the third embodiment; Fig. 18 is a flowchart illustrating a self-position estimation processing routine according to the modified example of the third embodiment; Fig. 19 A conceptual representation of an example of a procedure for capturing a feature point and obtaining a feature value therefrom according to a known technology; Fig. 20 A conceptual representation of an example of the feature value is according to the known technology; Fig. 21 is a conceptual representation of an example of a case in which the direction of sun rays is changed according to known technology; and Fig. 22 is a conceptual representation of an example of the feature value in a case where the direction of sun rays is changed according to the known technology. DETAILED DESCRIPTION

[0045] Exemplary embodiments are explained with reference to the attached drawings. In the following explanation, a vehicle serves as an example of a moving body. The vehicle's surroundings, in a case where the vehicle estimates its own position, are defined as a parking space. A route from outside the parking space to a parking point is defined, for example, as a driving route.

[0046] A self-position estimation device, a self-position estimation method, and a self-position estimation program according to a first embodiment are described with reference to the Fig. 1, Fig. 2, Fig. 3, Fig. 4, Fig. 5, Fig. 6, Fig. 7 to Fig. 8 explained.

[0047] Fig. Figure 1A is a side view of a vehicle 50, which serves as a separate vehicle on which a camera 14 is mounted. As in Fig. As illustrated in Figure 1A, the vehicle 50 comprises the camera 14, which serves as an image acquisition unit used during a process for estimating the self-position of the vehicle 50 (hereinafter referred to as a self-position estimation process) according to the first embodiment.

[0048] The camera 14 according to the first embodiment is, for example, mounted on a trunk located at the rear of the vehicle 50 to capture an image behind the vehicle 50. The camera 14 is positioned in the vicinity of a substantially central section in a width direction of the vehicle, for example, in a state where an optical axis of the camera 14 is directed slightly downwards relative to a horizontal direction. In the first embodiment, the camera 14 is mounted at the rear of the vehicle 50. Alternatively, the camera 14 can be mounted at the front of the vehicle 50, depending, for example, on environmental or circumstances. Additionally, a monocular camera is used as the camera 14 in the first embodiment. Alternatively, other types of cameras, such as a stereo camera, can be used as the camera 14.

[0049] Fig. Figure 1B is a block diagram illustrating an example of an electrical configuration of a self-position estimation device 10 according to the first embodiment. As shown in Fig. As illustrated in Figure 1B, the self-position estimation device 10 is mounted on the vehicle 50 and is constructed such that it comprises a control unit 12, the camera 14, a storage unit 16 and a display unit 18.

[0050] The control unit 12 performs arithmetic processing, for example, to estimate the self-position of the vehicle 50. The control unit 12 comprises, for example, a central processing unit (CPU) 12A, a read memory (ROM) 12B, a read / write memory (RAM) 12C, and an input / output interface 12D (hereinafter referred to as an I / O 12D). The CPU 12A, the ROM 12B, the RAM 12C, and the I / O 12D are interconnected via a bus 12E.

[0051] The CPU 12A controls the entire self-position estimation device 10. The ROM 12B stores, for example, various programs and data, including a map generation program for creating a map used in the first embodiment, and a self-position estimation program for estimating the self-position of the vehicle (vehicle 50). The RAM 12C is a memory that serves as a workspace during the execution of different programs. Each program stored in the ROM 12B is loaded into the RAM 12C, so that the CPU 12A executes the program, thereby generating the map and estimating the self-position of the vehicle 50.

[0052] The control unit 12 is connected to the camera 14, the storage unit 16, and the display unit 18 via the I / O 12D. An image captured by the camera 14 is transmitted to the control unit 12 via the I / O 12D.

[0053] The storage unit 16 stores map information, which is used, for example, for self-position estimation processing according to the first embodiment. The storage unit 16 is not limited to a specific configuration. For example, a hard disk drive (HDD), a solid-state drive (SSD), and flash memory can be used as the storage unit 16. The storage unit 16 can be provided within the control unit 12 or be externally connectable. Instead of the ROM 12B, the storage unit 16 can store the map generation program and / or the self-position estimation program in addition to the map information generated by the control unit 12.

[0054] The display unit 18, for example, displays the image captured by the camera 14. The display unit 18 is not limited to a specific configuration. For example, a liquid crystal monitor, a cathode ray tube (CRT) monitor, or a flat panel display (FPD) monitor can be used as the display unit 18.

[0055] Fig. Figure 2A is a representation illustrating an example of feature points on a parking route according to the first embodiment. Fig. 2A illustrates a condition of a parking lot considered in the first embodiment (referred to below as an “environment”).

[0056] In the first embodiment, a route is defined from a "start" position, which serves as a starting point, to a "destination" position, which serves as a parking point, as shown in Fig. Figure 2A illustrates the route of vehicle 50. The sign "FP" in Fig. 2A indicates the position of the feature point, which will be described later. To estimate the self-position of the vehicle 50 of a specific size, a representative position within the vehicle 50 must first be determined (hereinafter referred to as a representative vehicle point X). In the first embodiment, the representative vehicle point (representative point) X is located at a midpoint between the rear wheels in the vehicle's width direction, as shown in Fig. Figure 2B illustrates this. The representative vehicle point X is not limited to the position shown above and may, for example, be located at the center of gravity of vehicle 50.

[0057] Fig. 2B, which is a top view of the in Fig. The environment illustrated in Figure 2A is a representation that illustrates an example of a camera position and positions of feature points in a keyframe according to the first embodiment.

[0058] As in Fig. As illustrated in Figure 2B, the vehicle 50, encompassing the camera 14, is driven from a starting point SP to a destination point GP. In the self-position estimation processing of the first embodiment, the environment is recorded and mapped by the camera 14 before the vehicle 50 actually drives, so that associated data correlated with the image previously captured by the camera 14 (i.e., the recorded image) is generated as a map. A character S in Fig. 2B specifies a driving trajectory of vehicle 50 at the time the map is generated. During actual parking, vehicle 50 is essentially driven along driving trajectory S. Hereinafter, a route along which vehicle 50 is driven during actual parking (i.e., actual driving) can be referred to as a driving route if it is distinct from driving trajectory S.

[0059] In Fig. Figure 2B illustrates the small circles indicating the positions of the feature points FP. In the first embodiment, each feature point exhibits a greater luminance difference than a predetermined value in the captured image. This luminance difference is based, for example, on shadows caused by a projecting and recessed section of a building, as well as the design or patterns of a wall surface of the building. Consequently, as shown in Fig. As illustrated in Figure 2B, the feature points are obtained, for example, on a wall surface WS of a building 30 in the vicinity and / or a corner section of the building 30. The multiple feature points FP are essentially selected and determined on a map. In the first embodiment, to distinguish the multiple feature points FP from one another, a "feature value" is correlated (registered) with each of the feature points FP. The feature value according to the first embodiment corresponds, for example, to a pattern of a difference in luminance in the vicinity of the corresponding feature point.

[0060] The symbol CP in Fig. 2B specifies multiple image capture points (i.e., for example, 12 image capture points in Fig. 2B), at which the camera captures 14 corresponding images during map generation. In the first embodiment, images captured at the multiple image acquisition points CP are referred to as "keyframes," which, together with the feature points FP and their correlated information, form the data for map generation. Each of the keyframes at this point in time serves as an example of a reference image. In particular, the keyframes are the multiple images previously captured at the multiple image acquisition points CP. Information regarding the feature points is correlated with the respective keyframes.In the following explanation, the image acquisition points where images are taken during map generation are referred to as map image acquisition points (CP), and the image acquisition points where images are taken during actual parking are referred to as driving image acquisition points. Additionally, the images taken at the driving image acquisition points are referred to as driving images.

[0061] As noted above, according to the first embodiment, the CPU 12A reads the self-position estimation program stored in the ROM 12B and writes the program to RAM 12C for execution, in order to be used as in Fig. 3 illustrated units or sections to function.

[0062] Fig. Figure 3 is a block diagram illustrating an example of a functional configuration of the self-position estimation device 10 according to the first embodiment. As shown in Fig. As illustrated in Figure 3, the CPU 12A of the self-position estimation device 10 functions according to the first embodiment as a detection unit 20, a calculation unit 22, an estimation unit 24 and an addition unit 26. Map information 16A is previously stored in the storage unit 16.

[0063] The map information 16A according to the first embodiment is explained below. The map information 16A includes the following information. (1) Coordinates of each feature point FP, shown in three dimensions (three-dimensional position coordinate (Xp, Yp, Zp)) (2) Three-dimensional position coordinate (Xc, Yc, Zc) and attitude (roll angle, pitch angle and yaw angle) of the camera in each keyframe (3) Feature value in each key image

[0064] During map generation, a driver pre-drives a vehicle along the driving trajectory S to estimate the coordinates of the feature points (three-dimensional positions) and the position of camera 14 in the key image from the image captured by camera 14 while driving. Such an estimation can be performed, for example, by simultaneous visual localization and mapping (Visual SLAM). In particular, the three-dimensional positions of the feature points and the position of the camera in the key image are estimated from the captured image using Visual SLAM. The estimation results and feature values ​​of the feature points in the key image are then recorded in the map information 16A.

[0065] In particular, an oriented FAST and a rotated BRIEF SLAM (ORB-SLAM) can be used as examples for capturing feature points. In the ORB-SLAM, a corner is captured as the feature point, with the capture of the corner being achieved through features from an accelerated segment test (FAST). The ORB-SLAM uses ORB to describe feature values. The ORB is based on binary robust independent elementary features (BRIEF) and is designed to include scale invariance and rotation invariance. The ORB-SLAM is disclosed in Publication 2, so a detailed explanation of it is omitted.

[0066] As noted above, a coordinate of each feature point, at least one feature value registered for each feature point, and the position and attitude of camera 14 are correlated and stored in the storage unit 16 as map information 16A when the multiple images serving as key images are taken at the map image acquisition points CP during map generation. The self-position estimation processing, which will be explained later, refers to the map information 16A stored in the storage unit 16.

[0067] The detection unit 20 according to the first embodiment detects the feature points in the driving image, which is recorded by the camera 14 in a state in which the vehicle is driven along the route. For example, the FAST is used to detect the feature points, as noted above.

[0068] The calculation unit 22 according to the first embodiment calculates the feature value that specifies the feature of each of the feature points recorded by the acquisition unit 20. For example, the ORB is used as the feature value, as noted above.

[0069] The estimation unit 24, according to the first embodiment, selects a similar key image, serving as an example of a similar image that is similar to the driving image, from the multitude of key images based on the feature values ​​calculated by the computation unit 22. In a case where the driving image is obtained for the first time (i.e., the initial or output key image is obtained), the similar key image that is most similar to the driving image is selected from a basket of visual words, for example, based on the feature values ​​of the feature points in the driving image. The basket of visual words is a tool for determining a degree of similarity between images by expressing a large number of local features in the image through vector quantification and histograms.Extracting the keyframe from the basket of visual words takes some time. Nevertheless, using the basket of visual words, which requires a relatively long operation time, may not pose a problem with respect to the initial driving image obtained when the vehicle is stopped at the starting point SP. In a case where the driving image is obtained a second time or later, the keyframe closest to the previously estimated camera position is selected to be the similar keyframe.

[0070] Next, the estimation unit 24 correlates the feature point in the driving image and the feature point in the similar key image by comparing the respective feature values ​​of the previously mentioned feature points in the driving image and the similar key image.

[0071] Fig. Figure 4 is a representation that illustrates a correlation between the feature points in the driving image and the feature points in the similar key image.

[0072] As in Fig. As illustrated in Figure 4, feature points R1, R2, R3, R4, R5, and R6 in the driving image are correlated to feature points P1, P2, P3, P4, P5, and P6 in the similar key image. Specifically, in a case where ORB is used as the respective feature value for the feature point, an ORB feature in the driving image and an ORM feature in the similar key image are compared, and a distance between them is calculated. The distance between the aforementioned ORB features is calculated, for example, using a Hamming distance.

[0073] Fig. Figure 5 is a representation that illustrates a distance between the feature value of the driving pattern and the feature value of the similar key pattern according to the first embodiment.

[0074] In Fig. In section 5, an ORB feature of a feature point in the driving image and several ORB features of a feature point in the similar key image are represented as 8 bits for the sake of simplicity. The number of bits for the ORB feature is not limited to a specific number, such as eight. The multiple ORB features of the similar key image are assigned indices 1, 2, ... for identification. The Hamming distance is represented by the number of bits that differ between the ORB feature in the driving image and each of the ORB features (indices 1, 2, ...) in the similar key image. In the example of Fig. Five bits in each of the indices 1, 2, ... that differ from the bits of the ORB feature of the feature point in the driving image are underlined. Specifically, the Hamming distance of index 1 with respect to the feature value of the driving image is four, and the Hamming distance of index 2 with respect to the feature value of the driving image is two. The minimum value of the Hamming distance among the aforementioned indices 1, 2, ... with respect to the feature value of the driving image is used as the distance between the feature values ​​of the driving image and the similar image.

[0075] In a case where the driving pattern is obtained for the first time, the individual feature value for each of the feature points in the similar key image is registered. This means that the number of indices indicating the feature value(s) of each of the feature points is one. In a case where the driving pattern is obtained a second time or later, the feature value(s) are additionally recorded, as described in Fig. As illustrated in Figure 5, the addition unit 26 is registered, which will be explained later, so that the number of indices serving as registered feature values ​​can be two or greater than two. Therefore, in the case where the driving image is obtained a second time or later, the minimum value of the Hamming distance calculated between the feature values ​​of the driving image and the similar image is applied as the distance between them.

[0076] In particular, estimation unit 24 calculates respective distances (for example, Hamming distances) between the multiple feature values ​​of the feature point in the similar key image and the feature value of the feature point in the driving image, in order to correlate the feature point in the driving image and the feature point in the similar key image where the multiple feature values ​​(for example, ORB features) are registered. Subsequently, in a case where the minimum value of calculated distances is less than or equal to a predetermined value, estimation unit 24 correlates the feature point in the driving image and the feature point in the similar key image. Consequently, the distance between the aforementioned feature values ​​can easily fall at or below the predetermined value.The correlation of the feature points between the driving image and the key image can also be obtained in a case where the feature value of the feature point in the driving image changes due to a change in brightness in the environment.

[0077] Instead of the ORB feature, a vector between the feature points can be used. In this case, a Euclidean distance can be used instead of the Hamming distance.

[0078] Next, the estimation unit 24 estimates the position and attitude of its own vehicle on the route based on the correlation result obtained above. Specifically, the estimation unit 24 estimates the position and attitude of the camera 14 based on the position of the feature point in the driving image and the position of the feature point in the similar key image, which are correlated with each other. In the first embodiment, a sum of projection differences, expressed as differences between the respective positions of the feature points in the driving image and the respective positions of projection points obtained by projecting the feature points in the similar key image onto the driving image, is minimized at the estimated position and attitude of the camera 14, based on the position and attitude of the camera 14 at the time the similar key image is captured.

[0079] Fig. Figure 6 is a representation that explains the projection difference according to the first example. Fig. Figure 6 illustrates the positions of feature points R1 to R6 in the map image, the positions of feature points P1 to P6 on a map coordinate CM correlated with the map (i.e., three-dimensional positions of feature points P1 to P6), and the positions of projection points V1 to V6, which are obtained by projecting the positions of feature points P1 to P6 correlated with the map onto the map image. In the map coordinate CM, axes X, Y, and Z indicate a horizontal direction, a depth direction, and a vertical direction, respectively. In this case, a projection difference ε1 with respect to the position of feature point P1 serves as a difference between the position of feature point R1 and the projection point V1. The preceding definition of the projection difference is likewise applied to the positions of feature points P2 to P6.Because false correlation is minimized, the position and attitude of camera 14 can be accurately estimated according to the present embodiment.

[0080] Next, the estimation unit 24 estimates the position and attitude of the own vehicle (i.e., obtains an own position estimate) on the route by converting the position and attitude of camera 14, estimated as above, to a representative point of the own vehicle. The position of the own vehicle corresponds to the position of the representative vehicle point X on the map. In the first embodiment, because the relative position between the representative vehicle point X and camera 14 is pre-detected, the position and attitude of camera 14 are converted to the position of the representative vehicle point X based on the relative position relationship described above.

[0081] The addition unit 26 according to the first embodiment additionally registers the feature value of the feature point on the driving image correlated with the feature point on the similar key image by the estimation unit 24 as the feature value of the feature point in the key image that serves as the similar key image in the map information 16A. For example, the addition unit 26 can selectively additionally register the feature value of the feature point to the map information 16A whose projection difference is less than or equal to a predetermined value (such a feature point can be referred to as an inside one) among the feature points in the driving image correlated with the feature points in the similar key image. In the example of Fig. 6. The feature values ​​of feature points R1, R2, R3, and R6 are additionally recorded in the driving image. The feature value of the feature point with a large projection difference is prevented from being recorded.

[0082] In a case where the number of multiple characteristic values ​​registered for the characteristic point in the similar key image reaches an upper limit, the addition unit 26 can calculate distances (for example, Hamming distances) between all pairs of characteristic values ​​selected from the registered multiple characteristic values ​​and the characteristic value to be added. The addition unit 26 can delete one of the characteristic values, where the median distance of such a characteristic value with respect to the other characteristic values ​​is the smallest among the median distances between all the preceding pairs of characteristic values.

[0083] Fig. Figure 7 is a representation that explains the feature value that is deleted in a case where the number of multiple feature values ​​registered for each of the feature points in the similar key image reaches the upper limit.

[0084] In the example of Fig. 7 is the upper limit of four, so indices 1 to 4 are assigned to the registered characteristic values ​​for a characteristic point. Another index, i.e., an index 5, is assigned to an additional characteristic value for the preceding characteristic point. Subsequently, the Hamming distance between index 1 and each of the indices 1 to 5 is calculated. The calculation result (0, 15, 9, 13, 20) is obtained as an example. In this case, the median, or mean, of the Hamming distances is 13 (which in Fig. (7 is underlined). Similarly, the calculation result (15, 0, 10, 5, 29) is obtained for index 2 as an example. In this case, the median of the Hamming distances is 10. Furthermore, the calculation result (9, 10, 0, 7, 12) is obtained for index 3 as an example. In this case, the median of the Hamming distances is 9. Furthermore, the calculation result (13, 5, 7, 0, 31) is obtained for index 4 as an example. In this case, the median of the Hamming distances is 7. Finally, the calculation result (20, 29, 12, 31, 0) is obtained for index 5 as an example. In this case, the median of the Hamming distances is 20.

[0085] Consequently, the median of the Hamming distances for each of the indices 1 to 5 is obtained as (13, 10, 9, 7, 20). The minimum value of the median of the Hamming distances is therefore 7, so the corresponding feature value of index 4 is deleted.

[0086] Next, an operation of the self-position estimation device 10 according to the first embodiment will be performed with reference to Fig. 8 explained. Fig. Figure 8 is a flowchart illustrating an example of a processing sequence by the self-position estimation program according to the first embodiment.

[0087] In a case where an instruction to start the self-position estimation program is executed, the self-position estimation processing is initiated by a driver driving vehicle 50 towards the destination GP from a state where vehicle 50 is stopped at the starting point SP, as in Fig. As shown in Figure 2B, the process is carried out. Alternatively, the self-position estimation processing can be applied, for example, during the automated driving of vehicle 50 from the starting point SP to the destination point GP. In the first embodiment, the map information 16A is generated in advance and stored in the memory unit 16.

[0088] In the first embodiment, the self-position estimation program is pre-stored, for example, in ROM 12B. Alternatively, the self-position estimation program can be stored, for example, in a state where it is stored on a portable storage medium to be readable by a computer, or it can be delivered via a communication device, such as a network interface. The portable storage medium can, for example, include a Compact Disc (CD-ROM), a Digital Versatile Disc (DVD-ROM), or a Universal Serial Bus (USB) device.

[0089] In step 100 in Fig. 8. The acquisition unit 20 receives the driving image, which is recorded by the camera 14 while the vehicle 50 is in motion. The driving image is acquired at predetermined time intervals after the vehicle 50 has started from the starting point SP. For example, the driving image is acquired every 33 ms (milliseconds), which serves as a common video rate.

[0090] The recording unit 20 records the feature points from the driving image in step 102. The feature points are recorded, for example, by the FAST.

[0091] In step 104, the calculation unit 22 calculates the characteristic value that represents one characteristic of each of the characteristic points recorded in the preceding step 102. The characteristic values ​​are calculated, for example, based on the ORB.

[0092] In step 106, the estimation unit 24 determines whether the driving image is being obtained for the first time or not. According to the self-position estimation processing of the first embodiment, a search procedure for finding the similar key image is modified in subsequent steps depending on whether the driving image is being obtained for the first time, for the second time, or later. If it is determined that the driving image is being obtained for the first time (i.e., a positive determination is made in step 106), the process proceeds to step 108. If it is determined that the driving image is not being obtained for the first time, i.e., the driving image is being obtained for the second time or later (i.e., a negative determination is made in step 106), the process proceeds to step 110.

[0093] In step 108, the estimation unit 24 selects the key image that is most similar to the driving image, for example, using the basket of visual words, based on the feature values ​​calculated in the preceding step 104. Conversely, in step 110, the estimation unit 24 selects the similar key image that is closest to the previously estimated camera position.

[0094] The estimation unit 24 correlates the feature values ​​of the respective feature points between the similar key image and the driving image (i.e., pairing), as in Fig. 4 and Fig. 5 is illustrated in step 112.

[0095] In step 114, the estimation unit 24 estimates the position and attitude of the camera 14 so that a projection difference is minimized, where the projection difference is defined as a difference between the position of the feature point in the driving image and a position of a projection point obtained by projecting the feature point in the similar key image onto the driving image.

[0096] The estimation unit 24 converts the position and attitude of camera 14, estimated in step 114 above, to the position and attitude of the vehicle itself in step 116. The position of the vehicle itself at this time corresponds to the position of the representative vehicle point X on the map. In the first embodiment, because the relative position between the representative vehicle point X and camera 14 was pre-detected, the position and attitude of camera 14 are converted to the position of the representative vehicle point X based on the aforementioned relative position relationship. Because such a conversion is obtained through a simple calculation, the position of the vehicle itself is estimated within a short time period while the vehicle is being driven, which can reduce the delay time of a vehicle operation control system.As a result, the vehicle can be guided with high precision to a target route and parked in a confined space, which, for example, leads to a reduction in parking space.

[0097] In step 118, the addition unit 26 additionally registers the feature value of the feature point whose projection difference is less than or equal to the predetermined value (such a feature point is an inside one) on the map information 16A, as the feature value of the feature point in the similar key image, among the feature points in the driving image correlated with the feature points in the similar key image in the preceding step 112.

[0098] The addition unit 26 determines in step 120 whether or not an image reference is complete at all driving image acquisition points. If the driving image is being acquired for the first time, a negative determination is made in step 120, so the process returns to step 100 to continue acquiring the driving image. If the driving image is being acquired a second time or later, a positive determination is made in step 120, so the current routine of the self-position estimation program is terminated.

[0099] According to the first embodiment, the feature value of the feature point in the driving image is correlated with the feature value of the feature point in the similar key image (reference image) as the feature value of the similar key image in addition to the feature values ​​that were previously recorded. The feature value of the feature point in the driving image that is subsequently obtained is compared with the recorded feature values, including the previously added feature value. Consequently, the range for determining that the feature values ​​are similar is increased, thereby achieving correlation between the feature points even when the feature value varies due to changes in ambient light. The vehicle's own position can be accurately estimated.In a case where the self-position estimation device according to the first embodiment is applied to an automatic parking system, the vehicle can be guided with high accuracy to a target route and parked in a confined space, which, for example, leads to a reduced parking space.

[0100] The self-position estimation device according to the first embodiment has been explained above. Alternatively, the embodiment can be a program that causes a computer to operate and function as each section contained in the aforementioned self-position estimation device. Furthermore, the embodiment can alternatively be a storage medium readable by a computer that stores the aforementioned program.

[0101] The design of the self-position estimation device according to the first embodiment is an example and can be appropriately and adequately changed or modified.

[0102] The sequence of the program described above is an example and can be modified appropriately, i.e., for example, an unnecessary step can be omitted, a new step added, or the process sequence changed appropriately.

[0103] In the first embodiment described above, the program is executed, so that the process is implemented through a software configuration using the computer. Alternatively, the process can be implemented through a hardware configuration or a combination of hardware and software configuration.

[0104] The self-position estimation device 10 according to a second embodiment is explained below. Configurations of the second embodiment that are essentially the same as those of the first embodiment bear the same reference numerals, and a detailed explanation is omitted. In the second embodiment, the keyframe serves as a reference image. The feature point FP in Fig. 2B correlates with the key image and serves as an environmental feature point in the second embodiment. The feature point FP in Fig. 2B correlates with the driving pattern and serves as a correlation feature point according to the second embodiment.

[0105] Setting the feature point detection area according to the second embodiment is described with reference to the Fig. 9 and Fig. 10 explained. As noted above, the distribution of feature points FP in the image (i.e., the key image and the driving image) captured by camera 14 as the vehicle 50 drives can change depending on factors such as the road surface condition on which the vehicle 50 is driven. In the second embodiment, a gradient (slope or incline) of a road surface on which the vehicle 50, serving as a target for estimating its position, is assumed to be a factor influencing the distribution of feature points across the entire captured image. In particular, in a case where there is a high probability that the distribution of feature points across the entire captured image is biased or uneven due to the gradient of the road surface, a feature point (correlated feature) capture area is configured to be modified.

[0106] Fig. Figure 9A illustrates vehicle 50 being driven on a road surface R with a gradient. Fig. Figure 9A illustrates an example where the road surface R ahead of vehicle 50 is inclined upwards. An example where the road surface R ahead of vehicle 50 is inclined downwards exhibits the same concept. Fig. 9A specifies position PS0 as a reference position of vehicle 50, and position PS1 specifies a position at the midpoint of a climb by vehicle 50. Vehicle 50 is located at each of positions PS0 and PS1. At this point, the tilt angle of camera 14 of vehicle 50 located at position PS0 is defined as θf, and the tilt angle of camera 14 of vehicle 50 located at position PS1 is defined as θ1. The tilt angle according to the second embodiment, i.e., a tilt angle θ, is a tilt angle of the optical axis of camera 14 measured relative to a horizontal direction. A direction approaching a ground surface from the horizontal direction is defined as a positive direction. A road surface gradient difference Δθ is defined by (θ1 - θf) (i.e., Δθ = θ1 - θf). In the above definition, the road surface gradient difference Δθ is equal to a Fig. 9A illustrates the angle. In the second embodiment, although the optical axis of the camera 14 is arranged to be directed slightly downwards as mentioned above, the following explanation is carried out in a state in which the direction of the optical axis of the camera 14 is directed horizontally towards the road surface R, which is flat and not inclined, for the purpose of ease of understanding.

[0107] Fig. Figure 9B schematically illustrates a feature point acquisition area As (As0), which serves as an acquisition area in an image G captured by the camera 14. The feature point acquisition area As according to the second embodiment corresponds to an area specified by a search area of ​​the feature points FP in the image G. In particular, the feature points FP are not deleted from the entire image G, but rather from a specified and limited area thereof. As in Fig. As illustrated in Figure 9B, the feature point acquisition area As is specified by an upper end Gu (Gu0) and a lower end Gd (Gd0), which are specified as upper and lower positions respectively in the image G.

[0108] The feature point acquisition area As is not limited to being specified in the manner described above. For example, the feature point acquisition area As can be specified by an intermediate point Gc between the upper end Gu and the lower end Gd and an image width Ws. Fig. Figure 9B illustrates in particular the feature point detection area As0 in a case where the vehicle 50 is located at the reference position on a substantially flat (i.e., non-sloping) road surface. The feature point detection area As0 is defined by the upper end Gu0 and the lower end Gd0.

[0109] Fig. Figure 9C illustrates a relationship between the road surface gradient difference Δθ and the feature point acquisition area As. In particular, it illustrates Fig. 9C the feature point detection area in a state where the road surface gradient difference Δθ falls within a range of Δθ1 to Δθ2 in a case where the road surface slopes upwards ahead of the vehicle. A point at which the road surface gradient difference Δθ in Fig. 9C, where zero indicates a position where the road surface gradient difference is essentially zero, and the road surface R is flat (i.e., not sloped). This means that the positive and negative signs of the road surface gradient difference Δθ change at the point Δθ = 0 (i.e., Δθ1 < 0 and Δθ2 > 0). The feature point acquisition area As0 in a case where the road surface gradient difference Δθ is zero is a region between the upper end Gu0 and the lower end Gd0. Fig. Figure 9C illustrates the example in which the width of the feature point detection area As is constant within a range of road surface gradient difference Δθ from Δθ1 to Δθ2. The width of the feature point detection area As can be changed depending on the road surface gradient difference Δθ.

[0110] The Fig. 10A and Fig. Figures 10B each illustrate an example of the feature point detection area As in an actual image. Fig. Figure 10A illustrates an example of the feature point acquisition area (i.e., the feature point acquisition area As0) in a case where the vehicle 50 is on the road surface R, which is flat, and whose road surface gradient difference Δθ is essentially zero. As shown in Fig. As illustrated in Figure 10A, in the feature point acquisition area As0, the upper end Gu0 is defined at a position below the upper end of the image G by a predetermined length, while the lower end Gd0 is defined at a position above the lower end of the image G by a predetermined length. This means that in the feature point acquisition area As0, the acquisition of the feature point FP is not performed within a predetermined area at the upper end of the image G and a predetermined area at the lower end of the image G.

[0111] Fig. Figure 10B illustrates an example of the feature point acquisition area (i.e., a feature point acquisition area As1) in a case where the vehicle 50 is located on the road surface R, whose road surface gradient difference Δθ is greater than zero (Δθ > 0). As in Fig. As illustrated in Figure 10B, in the feature point acquisition area As1, an upper end Gu1 is defined at a position essentially equal to the upper end of the image G, whereas a lower end Gd1 is defined at a position in the vicinity of a center in a top and bottom direction of the image G. This means that in the feature point acquisition area As1, the acquisition of the feature point FP is not performed within a range from the upper end of the image G to the lower end Gu1, nor within a range from the lower end Gd1 to the lower end of the image G. In explaining a relationship between the feature point acquisition area As0 and the feature point acquisition area As1 with reference to Fig. 9A it can be assumed that the feature point detection area in a case where the vehicle 50 is at position PS0 is the feature point detection area As0, and the feature point detection area in a case where the vehicle 50 is at position PS1 is the feature point detection area As1.

[0112] Next, a self-position estimation method according to the second embodiment will be used with reference to Fig. 11 explained. Fig. Figure 11 is a flowchart that describes the sequence of the self-position estimation program for estimating the self-position of vehicle 50. According to the in Fig. Figure 11 illustrates a self-position estimation program. When an instruction to start the program is executed, the CPU 12A of the control unit 12, which serves as a setting unit, reads the self-position estimation program from the memory device, such as the ROM 12B, as an example, and develops the program in the memory device, such as the RAM 12C, as an example, in order to execute the program. In the second embodiment, a map is generated beforehand and stored in the memory unit 16. The self-position estimation processing is carried out in conjunction with the driving of the vehicle 50 by a driver towards the target point GP from a state in which the vehicle 50 is stopped at the starting point SP, as shown in Figure 11. Fig. 2B illustrates this. Alternatively, the self-position estimation processing can be applied as an example to the automatic driving of vehicle 50 from the starting point SP to the destination point GP.

[0113] In the second embodiment, the self-position estimation program is pre-stored, for example, in ROM 12B. Alternatively, the self-position estimation program can be provided in a state where it is stored in a portable storage medium to be readable by a computer, or it can be delivered via a communication device, such as a network interface.

[0114] The driving image is obtained by capturing it in step S100 using camera 14. The driving image is acquired at predetermined time intervals after the vehicle starts 50 meters from the starting point SP. For example, the driving image is acquired every 33 ms (milliseconds), which is a common video rate.

[0115] Step S102 determines whether the driving image is being acquired for the first time or not. According to the self-position estimation processing of the second embodiment, a feature point detection area setting procedure and a key image search procedure are modified in subsequent steps depending on whether the driving image is being acquired for the first time, for the second time, or later. If a positive determination is made in step S102, the process proceeds to step S104, whereas if a negative determination is made, the process proceeds to step S112.

[0116] The feature point acquisition area As is set to an initial or output feature point acquisition area, which is specified beforehand, in step S104 because the road surface gradient difference Δθ has not yet been estimated at the time when the driving image is first acquired. The output feature point acquisition area can be specified according to any method. For example, the feature point acquisition area As0, which is located at a midpoint of the top-and-bottom direction in image G, is as shown in Fig. Figure 9C illustrates how the initial feature point acquisition area is defined. As noted above, the feature point acquisition area As0 is a limited area within image G. Because the initial feature point acquisition area is the limited area, as described above, it prevents feature points on the road surface, which are difficult to distinguish from one another, from being captured. For example, feature points on a building, which can be easily distinguished from one another, are more readily captured.

[0117] The characteristic points are recorded, and their characteristic values ​​are calculated in step S106. The characteristic points can be recorded by the FAST, and the characteristic values ​​can be calculated based on the ORB as mentioned above.

[0118] A keyframe, KF0, which is the most similar keyframe to the driving image, is selected, for example, using a basket of visual words based on the feature values ​​calculated in step S106. The basket of visual words is a tool for determining the degree of similarity between images by expressing a large number of local features in the image through vector quantification and histograms. Extracting the keyframe from the basket of visual words requires a certain amount of time. Nevertheless, because such an operation is performed to obtain the initial driving image when the vehicle is stopped at the starting point SP, using the basket of visual words, despite its relatively long operation time, should not pose a problem.

[0119] The feature values ​​are compared between the key image KF0 and the driving image in step S110 to correlate the feature points between the key image KF0 and the driving image (i.e., pairing). Because the feature points captured in the feature point acquisition area As0 are easily distinguishable from one another, an accurate correlation is achievable.

[0120] In step S124, the position and attitude of the camera are estimated so that a projection difference is minimized, where the projection difference is defined as the difference between a three-dimensional position (i.e., a position on the map) of the feature point obtained on the basis of a correlation between the driving image and the key image KF0, and the position of the feature point in the driving image. The projection difference according to the present embodiment is determined with reference to Fig. 12 explained. Fig. Figure 12 illustrates positions a, b, and c of the feature points in the driving image, positions A, B, and C of the feature points on a map coordinate CM correlated on the map, and positions a', b', and c' of the feature points obtained by projecting the positions A, B, and C of the feature points onto the map coordinate CM in the driving image. At this point, a projection difference εa for position A is a difference between position a and position a'. The same is applied to positions B and C. Because the difference between the position of the feature point in the driving image and the position of the feature point on the map coordinate is small according to the present embodiment, the position and attitude of the camera are estimated accurately.

[0121] The position and attitude of the camera, estimated in step S124, are converted in step S126 into the position and attitude of the vehicle itself. The vehicle's own position (i.e., its own position) corresponds to the position of the representative vehicle point X on the map. In the second embodiment, because a relative position between the representative vehicle point X and camera 14 has been pre-detected, the position and attitude of camera 14 are converted to the position of the representative vehicle point X based on the aforementioned relative position relationship. Because such a conversion is obtained through a simple calculation, the vehicle's own position is estimated within a short time period while the vehicle is in motion, which can reduce the delay time of a vehicle operation control system.As a result, the vehicle can be guided to a target route with high precision and parked in a confined space, which, for example, leads to a reduction in parking space.

[0122] Step S128 determines whether images (driving images) are acquired at all driving image acquisition points or not. If the driving image is being acquired for the first time, a negative determination is made in step S128, and therefore the process returns to step S100 to continue acquiring the driving image.

[0123] On the other hand, in step S112, a key image KFf is sought and selected that is the closest key image to a forward position by a predetermined distance, i.e., for example, five meters, from the position of camera 14, which was previously estimated. In particular, because a distance to the road surface R, which appears at the center of the image, is previously obtained based on an installation angle of camera 14, a key image that is closest to this position is selected.

[0124] A difference (θ1 - θf) between the camera tilt angle θf of the key image KFf and the camera tilt angle θ1 previously estimated is defined in step S114 as the road surface gradient difference Δθf.

[0125] In step S116, the feature point acquisition area As is defined using a Fig. The diagram illustrated in Figure 9C is set based on the road surface gradient difference Δθf specified in step S114. Specifically, the upper end Gu and the lower end Gd of the feature point capture area As are specified depending on the magnitude of the road surface gradient difference Δθf. As a result, regardless of the road surface gradient difference Δθ, feature points on the road surface that are difficult to distinguish from one another are prevented from being captured, and feature points that are easily distinguishable from one another, such as feature points on a building, can be captured more effectively (see Figure 9C). Fig. 10B). Additionally, because the number of feature point calculations is limited by the limitation of the feature point acquisition area As, calculations for obtaining feature values ​​and correlating future points can decrease. Therefore, even in a system where computing resources are limited, estimating the position of one's own vehicle within a short time period is achievable.

[0126] In step S118, the recording of the characteristic points and the calculation of the characteristic values ​​of the recorded characteristic points are carried out in the same way as in step S106.

[0127] In step S120, a key image KF1 is selected that includes the largest number of feature points that were previously correlated.

[0128] The feature values ​​are compared between the key image KF1 and the driving image, so that the feature points are correlated with each other (pairing). In the second embodiment, regardless of the road surface gradient difference Δθ, a large number of feature points, which serve as comparison targets and are easily distinguishable from one another, are provided. Therefore, an accurate correlation can be achieved.

[0129] Next, the operations of steps S124 to S128, as described above, are performed. However, in step S124, the key image KF0 is replaced by the key image KF1. Subsequently, if a positive determination is made in step S128, the currently executed own position estimation program is terminated.

[0130] As noted above, according to the self-position estimation device, the self-position estimation method, and the self-position estimation program of the second embodiment, estimating the vehicle's own position within a short time period is also achievable in a system with limited computing resources. In the second embodiment, because the detection range of the feature points in the image is limited, the detection time is reduced. Furthermore, because the number of detected feature points is limited, calculations for obtaining the feature values ​​and correlating future points can be reduced. Therefore, estimating the vehicle's own position within a short time period is also achievable in a system with limited computing resources.

[0131] Additionally, according to the self-position estimation device, the self-position estimation method, and the self-position estimation program of the second embodiment, the vehicle's self-position can be estimated with high accuracy, even when the road surface gradient changes. Because the feature point detection area in the driving image is modified depending on the road surface gradient of the vehicle's self-position relative to the vehicle's frontal direction, feature points, for example, on a building that are easily distinguishable from one another, are detected more frequently, whereas feature points, for example, on a road surface that are difficult to distinguish from one another, are detected less frequently. The feature points can be precisely correlated between the key image and the driving image.Even with changes in the road surface gradient, the vehicle's own position can be accurately estimated with the minimum number of feature point measurements.

[0132] Furthermore, in a case where the self-position estimation device, the self-position estimation method, and the self-position estimation program of the second embodiment are applied to an automatic parking system, the vehicle is guided to a target route with high accuracy. The vehicle can be parked in a narrow space, resulting in a reduced parking footprint. In the second embodiment, the vehicle's own position is estimated within a short time period while the vehicle is being driven, which can reduce the delay time of the vehicle operation control. Therefore, the vehicle can be guided to a target route with high accuracy and, for example, parked in a narrow space, resulting in a reduced parking footprint.

[0133] Next, the self-position estimation device according to a third embodiment will be explained. As in Fig. As illustrated in Figure 13, a self-position estimation system 40 is constructed according to the third embodiment, comprising an imaging device 210, which serves as an image acquisition unit, a sensor 220 and a self-position estimation device 300.

[0134] The imaging device 210 is a camera fixed to the vehicle to capture an image in a predetermined direction relative to the vehicle.

[0135] In particular, the imaging device 210 is, for example, a vehicle-mounted monocular camera that images and records a road ahead of the vehicle. The imaging device 210 records a road image ahead of the vehicle as an example of a road image.

[0136] The captured image (i.e., recorded image) is not limited to an image in front of the vehicle and can be an image to the rear of the vehicle or an image on a right or left side of the vehicle.

[0137] The sensor 220 consists of several sensors comprising at least one sensor for acquiring brightness, such as an illuminance meter, at least one sensor for acquiring time, and at least one sensor for acquiring three-dimensional position information or an angle, such as a GPS and a magnetic sensor. The aforementioned magnetic sensors can, for example, be provided on a device that differs from the self-position estimation device 300. In the third embodiment, the sensor 220 consists of the illuminance meter and the GPS.

[0138] The self-position estimation device 300 estimates the vehicle's position using an environmental map that stores feature values ​​registered for several feature points whose positions are known. The self-position estimation device 300 comprises a CPU, RAM, and ROM, which stores a program for executing a self-position estimation processing routine, described later. The self-position estimation device 300 is functionally structured as follows.

[0139] The self-position estimation device 300 comprises an image input unit 310, a feature point extraction unit 320, a feature value calculation unit 330, a sensor information input unit 340, a light source direction estimation unit 350, a sunshine state estimation unit 360, a correction factor decision unit 370, a correction factor storage unit 380, a feature value correction unit 390, an estimation unit 400, an environment map feature value storage unit 410 and a driver assistance system 500.

[0140] The image input unit 310 receives an input of an image from the imaging device 210.

[0141] In particular, the image input unit 310 receives an input of a captured image from the imaging device 210 and sends the aforementioned captured image to the feature point extraction unit 320.

[0142] The feature point extraction unit 320 extracts the multiple feature points from the image in front of the vehicle, which is captured by the imaging device 210.

[0143] In particular, the feature point extraction unit 320 extracts the multiple feature points from the image using a feature point extraction algorithm (for example, FAST).

[0144] The feature point extraction unit 320 sends the extracted feature points to the feature value calculation unit 330 and the correction factor decision unit 370.

[0145] The feature value calculation unit 330 calculates the feature values ​​based on a luminance of each of the feature points.

[0146] In particular, the feature value calculation unit 330 calculates the feature values ​​for the respective feature points obtained by the feature point extraction unit 320. For example, the feature value calculation unit 330 calculates the feature values ​​in 32 bytes each, based on ORB features.

[0147] The characteristic value calculation unit 330 sends the calculated characteristic values ​​to the characteristic value correction unit 390.

[0148] The sensor information input unit 340 receives brightness information from sensor 220 (obtained by the irradiance meter) and three-dimensional position information, along with the corresponding time, obtained via GPS. The sensor information input unit 340 transmits the three-dimensional position information and the corresponding time to the light source direction estimation unit 350 and sends brightness information to the sunshine condition estimation unit 360.

[0149] The light source direction estimation unit 350 estimates a light source direction relative to an imaging direction in which the image is captured by the imaging device 210, based on sensor information, at the time the image is captured.

[0150] In particular, the Light Source Direction Estimator 350 estimates the direction of the light source (for example, the direction of the sun relative to north) based on the time at which the position information is obtained by the GPS (hereinafter referred to as GPS position time). The third embodiment describes, for example, a case in which the light source is the sun. The Light Source Direction Estimator 350 estimates an angle and altitude of the sun at GPS position time based on an average sunrise time, sunset time, and noon on the date of the GPS position time, as an example.

[0151] The Light Source Direction Estimation Unit 350 estimates the imaging direction based on the sensor information used to calculate the vehicle's direction. For example, the vehicle's direction relative to north is estimated using the latitude / longitude obtained by the GPS at the current GPS position time and the latitude / longitude obtained by the GPS at a previous GPS position time. If the vehicle direction and the imaging direction coincide, the estimated vehicle direction is considered the imaging direction. If the vehicle direction and the imaging direction do not coincide (i.e., differ), the imaging direction can be estimated taking such a difference into account.

[0152] The light source direction estimation unit 350 estimates the light source direction relative to the imaging direction at the time the image is acquired, based on the estimated light source direction and imaging direction. The light source direction estimation unit 350 sends the estimation result, i.e., the estimated light source direction relative to the imaging direction, to the correction factor decision unit 370.

[0153] The Sunshine State Estimation Unit 360 estimates the sunshine state of the vehicle based on the brightness obtained from the sensor information.

[0154] In particular, the Sunshine State Estimator 360 estimates cloud cover based on radiation information from the radiation meter and predicts whether or not shadows will form. For example, if the brightness measured by the radiation meter is less than or equal to a predetermined value, it is estimated that no clouds or shadows will form.

[0155] The sunshine state estimation unit 360 sends the estimation result, i.e., whether shadows are formed or not, to the correction factor decision unit 370.

[0156] The correction factor decision unit 370 determines a feature value correction factor to correct the feature value of each of the several feature points extracted by the feature point extraction unit 320, such that a feature value is brought into a state in which the feature point of it is brought into the light source direction in which the environment map is generated by obtaining and storing the feature values ​​of the feature points whose positions are known, on the basis of the feature value of the feature point extracted by the feature point extraction unit 320, the estimated light source direction relative to the estimated imaging direction, and the light source direction relative to the imaging direction, which is previously obtained at the time when the feature value of the environment map is obtained.

[0157] In particular, the correction factor decision unit 370 first obtains the sunshine condition at the time the environment map is generated from the environment map feature value storage unit 410, and determines whether the sunshine condition estimated by the sunshine condition estimation unit 360 influences the sunshine condition at the time the environment map is generated or not.

[0158] The influence of the sunshine condition at the time the environment map is generated is determined on the basis of shadows estimated by the sunshine condition estimation unit 360, a difference between the direction of the sun (sunbeams) estimated by the light source direction estimation unit 350 and the direction of the sun when referring to the feature values ​​of the environment map, and whether the estimated direction of the sun is in front of the vehicle or not (i.e., whether the camera is facing into the light or not).

[0159] For example, in a case where there are no shadows in the sunshine state at the time the environment map is generated, it can be estimated that the amount of cloud cover is high. Therefore, at this time, the correction factor decision unit 370 determines that the sunshine state estimated by the sunshine state estimation unit 360 does not affect the sunshine state at the time the environment map is generated.

[0160] In a case where the correction factor decision unit 370 determines that the sunshine state estimated by the sunshine state estimation unit 360 influences the sunshine state at the time the environment map is generated, it is subsequently determined whether the direction of the sun relative to the imaging direction at the time the environment map is generated matches the direction of the sun relative to the imaging direction at the time the image (driving image) is taken or not.

[0161] In a case where it is determined that the direction of the sun relative to the imaging direction estimated by the light source direction estimation unit 350 is the same as the direction of the sun relative to the imaging direction at the time the feature value of the environment map is obtained, the correction factor decision unit 370 determines that the sunshine state estimated by the sunshine state estimation unit 360 is prevented from influencing the sunshine state at the time the environment map is generated.

[0162] Even if the direction of the sun relative to the imaging direction, as estimated by the light source direction estimation unit 350, does not match the direction of the sun relative to the imaging direction at the time the feature value of the environment map is obtained, as long as the sun's direction is in front of the vehicle (i.e., light is emitted from the front), the camera is pointed towards the light, and the luminance decreases uniformly and evenly, the correction factor decision unit 370 determines that the sunshine state estimated by the sunshine state estimation unit 360 is prevented from influencing the sunshine state at the time the environment map is generated.

[0163] On the other hand, in a case of sunlight being emitted from the rear, the correction factor decision unit 370 determines whether the direction of the sun relative to the imaging direction estimated by the light source direction estimation unit 350 is on the right or on the left relative to the direction of the sun at the time the environment map is generated.

[0164] The correction factor decision unit 370 estimates whether the feature point is a protrusion or a recess (i.e., it projects forward or is recessed). For example, an average luminance on a right side and an average luminance on a left side relative to the feature point can be obtained. Specifically, an area centered on the feature point (for example, ±15 pixels) is divided into a right half and a left half. The average luminance in the right half and the average luminance in the left half are then each referenced. In a case where the sun's rays shine on the left side relative to the feature point, and the average luminance of the left side is higher, the feature point is estimated to be the protrusion.In a case where the sun's rays shine on the left side relative to the feature point, and the average luminance of the right half is higher, the feature point is estimated to be the indentation. Additionally, in a case where the sun's rays shine on the right side relative to the feature point, and the average luminance of the right side is higher, the feature point is estimated to be the projection. In a case where the sun's rays shine on the right side relative to the feature point, and the average luminance of the left side is higher, the feature point is estimated to be the indentation.

[0165] Alternatively, as a method for estimating whether the feature point is the protrusion or the recess, an area centered on the feature point (for example, ± 16 pixels) can be divided into squares, or divided radially with respect to the feature point. The average luminance in the resulting areas can then be calculated and compared.

[0166] Accordingly, the correction factor decision unit 370 determines whether the feature point is the advantage or the exception for all of the feature points.

[0167] The correction factor decision unit 370 determines the feature value correction factor based on whether the direction of the sun relative to the estimated imaging direction by the light source direction estimation unit 350 is on a right side or a left side relative to the direction of the sun at the time the environment map is generated, and based on whether the feature point is the protrusion or the recess.

[0168] First, in a case where the correction factor decision unit 370 determines that the estimated sunshine state does not affect the sunshine state obtained at the time the environment map is generated, the feature value correction is not necessary. Therefore, the correction factor decision unit 370 determines a feature value correction factor A (see Fig. 14) comprising sequences that are all zero, as the feature value correction factor.

[0169] The correction factor decision unit 370 determines a feature value correction factor B (see Fig. 14) as the feature value correction factor in a case where the direction of the sun relative to the imaging direction estimated by the light source direction estimation unit 350 is a right side of the direction of the sun at the time the environment map is generated, and the feature value is the protrusion. Additionally, in a case where the direction of the sun relative to the imaging direction estimated by the light source direction estimation unit 350 is a right side of the direction of the sun at the time the environment map is generated, and the feature value is the recess, the correction factor decision unit 370 determines a feature value correction factor B' (see Fig. 14) as the feature value correction factor.

[0170] Furthermore, the correction factor decision unit 370 determines a feature value correction factor C (see Fig. 14) as the feature value correction factor in a case where the direction of the sun relative to the imaging direction estimated by the light source direction estimation unit 350 is a left side of the direction of the sun at the time the environment map is generated, and the feature point is the protrusion. In a case where the direction of the sun relative to the imaging direction estimated by the light source direction estimation unit 350 is a left side of the direction of the sun at the time the environment map is generated, and the feature point is the recess, the correction factor decision unit 370 determines a feature value correction factor C' (see Fig. 14) as the feature value correction factor.

[0171] The correction factor decision unit 370 determines feature value correction factors for all of the feature points and sends the determined feature value correction factors to the feature value correction unit 390.

[0172] The correction factor storage unit 380 stores various patterns of feature value correction factors that are predetermined for the feature values ​​of the feature points on the map.

[0173] The feature value correction factor is determined separately and individually depending on whether the estimated sunshine condition influences the sunshine condition at the time the environmental map is generated, the difference in the sun's direction relative to the feature point, whether the feature point is a protrusion or a recess, and the sun's direction relative to the imaging direction in which the image is taken. Examples of feature value correction factors are given in Fig. 14 illustrates. In the example of Fig. 14. Each of the feature value correction factors comprises a specific length depending on the feature value; for example, it comprises 256 sequences, each expressed by one of three values ​​[+1, 0, -1]. The feature value comprises a bit sequence consisting of 256 bits, each expressed by either 1 or 0. Therefore, the values ​​[+1, 0, -1] of the feature value correction factor sequences mean changing each bit in the bit sequence of the corresponding feature value to 1, leaving the bit as it is, and changing the bit to 0.

[0174] Different types and patterns of feature value correction factors are determined to fit the bits of each of the feature values ​​by considering different cases and situations based on whether the estimated sunshine condition affects the sunshine condition at the time the environment map is generated, whether the feature point is a protrusion or a recess, and the direction of the sun relative to the imaging direction in which the image is taken.

[0175] In a case where the estimated sunshine state does not affect the sunshine state at the time the environment map is generated, feature value correction is not necessary. Therefore, the feature value correction factor, where all sequences are zero, i.e., the feature value correction factor A in Fig. 14, decided or determined.

[0176] In a case where the estimated sunshine condition influences the sunshine condition at the time the environment map is generated, i.e., a shadow is formed, the feature value correction factor is considered in a case where the direction of the sun relative to the imaging direction is to the right of the direction of the sun at the time the environment map is generated, and the feature point is the protrusion, i.e., the feature value correction factor B in Fig. 14, for the feature value of the feature point on the map, decided or determined. Two regions (points) (for example, two areas 1a and 1b in Fig. 21) The luminance values ​​of the feature point on the map are compared by calculating the luminance in the two areas to obtain each bit of the feature point's value. Depending on whether the two areas are on the same side, i.e., the left or right side, relative to the feature point on the map, the value is determined by one of the 256 sequences of the feature value correction factor.

[0177] In a case where the two areas are on the same side, no effect is assumed on each bit of the feature value, even if shadows are formed relative to the feature point. Therefore, the value of the corresponding feature value correction factor sequence is set to zero.

[0178] For example, the feature value correction factor can be determined using two 5 x 5 pixel regions (xa, xb), where luminance calculations are performed on the two regions for comparison to obtain the x-th bit of the feature value. If the two aforementioned regions are located on the left or right side of the feature point, the value of the x-th sequence of the feature value correction factor for the preceding feature point is set to zero.

[0179] In a case where region xa is on the right side and region xb is on the left side (for example, regions 1a and 1b in Fig. 21) In regions 1a and 1b, where the luminance calculation is performed, the left side is darker (i.e., contains low luminance) when the sun's direction relative to the imaging direction at the time the image is taken is darker, and the right side is darker relative to the sun's direction at the time the environment map is generated, and the feature point is the protrusion because the shadow is formed on the left side. That is, the luminance in region xa is greater than the luminance in region xb. Therefore, the x-th bit of the feature value, which is affected by the sunlight state, is one (1).

[0180] As long as the xth bit of the feature value of the feature point on the map is one, no correction of the feature value is necessary. Therefore, the value of the xth sequence of the feature value correction factor is set to zero. Conversely, if the xth bit of the feature value of the feature point on the map is zero, the xth bit of the extracted feature value should be corrected to zero, so that the value of the xth sequence of the feature value correction factor is set to minus one (-1).

[0181] On the other hand, in a case where area xa is on the left and area xb is on the right, the luminance in area xa is less than the luminance in area xb, if the direction of the sun relative to the imaging direction at the time the image is taken is on the right relative to the direction of the sun at the time the environment map is generated, and the feature point is the protrusion, so that the x-th bit of the feature value is affected by the sunshine state is zero.

[0182] As long as the xth bit of the feature value of the feature point on the map is zero, no feature value correction is necessary. Therefore, the value of the xth sequence of the feature value correction factor is set to zero. Conversely, if the xth bit of the feature value of the feature point on the map is one, the xth bit of the feature value of the extracted feature point should be corrected to one, so that the value of the xth sequence of the feature value correction factor is set to plus one (+1). The above operation is performed with respect to all bits to determine the feature value correction factor B.

[0183] Similarly, the feature value correction factor B' is determined for a case where the sun's direction relative to the imaging direction at the time the image is taken is to the right relative to the direction at the time the environment map is generated, and the feature point is the depression or recess. The feature value correction factor C is determined for a case where the sun's direction relative to the imaging direction at the time the image is taken is to the left relative to the direction of the sun at the time the environment map is generated, and the feature point is the protrusion.The feature value correction factor C' is determined for a case where the direction of the sun relative to the imaging direction at the time the image is taken is on the left side, as compared to the direction of the sun at the time the environment map is generated, and the feature point is the depression.

[0184] Accordingly, various patterns of feature value correction factors are determined so that, even if the estimated sunshine condition influences the sunshine condition at the time the environment map is generated (for example, due to shadows), the feature value of the extracted feature point can be corrected with respect to such influence based on the direction of the sun's rays into the two areas (points) being compared and the shape (protrusion, depression) of the feature point.

[0185] The feature value correction unit 390 corrects the feature value of each of the multitude of feature points based on the feature value and the feature value correction factor, which conforms to the pattern according to the feature point (i.e., its shape and the direction of the sun).

[0186] In particular, the feature value correction unit 390 corrects the feature value by adding corresponding values ​​of the sequences of the feature value correction factor to corresponding bits of the feature value's bit profile. If a resulting bit obtained from the above addition exceeds +1, that bit of the feature value is corrected to one. If a resulting bit obtained from the above addition falls below zero, that bit of the feature value is corrected to zero.

[0187] In one case of Fig. For example, in 21 the feature value is represented by “1, 0, 0, ..., 1”, as in Fig. Figure 22 illustrates this. In a case where the characteristic value correction factor is represented by "-1, 1, 1, ..., 0", a value after the correction (corrected characteristic value) is represented by "0, 1, 1, ..., 1", which corresponds to the characteristic value in Fig. 20 agrees, which in the case of Fig. 19 is illustrated.

[0188] The feature value correction unit 390 sends the corrected feature values ​​for all of the feature points to the estimation unit 400.

[0189] The estimation unit 400 estimates the position of the vehicle based on the corrected feature values ​​of the multiple feature points extracted by the feature point extraction unit 320 and the feature values ​​of the feature points on the map.

[0190] In particular, the estimation unit 400 estimates the vehicle's position on the map by comparing the corrected feature values ​​with the feature values ​​at the feature points on the map. For example, the estimation unit 400 determines a position corresponding to one or a position with the feature value that shows the highest similarity between the corrected feature values ​​and the feature values ​​of the multiple feature points on the environment map as an estimated vehicle position on the environment map.

[0191] The estimation unit 400 then outputs the estimated vehicle position to the driver assistance system 500.

[0192] The environmental map feature value storage unit 410 stores the previously generated environmental map, the feature values ​​on the environmental map, and the sunshine condition at the time the environmental map is generated. The environmental map is generated, for example, in a mapping mode of the SLAM.

[0193] The driver assistance system 500 performs driver assistance based on the estimated vehicle position.

[0194] Next, an operation of a self-position estimation device according to the third embodiment will be described with reference to Fig. 15 explained. Fig. Figure 15 shows a self-position estimation processing routine that is performed by the self-position estimation device 300 according to the present embodiment.

[0195] First, in step T100, the correction factor decision unit 370 and the estimation unit 400 obtain the feature values ​​on the environmental map from the environmental map feature value storage unit 410. The correction factor decision unit 370 also obtains the sunshine condition at the time the environmental map is generated from the environmental map feature value storage unit 410.

[0196] The sensor information input unit 340 inputs and receives brightness information obtained by a radiation meter, three-dimensional position information obtained by a GPS, and the time of these inputs and outputs in step T110.

[0197] The image input unit 310 receives an input of an image from the imaging device 210 in step T120.

[0198] The feature point extraction unit 320 extracts several feature points from the image ahead of the vehicle, which is captured by the imaging device 210, in step T130.

[0199] The feature value calculation unit 330 calculates the feature values ​​based on a luminance of the multiple feature points in step T140.

[0200] In step T150, the correction factor decision unit 370 determines the feature value correction factor to correct the feature value of each of the plurality of feature points extracted by the feature point extraction unit 320, so that such a feature value is brought into a state in which the feature point is moved in the direction of the light source in which the environment map is generated by obtaining and storing the feature values ​​of the feature points whose positions are known, on a basis of the feature value of the feature point extracted by the feature point extraction unit 320, the estimated direction of the light source relative to the estimated imaging direction, and the direction of the light source relative to the imaging direction, which is obtained in advance at the time when the feature value of the environment map is obtained.

[0201] The feature value correction unit 390 corrects the feature value of each of the multiple feature points based on the feature value and the feature value correction factor that conforms to the pattern of the corresponding feature point (i.e., its shape and the direction of the sun) in step T160.

[0202] The estimation unit 400 estimates the position of the vehicle (own vehicle) based on the corrected feature values ​​of the multiple feature points extracted by the feature point extraction unit 320 and the feature values ​​of the multiple feature points on the environment map in step T170.

[0203] The estimation unit 400 outputs the estimated vehicle position to the driver assistance system 500. The process returns to step T110 to repeat the operations from step T110 to step T180.

[0204] The aforementioned step T150 is performed by a correction factor decision processing routine as described in Fig. 16 illustrated and realized.

[0205] In step T200, the sunshine condition estimation unit 360 estimates the sunshine condition of the vehicle based on the brightness obtained from the sensor information.

[0206] The light source direction estimation unit 350 estimates the position of a light source and the direction of the vehicle at the time the image is taken, based on the sensor information in step T210.

[0207] In step T220, the light source direction estimation unit 350 estimates the light source direction relative to the imaging direction at the time the image is acquired, based on the estimated light source direction and the imaging direction in the aforementioned step T210.

[0208] The correction factor decision unit 370 selects the first feature point in step T230.

[0209] The correction factor decision unit 370 determines whether the sunshine condition estimated in step T200 influences the sunshine condition at the time the environment map is generated, or not, in step T240.

[0210] In a case where the estimated sunshine condition does not affect the sunshine condition at the time the environment map is generated (No in step T240), the correction factor decision unit 370 determines the feature value correction factor A as the feature value correction factor in step T250, and the process proceeds to step T330.

[0211] On the other hand, in a case where the estimated sunshine condition influences the sunshine condition at the time the environment map is generated (Yes in step T240), the correction factor decision unit 370 determines whether the direction of the sun relative to the imaging direction estimated in step T220 is on the right side or on the left side of the direction of the sun at the time the environment map is generated, in step T260.

[0212] In a case where the direction of the sun relative to the imaging direction is the right side relative to the direction of the sun when generating the environment map (No in step T260), the correction factor decision unit 370 subsequently determines whether the feature point is formed in a depression or not, in step T270.

[0213] In a case where the feature point is formed in a lead (No in step T270), the correction factor decision unit 370 determines the feature value correction factor B as the feature value correction factor in step T280, and the process continues to step T330.

[0214] In a case where the feature point is formed in a depression (Yes in step T270), the correction factor decision unit 370 determines the feature value correction factor B' as the feature value correction factor in step T290, and the process continues to step T330.

[0215] In a case where the direction of the sun relative to the imaging direction on the left side is relative to the direction of the sun when generating the environment map (Yes in step T260), the correction factor decision unit 370 subsequently determines whether the feature point is formed in a depression or not, in step T300.

[0216] In a case where the feature point is formed in a lead (No in step T300), the correction factor decision unit 370 determines the feature value correction factor C as the feature value correction factor in step T310, and the process continues to step T330.

[0217] In a case where the feature point is formed in a depression (Yes in step T300), the correction factor decision unit 370 determines the feature value correction factor C' as the feature value correction factor in step T320, and the process continues to step T330.

[0218] In step T330, the correction factor decision unit 370 determines whether the feature value correction factor has been determined for all of the feature points or not.

[0219] In a case where the characteristic value correction factor has not been determined for all of the characteristic points (No in step T330), the characteristic point for which the characteristic value correction factor has not been determined is then selected. The process then returns to step T240.

[0220] In a case where the feature value correction factor has been determined for all of the feature points (Yes in step T330), the process returns.

[0221] As noted above, according to the self-position estimation device of the third embodiment, the feature value correction factor serves to correct the feature value of each of the plurality of feature points extracted by the feature point extraction unit 320, so that the feature value is brought into a state in which the feature point is moved in the direction of the light source in which the environment map is generated, by obtaining the feature values ​​of the feature points, based on the feature value of the feature point extracted by the feature point extraction unit 320, the estimated direction of the light source relative to the estimated imaging direction, and the direction of the light source relative to the imaging direction, which is obtained previously at the time when the feature value of the environment map is obtained.The vehicle's position is then estimated based on the correction feature values ​​of the multiple feature points, which are corrected based on the feature values ​​and their correction values, and the feature values ​​of the multiple feature points on the environment map. Therefore, even if the estimated direction of the light source differs from the direction of the light source at the time the feature values ​​are referenced on the environment map, the vehicle's position can be estimated accurately.

[0222] In the third embodiment, the feature values ​​of the extracted feature points from the captured image are corrected. In a modified example of the third embodiment, the feature values ​​of the feature points on the environment map are corrected.

[0223] In the modified example of the third embodiment, the feature value correction factor is determined in the same way as in the third embodiment in a case where the estimated sunshine condition does not influence the sunshine condition at the time the environmental map is generated. The feature value correction factors in other cases are determined as follows.

[0224] In a case where the estimated sunshine condition influences the sunshine condition at the time the environment map is generated, the feature value correction factor for the feature value(s) of the feature point on the environment map is determined assuming that the estimated direction of the sun is on the right side or the left side relative to the direction of the sun at the time the environment map is generated, and whether the feature point is formed in a protrusion or a depression.

[0225] For example, the feature value correction factor is calculated assuming that the sun's direction relative to the imaging direction at the time the image is taken is on the right, compared to the sun's direction at the time the environment map is generated, and the feature point is formed in the protrusion (i.e., the feature value correction factor B in Fig. 14), for the feature value(s) of the feature point on the environment map.

[0226] In a case where both of the two 5 x 5 pixel areas (xa, xb) with respect to which the luminance calculation is performed for comparison to obtain the xth bit of the feature value are located on the left side or the right side of the feature point, the value of the xth sequence of the feature value correction factor is set to zero.

[0227] In a case where region xa is on the right and region xb is on the left, and luminance calculations are performed on regions xa and xb, the left side is darker (i.e., contains a lower luminance) if the sun's direction relative to the imaging direction at the time the image is taken is on the right, compared to the sun's direction at the time the environment map is generated. The feature point is the protrusion because the shadow is formed on the left side. That is, the luminance in region xa is greater than the luminance in region xb. Therefore, the x-th bit of the feature value affected by the sunlight state is one (1).

[0228] As long as the xth bit of the feature value of the feature point on the map is one, no correction of the feature value is necessary. Therefore, the value of the xth sequence of the feature value correction factor is set to zero. Conversely, if the xth bit of the feature value of the feature point on the map is zero, the xth bit of the feature value of the feature point on the environment map should be corrected to one, so that the value of the xth sequence of the feature value correction factor is set to plus one (+1).

[0229] In a case where region xa is on the left and region xb is on the right, the luminance in region xa is less than the luminance in region xb if the sun's direction relative to the imaging direction at the time the image is taken is on the right, compared to the sun's direction at the time the environment map is generated, and the feature point is the protrusion. Therefore, the xth bit of the feature value affected by the sunlight state is zero.

[0230] As long as the xth bit of the feature value of the feature point on the map is zero, no correction of the feature value is necessary. Therefore, the value of the xth sequence of the feature value correction factor is set to zero. Conversely, if the xth bit of the feature value of the feature point on the map is one, the xth bit of the feature value of the feature point on the surrounding map should be corrected to zero, so that the value of the xth sequence of the feature value correction factor is set to minus one (-1).

[0231] The above operation is performed on all bits in order to determine the feature value correction factor B.

[0232] Similarly, the feature value correction factor B' is determined for a case where the sun's direction relative to the imaging direction at the time the image is taken is to the right compared to the sun's direction at the time the environment map is generated, and the feature point is the depression. The feature value correction factor C is determined for a case where the sun's direction relative to the imaging direction at the time the image is taken is to the left compared to the sun's direction at the time the environment map is generated, and the feature point is the protrusion.The feature value correction factor C' is determined for a case where the direction of the sun relative to the imaging direction at the time the image is taken is the left side compared to the direction of the sun at the time the environment map is generated, and the feature point is the depression.

[0233] Accordingly, various patterns of feature value correction factors are determined so that, even if the estimated sunshine conditions influence the sunshine condition at the time the environment map is generated (for example, due to shadows), the feature value of the feature point on the environment map with respect to such influence is correctable based on the direction of sun rays in the two areas (points) being compared and the shape (protrusion, depression) of the feature point.

[0234] The construction of a self-position estimation system 60 according to the modified example of the third embodiment is explained below.

[0235] Fig. Figure 17 is a block diagram illustrating the self-position estimation device 300 of the self-position estimation system 60 according to the modified example of the third embodiment.

[0236] The feature value calculation unit 330 calculates the feature values ​​of the multiple feature points based on their luminance.

[0237] Specifically, the feature value calculation unit 330 calculates the feature values ​​of the multiple feature points obtained from the feature point extraction unit 320. For example, the feature value calculation unit 330 calculates the feature values ​​in each of 32 bytes based on ORB features.

[0238] The feature value calculation unit 330 sends the calculated feature values ​​to the estimation unit 400.

[0239] The feature value correction unit 390 corrects the feature values ​​of the multiple feature points based on the feature values ​​of the feature points on the environment map and the corresponding feature value correction factors for the feature points.

[0240] The feature value correction unit 390 sends the corrected feature values ​​on the environment map to the estimation unit 400.

[0241] The estimation unit 400 estimates the position of the vehicle based on the corrected feature values ​​of the multiple feature points on the environment map as well as the feature values ​​of the multiple feature points extracted by the feature point extraction unit 320.

[0242] In particular, the estimation unit 400 estimates the vehicle's position on the map by comparing the corrected feature values ​​of the multiple feature points on the environment map with the feature values ​​of the extracted multiple feature points. For example, the estimation unit 400 determines a position corresponding to, or a position exhibiting the feature value with the greatest similarity between the corrected feature values ​​on the environment map and the feature values ​​of the multiple feature points as an estimated vehicle position on the environment map.

[0243] The estimation unit 400 then outputs the estimated vehicle position to the driver assistance system 500.

[0244] An operation of the self-position estimation device according to the modified example of the third embodiment is explained below. Fig.Figure 18 is a flowchart illustrating a self-position estimation processing routine according to the modified example of the third embodiment. In step T460, the feature value correction unit 390 corrects the feature value of each of the multiple feature points in the environment map based on the feature value of the corresponding feature point on the environment map and the feature value correction factor for the corresponding feature point.

[0245] As noted above, according to the self-position estimation device of the modified example of the third embodiment, the feature value correction factor for correcting the feature value of each of the feature points on the environment map is determined such that the feature value is brought into a state in which the feature point of it, which is extracted by the feature point extraction unit 320, is brought into the light source direction in which the environment map is generated, by obtaining and storing the feature values ​​of the feature points, based on the feature value and the feature point extracted by the feature point extraction unit 320, the estimated light source direction relative to the estimated imaging direction, and the light source direction relative to the imaging direction, which is previously obtained at the time when the feature value of the environment map is obtained.The vehicle's position is then estimated based on the corrected feature values ​​of the feature points, which were corrected using the feature values ​​and their correction values, and the feature values ​​of the multiple feature points extracted by the feature point extraction unit. Therefore, even if the estimated direction of the light source differs from the direction of the light source at the time the feature values ​​are referenced on the environment map, the vehicle's position can be estimated accurately.

[0246] The embodiments are not limited to featuring the above designs and may be appropriately modified or changed.

[0247] In the preceding embodiments, the sunshine condition estimation unit 360 estimates whether a shadow is formed or not using the radiation meter. Alternatively, it can estimate whether a shadow is formed or not based on an input captured image. In this case, it can be estimated that the shadow is not formed if, for example, the difference between the average luminance at a shaded area and the average luminance at an area different from the shaded area is greater than or equal to a predetermined threshold within any range.

[0248] The preceding examples described the case in which the program is installed beforehand. Alternatively, such a program could, for example, be stored on a computer-readable storage medium for deployment.

[0249] A self-position estimation device (10, 300) comprises an image acquisition unit (14, 210) that acquires driving images and reference images at a plurality of positions along a predetermined driving route, a detection unit (20, 320) that detects feature points in each of the driving images and feature points in each of the reference images, a storage unit (16) that stores map information comprising the feature points in each of the reference images and a position and attitude of the image acquisition unit at a time when each of the reference images is acquired by the image acquisition unit, and an estimation unit (24, 400) that selects a similar image, like one of the driving images, from the reference images in order to correlate the feature points on one of the driving images and feature points on the similar image.wherein the estimation unit estimates a position and attitude of one's own vehicle (50) on a predetermined route based on a correlation result.

Claims

[1] Self-position estimation device (10, 300), with: an image acquisition unit (14, 210) that captures a multitude of driving images in a state in which a self-propelled vehicle (50) is driven along a predetermined driving route, and a multitude of reference images at a multitude of positions along the predetermined driving route; a capture unit (20, 320) that captures feature points in each of the plurality of driving images and feature points in each of the plurality of reference images, each of which is correlated with the reference images, wherein the feature points in each of the reference images captured by the capture unit (20) serve as environment feature points in an environment along the predetermined driving route; a storage unit (16) that stores map information comprising the feature points in each of the plurality of reference images as well as a position and attitude of the image acquisition unit (14, 210) at a time when each of the plurality of reference images is acquired by the image acquisition unit (14, 210); an estimation unit (24, 400) that selects a similar image to one of the multitude of driving images from the multitude of reference images in order to correlate the feature points in that one of the multitude of driving images and feature points in the similar image, wherein the estimation unit (24, 400) estimates a position and attitude of its own vehicle (50) on the predetermined driving route based on a correlation result; and a setting unit (12) that estimates a road surface gradient difference of the predetermined driving route, and sets a detection range (As) of correlation feature points in each of the plurality of driving images correlated with the surrounding feature points based on the estimated road surface gradient difference, wherein the setting unit (12) selects that similar image which is similar to one of the plurality of driving images from the plurality of reference images based on the correlation feature points detected in the detection range. [2] Self-position estimation device (10) according to claim 1, wherein The map information specifies feature values ​​that indicate the respective characteristics of the environmental feature points. the estimation unit (24) selects the similar image from the multitude of reference images, wherein the similar image includes the largest number of environmental feature points correlated with the correlation feature points in the driving image, wherein the estimation unit (24) correlates the environmental feature points and the correlation feature points by comparing the feature points of the correlation feature points and the feature values ​​of the environmental feature points. [3] Self-position estimation device (10) according to claim 1 or 2, wherein the image acquisition unit (14) is fixed at a predetermined position of the vehicle (50) in order to take an image of a predetermined area in front of the image acquisition unit (14), The setting unit (12) selects from the multitude of reference images the similar image that is most similar to the predetermined area in front of the image acquisition unit (14), and estimates the road surface gradient difference from a difference between a road surface gradient based on the position and attitude of the image acquisition unit (14) correlated with the selected similar image and a road surface gradient based on the position and attitude of the image acquisition unit (14) estimated in the driving image that is previously recorded. [4] Self-position estimation device (10) according to one of claims 1 to 3, wherein the adjustment unit (12) moves the detection area (As) in an up-and-down direction in the driving image based on the road surface gradient difference. [5] Self-position estimation device (10) according to claim 3 or 4, wherein the setting unit (12) specifies a position of the detection area (As) in the driving image on an upper side with an increase in the road surface gradient of a position of the own vehicle (50), as compared to a road surface gradient of the predetermined region. [6] Self-position estimation device (300) which estimates a position of its own vehicle using an environment map which stores a feature value of each of a plurality of feature points whose positions are known, wherein the self-position estimation device (300) comprises: a feature point extraction unit (320) that extracts a multitude of feature points from an image capturing the environments of the vehicle itself; a feature value calculation unit (330) that calculates a feature value from each of the plurality of feature points extracted by the feature point extraction unit (320), wherein the feature value is based on a luminance of each of the plurality of feature points; a light source direction estimation unit (350) that estimates a light source direction relative to an imaging direction at a time when the image is acquired, based on sensor information; a correction factor decision unit (370) for selecting a feature value correction factor for correcting the feature value of each of the plurality of feature points extracted by the feature point extraction unit (320), such that the feature value is brought into a state in which the feature point of it, which is extracted by the feature point extraction unit (320), is brought into a light source direction in which the environment map is generated by obtaining the feature value from each of the plurality of feature points, on a basis of the feature value from each of the plurality of feature points extracted by the feature point extraction unit (320), the estimated light source direction, and the light source direction which is previously obtained when the feature value is obtained from each of the plurality of feature points on the environment map; a feature value correction unit (390) that corrects the feature value of each of the plurality of feature points extracted by the feature point extraction unit (320) based on the feature value of each of the plurality of feature points extracted by the feature point extraction unit (320) and the feature value correction factor for each of the plurality of feature points; and an estimation unit (400) that estimates the position of the own vehicle based on the corrected feature value for each of the multitude of feature points extracted by the feature point extraction unit (320) and the feature value of each of the multitude of feature points on the environment map. [7] Self-position estimation device (300) which estimates a position of its own vehicle using an environment map which stores a feature value for each of a plurality of feature points whose positions are known, wherein the self-position estimation device (300) comprises: a feature point extraction unit (320) that extracts a multitude of feature points from an image capturing the environments of the vehicle itself; a feature value calculation unit (330) that calculates a feature value from each of the plurality of feature points extracted by the feature point extraction unit (320), wherein the feature value is based on a luminance of each of the plurality of feature points; a light source direction estimation unit (350) that estimates a light source direction relative to an imaging direction at a time when the image is acquired, based on sensor information; a correction factor decision unit (370) for selecting a feature value correction factor for correcting the feature value of each of the plurality of feature points on the environment map, such that the feature value is brought into a state in which the feature point of it, which is extracted by the feature point extraction unit (320), is brought into a light source direction in which the environment map is generated by obtaining the feature value from each of the plurality of feature points, on the basis of the feature value from each of the plurality of feature points extracted by the feature point extraction unit (320), the estimated light source direction, and the previously obtained light source direction when the feature value is obtained from each of the plurality of feature points on the environment map; a feature value correction unit (390) that corrects the feature value of each of the plurality of feature points on the environment map based on the feature value of each of the plurality of feature points on the environment map and the feature value correction factor for each of the plurality of feature points; and an estimation unit (400) that estimates the position of the own vehicle based on the corrected feature value for each of the multitude of feature points on the environment map and the feature value of each of the multitude of feature points extracted by the feature point extraction unit (320). [8] Self-position estimation device (300) according to claim 6 or 7, further comprising a sunshine state estimation unit (360) which estimates a sunshine state of the own vehicle based on a brightness obtained from the sensor information, wherein the correction factor decision unit (370) selects the feature value correction factor to correct the feature value of each of the plurality of feature points based on the feature value of each of the plurality of feature points, the estimated light source direction, and the previously obtained light source direction, when the feature value of the environment map is obtained in a case in which the sunshine state obtained by the sunshine state estimation unit (360) influences the feature value of each of the feature points due to a difference in the light source direction.

Images