Self-position estimation device

Abstract

To provide a self-localization device that can reliably estimate its own position while keeping manufacturing costs down. [Solution] The self-position estimation devices 100, 100a, and 100b for estimating the self-position of a moving object V1 include: a first distance estimation unit 11 that extracts feature points p1 to p11 and feature quantities of the feature points from each of a plurality of captured images F1 and F2 obtained in time series by an imaging device, and estimates a first distance, which is the distance to each feature point, using the feature points and feature quantities extracted from the plurality of captured images; a second distance estimation unit 12 that recognizes a specific object contained in each of the plurality of captured images, identifies changes in the size of the same specific object in the plurality of captured images, and estimates a second distance, which is the distance to the specific object, based on the changes in size; and a feature point selection unit 13 that selects feature points p3 to p9, which are feature points used for self-position estimation, from among the feature points extracted by the first distance estimation unit, based on the first distance and the second distance.

Description

Technical Field

[0004] , , ,

[0005] , , , ,

[0001] The present disclosure relates to a self-position estimation device.

Background Art

[0002] Patent Document 1 discloses a method for estimating the self-position of a vehicle. In this method, for each captured image obtained by a camera mounted on the vehicle, a distance is estimated using a machine learning model, and the reliability of the distance is calculated by analyzing the correspondence relationship with machine learning data, and a region composed of pixels with high reliability is specified. Then, the self-position is estimated using the feature points in the region with high reliability. The machine learning model for estimating the distance for each pixel calculates the distance by analyzing the correspondence relationship between the shape of the blur for each pixel and the blur of the machine learning data.

Prior Art Document

Patent Document

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the self-position estimation method of Patent Document 1, a dedicated machine learning model that is not used in conventional driving support or the like is required to estimate the distance to each pixel based on the shape of the blur of each pixel. Therefore, dedicated training is required to obtain such a machine learning model, leading to an increase in development costs. In addition, since the accuracy of self-position estimation depends on the performance (reliability) of the machine learning model, there is also a problem that stable self-position estimation cannot be achieved. For these reasons, there is a need for a self-position estimation device that uses captured images and can stably estimate the self-position while suppressing the manufacturing cost.

Means for Solving the Problems

[0005] As one embodiment of this disclosure, a self-position estimation device (100, 100a, 100b) is provided for estimating the self-position of a mobile body (V1) equipped with an imaging device (500). This self-position estimation device includes: a first distance estimation unit (11) that extracts feature points (p1~p11) and feature quantities of the feature points from each of a plurality of captured images (F1, F2) obtained in time series by the imaging device, and estimates a first distance, which is the distance to each feature point, using the feature points and feature quantities extracted from the plurality of captured images; a second distance estimation unit (12) that recognizes a specific object in each of the plurality of captured images which is a predetermined type of object included in the captured image and which contains the feature points, identifies changes in the size of the same specific object in the plurality of captured images, and estimates a second distance, which is the distance to the specific object, based on the changes in size; and a feature point selection unit (13) that selects the feature points to be used for estimating the self-position from among the feature points extracted by the first distance estimation unit, based on the first distance and the second distance.

[0006] This type of self-localization device selects the feature points to be used for self-localization based on a first distance estimated using feature points and features extracted from multiple captured images, and a second distance estimated based on the change in size of a specific target recognized from the captured images across multiple captured images. This allows for stable self-localization while keeping manufacturing costs down. Estimating the first distance using feature points and features extracted from multiple captured images can be achieved without performing special machine learning. Furthermore, for specific targets of a predetermined type, it is relatively easy to learn the relationship between changes in size and changes in distance to the specific target. [Brief explanation of the drawing]

[0007] [Figure 1] This block diagram shows a schematic configuration of a self-localization device as one embodiment of the present disclosure. [Figure 2] This is a flowchart showing the procedure for self-localization processing in the first embodiment. [Figure 3] This figure shows an image obtained by an imaging device and an example of a bounding box set for a specific target. [Figure 4] This figure shows an example of a method for selecting the characteristic points used in the first embodiment. [Figure 5] This is a flowchart showing the procedure for self-localization processing in the second embodiment. [Figure 6] This figure shows an example of a method for selecting the characteristic points used in the second embodiment. [Figure 7] This is a flowchart showing the procedure for selecting feature points to be used in the third embodiment. [Figure 8] This is a block diagram showing the schematic configuration of the self-position estimation device according to the fourth embodiment. [Figure 9] This is a flowchart showing the procedure for self-localization processing in the fourth embodiment. [Figure 10] This is a block diagram showing the schematic configuration of the self-position estimation device according to the fifth embodiment. [Figure 11] This figure shows an example of a method for selecting characteristic points for use in other embodiments. [Modes for carrying out the invention]

[0008] A. First Embodiment: A1. Equipment configuration: The self-position estimation device 100 shown in Figure 1 is mounted on the vehicle V1 as part of the control unit 200 and estimates the vehicle V1's own position. In this embodiment, the vehicle V1 may be configured as any type of vehicle, such as an engine vehicle, a hybrid vehicle (HEV), a plug-in hybrid vehicle (PHEV), an electric vehicle (EV), or a fuel cell vehicle (FCV, FCHV). The vehicle V1 is configured to be capable of both autonomous driving (advanced driver assistance systems alone may be used) and manual driving. Autonomous driving means driving in a state where the control unit 200 controls at least some of the basic operations of the vehicle V1, such as driving, turning, and stopping, as well as various processes such as turning on the lights and operating the wipers. On the other hand, manual driving means driving in a state where the occupants of the vehicle V1 control all of the above basic operations. As shown in Figure 1, in addition to the control unit 200, the vehicle V1 includes an operation execution unit 300, an operation control unit 400, an imaging device 500, an illuminance sensor 600, and a movement state sensor 700.

[0009] The control unit 200 controls the vehicle V1. The control unit 200 includes a self-position estimation device 100 and a driving control device 150. Details of the self-position estimation device 100 will be described later. The driving control device 150 controls various processes related to the driving of the vehicle V1. When automatic driving is performed, the driving control device 150 outputs a command to the operation control unit 400 (described later) to control the driving of the vehicle V1 so that it travels along a preset driving path, based on the self-position estimated by the self-position estimation device 100. When manual driving is performed, the driving control device 150 outputs a command to the operation control unit 400 (described later) to control the driving of the vehicle V1 in accordance with the driver's intention to drive, which is input from the steering wheel and accelerator pedal (not shown).

[0010] The operation execution unit 300 performs the actions of the vehicle V1, namely driving, steering, and braking, and operates auxiliary equipment including various lamps. The operation execution unit 300 comprises a drive unit 310, a brake mechanism 320, and a steering mechanism 330. The drive unit 310 includes, for example, an engine or a traction motor. The brake mechanism 320 consists of a group of devices (actuators) involved in brake control, such as sensors, motors, valves, and pumps. The steering mechanism 330 consists of a group of devices (actuators) involved in steering, such as a power steering motor.

[0011] The motion control unit 400 controls the motion execution unit 300 according to control commands input from the driving control device 150. The motion control unit 400 includes a drive ECU 410 that controls the drive unit 310, a brake ECU 420 that controls the brake mechanism 320, and a steering ECU 430 that controls the steering mechanism 330. For example, if the drive unit 310 is an engine, the drive ECU 410 controls the opening and closing of the throttle valve, the ignition of the igniter, the opening and closing of the intake valve, etc., by controlling various actuators (not shown). The brake ECU 420 controls the operation of actuators related to brake control. The steering ECU 430 controls the operation of actuators related to steering.

[0012] The operation control device 150, the operation execution unit 300, and the operation control unit 400 are all composed of an ECU (Electronic Control Unit) equipped with a CPU and memory (not shown).

[0013] The imaging device 500 captures images of the surroundings of the vehicle V1. The imaging device 500 consists of one or more cameras equipped with a number of imaging elements such as CCDs and CMOSs. As a result of imaging by the imaging device 500, a plurality of imaging images (frame images) are acquired in time series. For example, one frame image may be acquired every 33 msec (milliseconds). The illuminance sensor 600 detects the illuminance of the light in the external environment of the vehicle V1. The movement state sensor 700 detects the movement state (driving state) of the vehicle V1. The "movement state of the vehicle V1" refers to, for example, the rotational speed of the wheels of the vehicle V1, the speed, acceleration, yaw, roll, pitch, etc. of the vehicle V1. The detection results by the imaging device 500, the illuminance sensor 600, and the movement state sensor 700 are acquired by the control unit 200 which is electrically connected.

[0014] The self-position estimation device 100 estimates the self-position of the vehicle V1. In the present embodiment, the self-position estimation device 100 includes a CPU 10, a ROM 20, and a RAM 30. The CPU 10, the ROM 20, and the RAM 30 are configured to be communicable with each other via an internal bus 90. The CPU 10 functions as a first distance estimation unit 11, a second distance estimation unit 12, a feature point selection unit 13, and a self-position estimation unit 14 by expanding and executing the control program stored in the ROM 20 in the RAM 30.

[0015] The first distance estimation unit 11 estimates the distance to a feature point (hereinafter referred to as the "first distance") using the feature points and feature amounts extracted from the imaging image obtained by the imaging device 500. When a plurality of feature points are extracted from the imaging image, the first distance is estimated using the feature amounts for each feature point. Details of the method for estimating the first distance will be described later.

[0016] The second distance estimation unit 12 recognizes a predetermined type of target included in the captured image, which is a target including feature points extracted from the captured image (hereinafter referred to as "specific target"), and estimates the distance to the specific target (hereinafter referred to as "second distance") based on the change in the size of the specific target. In the present embodiment, the specific target corresponds to a road sign arranged above the road. Such road signs include, for example, rectangular signs in which the direction and aspect (the place name existing at the end of the direction) are indicated by an arrow, a place name, etc. Further, for example, signs installed on a highway, such as rectangular signs indicating a service area, correspond to this. Although the sizes of these signs vary, the vertical and horizontal lengths of each sign are both predetermined. Thus, since the specific target is a road sign with a predetermined size, the distance to the specific target (or the change in distance) can be estimated from the change in its size. In the present embodiment, the distance to the specific target (the change in distance from all frames) is estimated from the change in the size of the specific target (road sign) detected from two captured images. As a machine learning model for detecting a target, for example, a convolutional neural network (CNN) learned by supervised learning using a learning dataset can be used. The learning dataset may include, for example, a plurality of training images including specific targets, and labels indicating how much the distance to the specific target is for the change in the size of each specific target in the plurality of training images.

[0017] The feature point selection unit 13 selects feature points (hereinafter referred to as "used feature points") used for estimating the self-position based on the first distance and the second distance. Details of the selection method will be described later.

[0018] The self-position estimation unit 14 obtains the distance to the selected used feature point, and estimates the self-position based on the difference between the distance obtained this time and the distance obtained in the past for the same feature point. For example, based on the previously specified self-position, the current self-position is estimated considering the above-described difference in distance. Further, the ROM 20 may store high-precision 3D map data.

[0019] A2. Self-localization process: The self-position estimation process shown in Figure 2 is a process for estimating the position of vehicle V1. This self-position estimation process is executed repeatedly at regular intervals after the power of the self-position estimation device 100 is turned on.

[0020] In step S105, the first distance estimation unit 11 and the second distance estimation unit 12 each acquire the captured images obtained by the imaging device 500. Hereafter, "step S" will be simply referred to as "S". In S105, all captured images obtained by the imaging device 500 may be acquired, or only the most recent captured image at the time S105 is executed may be acquired.

[0021] In S110, the first distance estimation unit 11 extracts feature points and their feature quantities from the image acquired in S105. The first distance estimation unit 11 extracts feature points using a feature point extraction algorithm such as FAST (Features from Accelerated Segment Test). The first distance estimation unit 11 also calculates feature quantities using a feature quantity calculation algorithm such as SIFT (Scale Invariant Feature Transform), ORB (Oriented FAST and Rotated BRIEF), and SURF (Speeded Up Robust Features). The feature quantity calculation unit 211 is not limited to these feature quantity calculation algorithms; for example, it may perform feature extraction using AKAZE (Accelerated KAZE), HOG (Histograms of Oriented Gradients), or CNN (Convolutional Neural Network).

[0022] In S115, the first distance estimation unit 11 performs feature matching on feature points obtained as a result of performing S110 on past captured images (past frames) and feature points obtained by performing S110 on the current captured image (current frame). That is, for each feature point obtained in the current S110, it searches for feature points in past frames that have a feature quantity corresponding to the feature quantity obtained for that feature point. S115 identifies the same feature point in past frames for each feature point extracted in the current S110.

[0023] In S120, the first distance estimation unit 11 uses the position of the same feature point identified by feature matching in S115 in past frames and its position in the current frame to calculate the distance to the feature point (target) (hereinafter referred to as the "first distance") by triangulation.

[0024] In this embodiment, steps S130 to S140, described below, are executed in parallel with steps S110 to S120 described above. Note that steps S110 to S120 and steps S130 to S140 may be executed in any order.

[0025] In S130, the second distance estimation unit 12 detects a specific target in the image acquired in S105. Such a specific target can be detected, for example, by so-called semantic segmentation using a machine learning model. Therefore, for example, a specific target can be detected using an object detection function for autonomous driving.

[0026] In S135, the second distance estimation unit 12 compares the size of the specific target in the current frame with the size of the specific target in past frames.

[0027] The upper part of Figure 3 shows an example of a past frame F1 obtained by the imaging device 500 installed in the vehicle V1, and the lower part of Figure 3 shows an example of a present frame F2 obtained by the imaging device 500. In this disclosure, the past frame is also called the "first image," and the present frame is also called the "second image." As shown in the past frame F1 and present frame F2 of Figure 3, there is a preceding vehicle V2 in lane R1 on which the vehicle V1 is traveling. The distance between the preceding vehicle V2 and vehicle V1 increases with the passage of time. There is an oncoming vehicle V3 in the oncoming lane R2. The distance to the oncoming vehicle V3 decreases with the passage of time. Therefore, the size of the oncoming vehicle V3 in the image increases with the passage of time. A road sign RS1 is installed above lane R1 and oncoming lane R2 as a specific landmark. Since the road sign RS1 is a stationary object, as vehicle V1 moves forward, in other words, as time passes, the size of the road sign RS1 increases. When the second distance estimation unit 12 detects a specific target, it sets a bounding box surrounding the detected target. In the example in Figure 3, the road sign RS1 in past frame F1 has a bounding box BB1 set. The road sign RS1 in current frame F2 has a bounding box BB2 set. The method for setting the bounding boxes BB1 and BB2 is the same as the method used in known object detection methods, so its explanation is omitted. The size of the same road sign RS1 differs between past frame F1 and current frame F2. Therefore, the sizes of the two bounding boxes BB1 and BB2 also differ. Specifically, the size of bounding box BB2 is larger than the size of bounding box BB1. In this embodiment, the second distance estimation unit 12 identifies the change in the size of the bounding box set for the specific target as a change in the size of the specific target.

[0028] Here, a specific landmark, i.e., a rectangular road sign, has a predetermined number of size variations, as shown in the following example. Height (mm) x Width (mm) 2100 × 3000 2300 × 2800 2300 × 3000 2400 × 3000 2800 × 3000 3000 × 3200 Because the size of road signs themselves varies, the distances calculated (estimated) from these size variations will also vary.

[0029] As shown in Figure 2, in S140, the second distance estimation unit 12 calculates the distance to the specific target based on the change in the size of the specific target obtained by the size comparison in S135. Hereafter, the distance to the specific target obtained by the execution of S140 by the second distance estimation unit 12 will be referred to as the "second distance". As described above, the second distance estimation unit 12 calculates the second distance using the detection results of the machine learning model.

[0030] In S145, the feature point selection unit 13 identifies a judgment distance range based on the second distance calculated in S140. The "judgment distance range" is used when selecting the feature points to use from among the feature points extracted in S110. This will be explained in detail using Figure 4.

[0031] In Figure 4, the horizontal axis shows the first distance calculated in S120 and the second distance calculated in S140. In Figure 4, the vertical axis shows the distance to the feature point and the specific target.

[0032] As described above, the distance to a specific target has variations, so Figure 4 shows a distance range Rd defined by a minimum value L1 and a maximum value L2 as a second distance. In this embodiment, the determination distance range Rt is determined based on the second distance, i.e., the distance range Rd. Specifically, the determination distance range Rt is obtained as a correction range obtained by multiplying the boundary values ​​of the distance range Rd by predetermined coefficients. More specifically, the determination distance range Rt is determined as a distance range where the minimum value Ll is the value obtained by multiplying the minimum value L1 of the distance range Rd by coefficient α, and the maximum value Lh is the value obtained by multiplying the maximum value L2 of the distance range Rd by coefficient β. In this embodiment, the determination distance range Rt has a wide range both on the shorter and longer sides compared to the distance range Rd. Therefore, for example, coefficient α may be 0.7. Also, for example, coefficient β may be 1.1. Note that coefficients α and β are not limited to these values.

[0033] As shown in Figure 2, in S150, the feature point selection unit 13 selects feature points whose first distance calculated in S120 falls within the determination distance range Rt as feature points to be used. In the example in Figure 5, in S120, the first distances of a total of 11 feature points p1 to p11 are calculated. Of these 11 feature points p1 to p11, seven feature points p3 to p9 fall within the determination distance range Rt. On the other hand, the first distance of two feature points p1 and p2 is greater than the maximum value Lh of the determination distance range Rt. Also, the first distance of two feature points p10 and p12 is less than the minimum value Ll of the determination distance range Rt. Therefore, in this case, feature points p3 to p9 are selected as feature points to be used. Note that in Figure 2, feature points to be used are shown as small white circles, and feature points that are not to be used are shown as small black circles.

[0034] The feature points extracted by the first distance estimation unit 11 include a mix of points with high and low reliability as feature points. Feature points in which the first distance falls within the determination distance range Rt, which is set based on the range of the second distance estimated by the second distance estimation unit 12, can be considered to have high reliability. On the other hand, feature points in which the first distance measured falls outside the determination distance range Rt can be considered to have low reliability as feature points. Therefore, in this embodiment, feature points in which the calculated first distance falls within the determination distance range Rt are selected as feature points to be used.

[0035] As shown in Figure 2, in S155, the feature point selection unit 13 estimates its own position using the feature points used, saves its history (including the distance estimation result to the target finally determined in the current frame), and updates the position information of the vehicle V1.

[0036] According to the self-localization device 100 of the first embodiment described above, the feature points to be used are selected based on a first distance estimated using feature points and feature quantities extracted from two frames F1 and F2, and a second distance estimated based on the change in size of the recognized specific target across the two frames F1 and F2. This allows for stable self-localization while keeping development costs down. Estimating the first distance using feature points and feature quantities extracted from two frames F1 and F2 can be achieved without performing special machine learning. Furthermore, for specific targets of a predetermined type, it is relatively easy to estimate the distance to the specific target from the change in size of the detected target within the image.

[0037] Furthermore, the first distance estimation unit 11 uses feature points that correspond to each other, derived from feature quantities extracted from past frame F1 (first captured image) and feature quantities extracted from current frame F2 (second captured image), to estimate the first distance. This allows for more accurate estimation of the first distance compared to a configuration that estimates the first distance using feature points of feature quantities that do not correspond to each other.

[0038] Furthermore, the second distance estimation unit 12 estimates the second distance using the detection results of a recognition model, which is a machine learning model trained to detect specific targets from captured images, thus enabling accurate estimation of the second distance. In addition, since a machine learning model trained to detect specific stationary targets from captured images can be easily trained, the manufacturing cost of the self-localization device can be reduced.

[0039] Furthermore, the second distance estimation unit 12 identifies the change in the size of the bounding box set for each specific target in the two frames F1 and F2 as a change in the size of the specific target, thus enabling accurate and low-cost identification of changes in the size of the specific target. Since the function of setting a bounding box for a specific target and identifying its size is a common function for driver assistance and the like, the self-position estimation device 100 of the first embodiment can reduce manufacturing costs.

[0040] Furthermore, the feature point selection unit 13 selects, from among the feature points extracted by the first distance estimation unit 11, feature points whose estimated first distance falls within the determination distance range Rt determined based on the second distance, as the feature points to be used. This allows for the selection of feature points with high reliability as the feature points to be used.

[0041] Furthermore, the feature point selection unit 13 identifies the correction range obtained by multiplying the boundary values ​​(minimum value L1 and maximum value L2) of the distance range Rd of the second distance by predetermined coefficients α and β as the judgment distance range Rt. By adjusting the coefficients α and β considering the usage environment and estimation trends, feature points with higher reliability can be selected as the feature points to be used.

[0042] B. Second Embodiment: The configuration of the self-position estimation device 100 and vehicle V1 in the second embodiment is the same as the configuration of the self-position estimation device 100 and vehicle V1 in the first embodiment shown in Figure 1. Therefore, the same reference numerals are used for the same components, and their detailed descriptions are omitted.

[0043] The self-position estimation process of the second embodiment shown in Figure 5 differs from the self-position estimation process of the first embodiment shown in Figure 2 in that step S142 is additionally executed and S145a and S145b are executed instead of S145. The other steps of the self-position estimation process of the second embodiment are the same as those of the self-position estimation process of the first embodiment, so the same reference numerals are used for the same steps and their detailed explanation is omitted.

[0044] As shown in Figure 5, after the completion of S120 or S140, the feature point selection unit 13 determines whether or not the vehicle V1 is traveling inside the tunnel (S142). In this embodiment, the feature point selection unit 13 can make this determination based on the illuminance detected by the illuminance sensor 600 and the high-precision 3D map data stored in the ROM 20.

[0045] If it is determined that the vehicle is not traveling inside a tunnel (S142: NO), the feature point selection unit 13 identifies the normal determination distance range based on the second distance calculated in S140 (S145a). The "normal determination distance range" is the same as the determination distance range in the first embodiment. In other words, S145a in the second embodiment is the same as S145 in the first embodiment.

[0046] In contrast, if it is determined that the vehicle is traveling inside a tunnel (S142: YES), the feature point selection unit 13 identifies a tunnel-specific determination distance range based on the second distance calculated in S140 (S145b). The "tunnel-specific determination distance range" refers to the determination distance range used when selecting feature points to use while the vehicle V1 is traveling inside a tunnel.

[0047] The horizontal and vertical axes in Figure 6 are the same as those in Figure 4. The distance range Rd in Figure 6 is the same as the distance range Rd shown in Figure 4 of the first embodiment. In the second embodiment, the "tunnel determination distance range" is set to the same range as the distance range Rd. That is, the minimum value Ll of the tunnel determination distance range Rtt is equal to the minimum value L1 of the distance range Rd, and the maximum value Lh is equal to the maximum value L2 of the distance range Rd. By setting the tunnel determination distance range Rtt in this way, the determination distance range becomes narrower than the normal determination distance range (determination distance range Rt). Therefore, the feature point p9, which was selected as a feature point for use in the first embodiment, is not selected as a feature point for use when driving inside a tunnel in the second embodiment.

[0048] Thus, the reason for setting the tunnel detection distance range Rtt to be narrower than the normal detection distance range Rt is that, inside a tunnel, monotonous walls continue, and the environment is somewhat dark, making it difficult to identify feature points. This increases the likelihood of extracting feature points with low reliability, or even the possibility that feature points cannot be obtained at all inside a tunnel. In other words, by narrowing the detection distance range set based on the second distance, it becomes less likely that feature points with low reliability will be selected as usable feature points, thereby suppressing a decrease in the accuracy of self-localization estimation. It can also be said that the tunnel detection distance range Rtt is derived by setting both coefficients α and β in the first embodiment to "1".

[0049] The self-position estimation device 100 of the second embodiment described above has the same effects as the self-position estimation device 100 of the first embodiment. In addition, when the movement conditions, including the condition that the vehicle V1 is traveling inside a tunnel, are met, the coefficient used to derive the determination distance range is set to "1", so that only feature points with higher reliability can be selected as feature points to be used. For this reason, even when the vehicle V1 is traveling inside a tunnel and it is difficult to accurately identify feature points, it is possible to suppress an excessive decrease in the accuracy of self-position estimation.

[0050] C. Third Embodiment: The configuration of the self-position estimation device 100 and vehicle V1 in the third embodiment is the same as the configuration of the self-position estimation device 100 and vehicle V1 in the first embodiment shown in Figure 1. Therefore, the same reference numerals are used for the same components, and their detailed descriptions are omitted.

[0051] The self-position estimation process of the third embodiment shown in Figure 7 differs from the self-position estimation process of the first embodiment shown in Figure 2 only in that it additionally executes step S160. The other steps of the self-position estimation process of the third embodiment are the same as those of the self-position estimation process of the first embodiment, so the same reference numerals are used for the same steps, and their detailed explanation is omitted.

[0052] After the completion of S155, the first distance estimation unit 11 corrects the parameters used in estimating the first distance (S160) so that feature points used in estimating the first distance that are not included in the correction range obtained by multiplying the boundary value of the distance range Rd by a predetermined coefficient (hereinafter referred to as "feature points outside the correction range") are included in the correction range (determined distance range Rt). The parameters to be corrected in S160 include, for example, the threshold used for feature extraction, the threshold used when making a determination in feature matching between frames, and the evaluation function and evaluation method during bundle adjustment. After the completion of S160, the process ends. Therefore, in this case, when the self-localization process is executed next time, the parameters used in estimating the first distance (S120) will be the corrected parameters. Thus, the probability that the first distance will be included in the determined distance range Rt can be increased. In other words, the first distance can be estimated with high accuracy.

[0053] The self-position estimation device 100 of the second embodiment described above has the same effects as the self-position estimation device 100 of the first embodiment. In addition, the first distance estimation unit 11 corrects the parameters used in estimating the first distance so that feature points outside the correction range are included within the correction range, thereby suppressing a decrease in the accuracy of the first distance estimation.

[0054] D. Fourth Embodiment: The self-position estimation device 100a of the fourth embodiment shown in Figure 8 differs from the self-position estimation device 100 of the first embodiment only in that the CPU 10 also functions as a notification unit 15. The other configurations of the self-position estimation device 100a of the fourth embodiment and the configuration of the vehicle V1 are the same as those of the self-position estimation device 100 and vehicle V1 of the first embodiment shown in Figure 1, so the same reference numerals are used for the same components, and their detailed descriptions are omitted.

[0055] The notification unit 15 notifies the driving control device 150 of predetermined information. In this embodiment, the "predetermined information" refers to information indicating that the accuracy of the self-position estimation device 100a has decreased.

[0056] The self-position estimation process of the fourth embodiment shown in Figure 9 differs from the self-position estimation process of the first embodiment shown in Figure 2 only in that steps S170 and S175 are additionally executed. The other steps of the self-position estimation process of the fourth embodiment are the same as those of the self-position estimation process of the first embodiment, so the same reference numerals are used for the same steps, and their detailed explanation is omitted.

[0057] After the completion of S155, the feature point selection unit 13 determines whether the number of feature points outside the correction range among the feature points used to estimate the first distance is equal to or greater than a threshold number, and whether this occurs over a predetermined number of consecutive frames. The "feature points outside the correction range" are the same as the feature points outside the correction range described in the third embodiment. In this embodiment, the "threshold number" is 10. However, it is not limited to 10, and may be any number of points that can identify a decrease in feature point extraction accuracy. Also, the "determined number of consecutive frames" in this embodiment is "10 frames". However, it is not limited to 10, and may be any number of frames that can identify a state where the decrease in feature point extraction accuracy is not a transient state.

[0058] If it is determined that the number of feature points outside the correction range is greater than or equal to a threshold number for a predetermined number of consecutive frames (S170: YES), the operation control device 150 notifies the operation control device 150 of predetermined information, namely information indicating that the estimation accuracy of the self-position estimated by the self-position estimation device 100a has decreased (S175). Conversely, if it is determined that the number of feature points outside the correction range is not greater than or equal to a threshold number for a predetermined number of consecutive frames (S170: NO), the process ends.

[0059] When the driving control device 150 receives notification that the accuracy of the self-position estimation device 100a has decreased, it may perform various processes as a result of receiving such notification. For example, if the vehicle V1 is in automatic driving mode, it is desirable to switch back to manual driving if the accuracy of the self-position estimation decreases. In this case, the driving control device 150 may use a display device or the like installed in the vehicle V1 to notify the driver to take the steering wheel and switch to manual driving.

[0060] The self-position estimation device 100a of the fourth embodiment described above has the same effect as the self-position estimation device 100 of the first embodiment. In addition, the notification unit 15 notifies the driving control device 150 of predetermined information when the number of feature points outside the judgment range is equal to or greater than the threshold number of points for a predetermined number of consecutive frames. As a result, the driving control device 150 can understand that the reliability of the self-position estimation result by the self-position estimation device 100a it uses is low.

[0061] E. Fifth Embodiment: The self-position estimation device 100b of the fifth embodiment shown in Figure 10 differs from the self-position estimation device 100 of the first embodiment only in that the CPU 10 also functions as an evaluation unit 16. The control unit 200a of the fifth embodiment differs from the control unit 200 of the first embodiment only in that it includes a second self-position estimation device 101. The other configurations of the self-position estimation device 100b of the fourth embodiment and the configuration of the control unit 200a are the same as the configurations of the self-position estimation device 100 and control unit 200 of the first embodiment shown in Figure 1, so the same reference numerals are used for the same configurations and their detailed descriptions are omitted. The other configurations of the vehicle V1 of the fourth embodiment are the same as the vehicle V1 of the first embodiment, so the same reference numerals are used for the same configurations and their detailed descriptions are omitted.

[0062] The second self-position estimation device 101 estimates the self-position of vehicle V1, similar to the self-position estimation device 100b. The second self-position estimation device 101 estimates the self-position of vehicle V1 based on the movement state (driving state) of vehicle V1 detected by the movement state sensor 700. For example, it estimates the self-position by an estimation method performed as so-called dead reckoning, which involves determining the distance traveled by vehicle V1 from the rotation speed of vehicle V1's wheels and identifying the turning of vehicle V1 from changes in yaw rate.

[0063] The evaluation unit 16 uses a first distance derived from the characteristic points used to evaluate the usefulness of the detection information from the movement state sensor 700 (information indicating the driving state of the vehicle V1) for self-position estimation by the second self-position estimation device 101. Specifically, it compares the self-position estimated using the first distance derived from the characteristic points used (i.e., the self-position estimated by the self-position estimation unit 14) with the self-position estimated by the second self-position estimation device 101. The evaluation unit 16 then evaluates the detection information from the movement state sensor 700 as highly useful if the distance between these estimated self-positions is within a predetermined distance range. On the other hand, if the distance between the estimated self-positions is outside the predetermined distance range, the evaluation unit 16 evaluates the detection information from the movement state sensor 700 as less useful. As described above, the self-position estimated by the self-position estimation unit 14 is highly accurate. Therefore, this self-position is used to accurately evaluate the usefulness of the detection information used for self-position estimation by the second self-position estimation device 101, which is another self-position estimation device.

[0064] The self-position estimation device 100b of the fifth embodiment described above has the same effects as the self-position estimation device 100 of the first embodiment. In addition, the evaluation unit 16 uses the first distance of the feature points used to evaluate the usefulness of the detection information of the movement state sensor 700 for the estimation of the self-position by the second self-position estimation device 101, so the usefulness of the detection information can be evaluated with high accuracy. This is because the feature points used are highly reliable feature points, and therefore the accuracy of the self-position determined based on the first distance obtained using such feature points is high.

[0065] F. Other embodiments: (F1) In each embodiment, the distance range Rd was determined as a range having a minimum value L1 and a maximum value L2, but the disclosure is not limited thereto. For example, in a configuration in which self-localization is performed only in a specific section of a particular highway, and in such a specific section there is only one type of road sign, as shown in Figure 11, only distance L3 can be estimated as a second distance. In such a configuration, the determination distance range Rt may be defined as a range in which the minimum value Ll is obtained by multiplying distance L3 by a predetermined coefficient α (e.g., 0.7), and the maximum value Lh is obtained by multiplying distance L3 by a predetermined coefficient β (e.g., 1.1).

[0066] (F2) In each embodiment, the second distance estimation unit 12 used a pre-trained machine learning model that was trained to estimate the distance to a specific object (road sign) from the change in size of the specific object captured in two captured images, but the disclosure is not limited thereto. The distance to a specific object may be estimated from the size of the specific object captured in a single captured image. In such estimation, for example, the relationship between the size of a specific object and the distance may be identified in advance through experiments or simulations and compiled into a table, and the distance to the specific object may be estimated by referring to the table based on the size of the specific object identified in each frame.

[0067] (F3) In each embodiment, the size of the bounding box set for a specific target was identified as the size of the specific target, but the disclosure is not limited thereto. For example, the second distance estimation unit 12 may perform semantic segmentation and, in the resulting image, identify the number of pixels occupied by the specific target as the number of occupied pixels. In this case, the second distance estimation unit 12 may determine the second distance from the change in the number of occupied pixels of the specific target.

[0068] (F4) In the second embodiment, the tunnel determination distance range was used when the vehicle V1 was traveling inside a tunnel, but the disclosure is not limited thereto. The tunnel determination distance range may also be used when the vehicle V1 is traveling at night. This is because when the vehicle V1 is traveling at night, there is a high possibility that low-confidence feature points will be extracted, similar to when it is traveling inside a tunnel. Needless to say, it is possible to apply this not only to tunnels and nighttime, but also to other driving scenes (e.g., daytime on a clear day) by adding a process to determine whether a scene requires special attention.

[0069] (F5) In the fourth embodiment, the recipient of notification of information indicating that the accuracy of the self-position estimation by the self-position estimation device 100a has decreased was the driving control device 150, but the disclosure is not limited thereto. A predetermined device that utilizes the self-position estimation result by the self-position estimation device 100a may also be the recipient of such information. For example, information indicating that the accuracy of the self-position estimation by the self-position estimation device 100a has decreased may be notified to a server device that exchanges information with the vehicle V1 via wireless communication.

[0070] (F6) Each embodiment is merely an example and can be modified in various ways. For example, the self-position estimation devices 100, 100a, and 100b may be mounted on any type of moving object, not just vehicles, but also ships, airplanes, etc. Also, in each embodiment, the self-position estimation devices 100, 100a, and 100b were configured with ECUs, but instead of ECUs, they may be configured with SoCs (System on Chip). In each embodiment, the coefficients α and β were set independently, but they may be set to the same value. Also, the installation location of road signs and the size of those road signs may be created as a dictionary for each road and stored in the ROM 20 in advance, and the second distance may be calculated using these sizes. Also, in the second embodiment, the coefficient α may be set to a value of 1 or more, and the coefficient β may be set to a value of less than 1.

[0071] (F7) The self-localization devices 100, 100a, 100b and their methods described herein may be implemented by a dedicated computer provided by configuring a processor and memory programmed to perform one or more functions embodied by a computer program. Alternatively, the self-localization devices 100, 100a, 100b and their methods described herein may be implemented by a dedicated computer provided by configuring a processor by one or more dedicated hardware logic circuits. Alternatively, the self-localization devices 100, 100a, 100b and their methods described herein may be implemented by one or more dedicated computers configured by a combination of a processor and memory programmed to perform one or more functions and a processor configured by one or more hardware logic circuits. The computer program may also be stored as instructions executed by the computer on a computer-readable non-transitional tangible recording medium.

[0072] This disclosure can be implemented in various forms. For example, it can be implemented in the form of a self-localization method, a method for selecting feature points to use, a computer program for implementing these methods and a self-localization device, or a non-temporary recording medium on which such a computer program is stored.

[0073] This disclosure is not limited to the embodiments described above, and can be implemented in various configurations without departing from its spirit. For example, the technical features in each embodiment corresponding to the technical features in the embodiments described in the summary of the invention can be replaced or combined as appropriate in order to solve some or all of the above-mentioned problems, or to achieve some or all of the above-mentioned effects. Furthermore, if a technical feature is not described as essential in this specification, it can be deleted as appropriate. [Explanation of Symbols]

[0074] 11...First distance estimation unit, 12...Second distance estimation unit, 13...Feature point selection unit, 100, 100a, 100b...Self-position estimation device, 500...Imaging device, F1, F2...Imagine captured image, p1~p11...Feature points, p3~p9...Feature points used, V1...Vehicle

Claims

1. A self-position estimation device (100, 100a, 100b) for estimating the self-position of a mobile body (V1) equipped with an imaging device (500), A first distance estimation unit (11) extracts feature points (p1 to p11) and feature quantities of the feature points from each of a plurality of captured images (F1, F2) obtained in a time series by the imaging device, and estimates a first distance, which is the distance to each feature point, using the feature points and feature quantities extracted from the plurality of captured images. A second distance estimation unit (12) recognizes a specific target, which is a predetermined type of target included in each of the plurality of captured images and contains the characteristic point, identifies changes in the size of the same specific target in the plurality of captured images, and estimates a second distance, which is the distance to the specific target, based on the changes in size. A feature point selection unit (13) selects, from among the feature points extracted by the first distance estimation unit, the feature points to be used for estimating the self-position, namely the feature points to be used (p3 to p9), based on the first distance and the second distance. A self-position estimation device equipped with the following features.

2. A self-position estimation device according to claim 1, The plurality of captured images include a first captured image and a second captured image obtained later in time than the first captured image. The first distance estimation unit is a self-position estimation device that uses the feature points, which correspond to the feature quantities extracted from the first captured image and the feature quantities extracted from the second captured image, for estimating the first distance.

3. In the self-position estimation device according to claim 1, The second distance estimation unit is a self-position estimation device that estimates the second distance using the detection results of a recognition model, which is a machine learning model trained to detect stationary targets from the captured image.

4. In the self-position estimation device according to claim 3, The second distance estimation unit is a self-position estimation device that identifies changes in the size of bounding boxes (BB1, BB2) set for each stationary target in the plurality of captured images as changes in the size of the specific target.

5. In the self-position estimation device according to claim 3, The second distance estimation unit performs semantic region division of the captured image and identifies the change in the number of pixels occupied by a stationary target obtained by the semantic region division as a change in the size of the specific target, in this self-position estimation device.

6. In a self-position estimation device according to any one of claims 1 to 5, The feature point selection unit selects, from among the feature points extracted by the first distance estimation unit, feature points whose estimated first distance falls within a determination distance range determined based on the second distance, as the feature points to be used, in the self-position estimation device.

7. In the self-position estimation device according to claim 6, The feature point selection unit identifies a correction range obtained by multiplying the boundary values ​​of the distance range of the second distance by predetermined coefficients (α, β) as the determination distance range, and is a self-position estimation device.

8. In the self-position estimation device according to claim 7, The feature point selection unit sets the coefficient to a value of 1 or less when the predetermined movement conditions of the moving body are met, and sets the coefficient to a value greater than 1 when the movement conditions are not met. A self-position estimation device, wherein the aforementioned movement conditions include the condition that the moving object is moving at night or moving inside a tunnel.

9. In a self-position estimation device according to any one of claims 1 to 5, The first distance estimation unit is a self-position estimation device that corrects the parameters used in estimating the first distance so that feature points outside the correction range, which are feature points that are not included in the correction range obtained by multiplying the boundary value of the determination distance range determined by the second distance by a predetermined coefficient, among the feature points used in estimating the first distance, which are within the correction range, are included within the correction range.

10. A self-position estimation device according to claim 6, A self-position estimation device further comprising a notification unit (15) that notifies a predetermined device that uses the self-position estimation result of a predetermined number of features, when the number of feature points outside the determination range, which are feature points whose estimated first distance is not included in the determination distance range, among the feature points extracted in each of the aforementioned captured images, is equal to or greater than a predetermined number of threshold points across a plurality of consecutive aforementioned captured images.

11. A self-position estimation device according to any one of claims 1 to 5, The mobile body has a second self-position estimation device (101) that estimates the self-position of the mobile body using detection information obtained from a sensor (700) mounted on the mobile body that detects the movement state of the mobile body, The self-position estimation device further comprises an evaluation unit (16) that uses the first distance of the feature points used to evaluate the usefulness of the detection information for the estimation of the self-position by the second self-position estimation device.

Images