Self-location estimation device and vehicle system

Abstract

To provide a self-position estimation device and a vehicle system which suppress increase in a calculation amount for self-position estimation and suppress reduction in accuracy of the self-position estimation.SOLUTION: In a vehicle including an information acquisition device, a vehicle system and a vehicle operation device, a self-position estimation device 210 of the vehicle system is mounted on a vehicle M1 having a camera 110. The self-position estimation device 210 for estimating the position of the vehicle comprises: a feature amount calculation unit 211 which extracts a feature point from an image captured in the travel direction of the vehicle by the camera and calculates a feature amount for each feature point; a feature point selection unit 212 which preferentially selects the feature point with high reliability as a preferential feature point in the feature points; a self-position calculation unit 213 which calculates the self-position by using the preferential feature point; and an error correction unit 214 which calculates a re-projection error of the self-position estimated for each of the plurality of images captured with a preset time interval to calculate the corrected self-position obtained by correcting the re-projection error.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a self-position estimation device and a vehicle system. [Background technology]

[0002] Known self-location estimation devices are those that estimate the location by comparing the features around the vehicle recognized by a camera mounted on the vehicle with map information. The self-location estimation device described in Patent Document 1 creates a database in advance of reference images and location information acquired in advance, and estimates the self-location by matching an image captured while traveling with the reference image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-8984 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the self-position estimation device described in Patent Document 1 has a problem in that the amount of data to be referenced is large, and the amount of calculation required for the self-position estimation process increases, which may make it impossible to estimate the self-position in real time. [Means for solving the problem]

[0005] The present disclosure can be realized in the following forms.

[0006] (1)According to one embodiment of the present disclosure, there is provided a self-location estimation device (210) that is mounted on a vehicle (M1) having a camera (110) and that estimates a position of the vehicle, and that includes: a feature amount calculation unit (211) that extracts feature points from images captured by the camera in the traveling direction of the vehicle and calculates a feature amount for each of the feature points; a feature point selection unit (212) that preferentially selects, from the feature points, feature points with high reliability as priority feature points; a self-location calculation unit (213) that calculates a self-location using the priority feature points; and an error correction unit (214) that calculates a re-projection error of the estimated self-location for each of the plurality of images captured at a predetermined time interval and calculates a corrected self-location by correcting the re-projection error. the self-location calculation unit calculates the self-location by matching the selected priority feature points with pre-stored three-dimensional map information; the error correction unit calculates three-dimensional coordinates of each of the priority feature points based on a correspondence relationship between each of the priority feature points among the plurality of images, re-projects the calculated three-dimensional coordinates of each feature point onto an image plane, and calculates the re-projection error as a distance between the priority feature point and the re-projected point on the image plane; the feature point selection unit preferentially selects, as the priority feature points, feature points that are commonly included in the plurality of time-series consecutive images, and selects the priority feature points so that a ratio of the number of the priority feature points to the total number of the feature points is at least 50% or more. . (2) According to another aspect of the present disclosure, there is provided a self-location estimation device (210) mounted on a vehicle (M1) having a camera (110) and estimating a position of the vehicle. The self-location estimation device includes a feature amount calculation unit (211) that extracts feature points from images captured by the camera in the traveling direction of the vehicle and calculates a feature amount for each of the feature points, a feature point selection unit (212) that preferentially selects feature points with high reliability as priority feature points from among the feature points, a self-location calculation unit (213) that calculates a self-location using the priority feature points, and an error correction unit (214) that calculates a re-projection error of the estimated self-location for each of the plurality of images captured at a predetermined time interval and calculates a corrected self-location by correcting the re-projection error, wherein the self-location calculation unit calculates the self-location by matching the selected priority feature points with pre-stored three-dimensional map information, and the error correction unit calculates the self-location by matching each of the priority feature points between the plurality of images. The self-location estimation device executes the following steps based on the correspondence between the priority feature points: calculating three-dimensional coordinates of each of the priority feature points based on the correspondence between the priority feature points; re-projecting the calculated three-dimensional coordinates of each feature point onto an image plane; and calculating the re-projection error as the distance between the priority feature point and the re-projected point on the image plane. The self-location estimation device further includes a cropping range designation unit (215), which designates a cropping range (TF) that is an image range included in the image and smaller than the size of the image. The feature point selection unit preferentially selects, from the feature points, feature points that exist within the cropping range as the priority feature points. The cropping range is a range that is smaller than an area in the image excluding an area that will be outside the imaging range in an image that is newly captured after the time interval has elapsed after capturing the image. (3) According to another aspect of the present disclosure, there is provided a self-location estimation device (210) mounted on a vehicle (M1) having a camera (110) and estimating a position of the vehicle. The self-location estimation device includes: a feature amount calculation unit (211) that extracts feature points from an image captured by the camera in the traveling direction of the vehicle and calculates a feature amount for each of the feature points; a feature point selection unit (212) that preferentially selects, from the feature points, feature points with high reliability as priority feature points; a self-location calculation unit (213) that calculates a self-location using the priority feature points; and an error correction unit (214) that calculates a re-projection error of the estimated self-location for each of the plurality of images captured at a predetermined time interval and calculates a corrected self-location by correcting the re-projection error. The self-location calculation unit calculates the self-location by matching the selected priority feature points with pre-stored three-dimensional map information, and the error correction unit calculates the self-location by matching the selected priority feature points with pre-stored three-dimensional map information. The self-location estimation device executes the following steps based on the correspondence between the priority feature points between the images: calculating three-dimensional coordinates of each of the priority feature points, re-projecting the calculated three-dimensional coordinates of each feature point onto an image plane; and calculating the re-projection error as a distance between the priority feature point and the re-projected point on the image plane. The self-location estimation device further includes a cropping range designation unit (215), which designates a cropping range (TF) that is an image range included in the image and smaller than the size of the image, and the feature point selection unit preferentially selects, from the feature points, feature points that exist within the cropping range as the priority feature points, and the size of the cropping range is set so that the horizontal angle of view is greater than 40° and the vertical angle of view is greater than 30°. (4) According to another aspect of the present disclosure, there is provided a self-location estimation device (210) mounted on a vehicle (M1) having a camera (110) and estimating a position of the vehicle. The self-location estimation device includes a feature amount calculation unit (211) that extracts feature points from images captured by the camera in the traveling direction of the vehicle and calculates a feature amount for each of the feature points, a feature point selection unit (212) that preferentially selects, from the feature points, feature points with high reliability as priority feature points, a self-location calculation unit (213) that calculates a self-location using the priority feature points, and an error correction unit (214) that calculates a re-projection error of the estimated self-location for each of the plurality of images captured at a predetermined time interval and calculates a corrected self-location by correcting the re-projection error, wherein the self-location calculation unit calculates the self-location by matching the selected priority feature points with pre-stored three-dimensional map information, and the error correction unit calculates the self-location by matching the selected priority feature points with pre-stored three-dimensional map information. the self-location estimation device further comprises a cropping range designation unit (215), which designates a cropping range (TF) that is an image range included in the image and is smaller than the size of the image, and the feature point selection unit preferentially selects, from the feature points, feature points that exist within the cropping range as the priority feature points, and the cropping range designation unit sets the size and position of the cropping range within the image using a route plan generated using the calculated self-location. (5) According to another aspect of the present disclosure, there is provided a self-location estimation device (210) mounted on a vehicle (M1) having a camera (110) and estimating a position of the vehicle. The self-location estimation device includes: a feature amount calculation unit (211) that extracts feature points from images captured by the camera in the traveling direction of the vehicle and calculates a feature amount for each of the feature points; a feature point selection unit (212) that preferentially selects, from the feature points, feature points with high reliability as priority feature points; a self-location calculation unit (213) that calculates a self-location using the priority feature points; and an error correction unit (214) that calculates a re-projection error of the estimated self-location for each of the plurality of images captured at a predetermined time interval and calculates a corrected self-location by correcting the re-projection error, wherein the self-location calculation unit calculates the self-location by matching the selected priority feature points with pre-stored three-dimensional map information, and the error correction unit calculates the self-location by matching each of the priority feature points between the plurality of images. the three-dimensional coordinates of each of the priority feature points based on the correspondence relationship between the feature points, the three-dimensional coordinates of each of the priority feature points being projected again onto an image plane, and the reprojection error being calculated as the distance between the priority feature point and the reprojected point on the image plane. The self-location estimation device further includes a cropping range designation unit (215), which designates a cropping range (TF) that is an image range included in the image and smaller than the size of the image, and the feature point selection unit preferentially selects, from the feature points, feature points that exist within the cropping range as the priority feature points, and the cropping range designation unit sets the size and position of the cropping range within the image by using information on the traveling conditions of the vehicle obtained from a sensor that measures the traveling direction and traveling speed of the vehicle.

[0007] According to this form of self-location estimation device, feature points with high reliability are preferentially selected from the feature points contained in the acquired image and the self-location is calculated, thereby reducing the number of feature points to be calculated, suppressing an increase in the amount of calculation required for self-location estimation, and suppressing a decrease in the accuracy of self-location estimation. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a block diagram showing a schematic configuration of a vehicle system according to a first embodiment. [Figure 2] 1 is a block diagram showing a schematic configuration of a self-position estimation device according to a first embodiment. [Figure 3] 4 is a flowchart showing a procedure of a vehicle control process according to the first embodiment. [Figure 4] 4 is a flowchart showing the procedure of a self-position estimation process according to the first embodiment. [Figure 5] FIG. 10 is an explanatory diagram showing an example of an acquired image. [Figure 6]FIG. 10 is an explanatory diagram showing an example of an acquired image. [Figure 7] FIG. 10 is an explanatory diagram showing an example of an acquired image. [Figure 8] FIG. 10 is a block diagram showing a schematic configuration of a vehicle system according to a second embodiment. [Figure 9] FIG. 10 is a block diagram showing a schematic configuration of a self-position estimation device according to a second embodiment. [Figure 10] 10 is a flowchart showing a procedure of a vehicle control process according to a second embodiment. [Figure 11] 10 is a flowchart showing the procedure of a self-position estimation process according to the second embodiment. [Figure 12] FIG. 10 is an explanatory diagram showing an example of a cutting range. [Figure 13] FIG. 10 is an explanatory diagram showing an example of a cutting range. [Figure 14] FIG. 10 is a block diagram showing a schematic configuration of a vehicle system according to a third embodiment. [Figure 15] 10 is a flowchart showing a procedure of a vehicle control process according to a third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] A. First embodiment: A-1. System Configuration: As shown in FIG. 1, the vehicle M1 includes an information acquisition device 100, a vehicle system 200, and a vehicle operation device 300. In this embodiment, the vehicle M1 is a vehicle equipped with an engine. The vehicle M1 is also capable of automatic driving and is configured to be able to switch between automatic driving and manual driving. "Automatic driving" refers to driving in which engine control, braking control, and steering control are automatically performed on behalf of the occupant. "Manual driving" refers to driving in which the occupant performs operations for engine control (depressing the accelerator pedal), braking control (depressing the brake pedal), and steering control (turning the steering wheel). The vehicle M1 is not limited to a vehicle equipped with an engine, and may also be an electric vehicle (EV) or a fuel cell vehicle (FCV).

[0010] The information acquisition device 100 acquires information about the position, driving status, and surrounding conditions of the vehicle M1. In this embodiment, the information acquisition device 100 includes a camera 110, a GNSS receiver 120, and an inertial sensor 130. The camera 110 captures an image of the vehicle M1 in the traveling direction. The GNSS receiver 120 detects the current position (longitude and latitude) of the vehicle M1 based on GNSS signals received from artificial satellites that make up the GNSS (Global Navigation Satellite System). The inertial sensor 130 includes an acceleration sensor and a yaw rate sensor, and detects triaxial acceleration, triaxial angular velocity, and the like that occur as the vehicle M1 travels.

[0011] The vehicle system 200 acquires the position information, driving conditions, and surrounding conditions of the vehicle M1 from the information acquisition device 100, estimates the vehicle's own position, and controls the vehicle operation device 300 of the vehicle M1 according to the estimated own position. The vehicle system 200 of this embodiment is configured with an ECU (Electronic Control Unit) equipped with a CPU and memory. The CPU of the ECU executes programs stored in advance in the memory, thereby functioning as a self-position estimation device 210, a map information acquisition unit 220, a fusion processing unit 230, a route planning unit 240, and a vehicle control unit 250.

[0012] The self-position estimation device 210 estimates the self-position of the vehicle M1 using an image acquired from the camera 110. The map information acquisition unit 220 acquires map information pre-stored in a memory provided in the vehicle system 200. The fusion processing unit 230 performs fusion processing using the self-position estimated by the self-position estimation device 210, the map information acquired by the map information acquisition unit 220, and the information acquired by the information acquisition device 100, to estimate a more reliable self-position of the vehicle M1. The route planning unit 240 generates a route plan for the vehicle M1 using the self-position estimated by the fusion processing unit 230. The vehicle control unit 250 controls the vehicle M1 in accordance with the route plan generated by the route planning unit 240. Specific processing in each functional unit will be described in the vehicle control processing section below.

[0013] As shown in FIG. 2, the self-location estimation device 210 includes a feature amount calculation unit 211, a feature point selection unit 212, a self-location calculation unit 213, and an error correction unit 214. The feature amount calculation unit 211 extracts feature points included in an image captured by the camera 110 and calculates a feature amount for each feature point. The feature point selection unit 212 selects feature points to be used in the self-location estimation calculation. The self-location calculation unit 213 calculates the self-location of the vehicle M1 using the selected feature points. The error correction unit 214 calculates a re-projection error of the self-location in multiple images and calculates the self-location with the re-projection error corrected. Specific processing in each functional unit will be described in the self-location estimation processing section below.

[0014] Vehicle operation device 300 includes engine mechanism 310, brake mechanism 320, and steering mechanism 330. Engine mechanism 310 is made up of a group of devices (actuators) that perform operations such as opening and closing a throttle valve, igniting an igniter, and opening and closing an intake valve. Brake mechanism 320 is made up of a group of devices (actuators) related to brake control, such as sensors, motors, valves, and pumps. Steering mechanism 330 is made up of a group of devices (actuators) related to steering, such as a power steering motor.

[0015] A-2. Vehicle control processing: 3, the vehicle system 200 estimates its own position using the information acquired from the information acquisition device 100 and the map information acquired from 220, and controls the vehicle M1 to travel according to a route plan generated based on the self-position. The vehicle control process is repeatedly executed while the vehicle M1 is traveling.

[0016] The vehicle system 200 performs a self-position estimation process (step S100), acquires GNSS information from the GNSS receiver 120 (step S110), and acquires inertial information from the inertial sensor 130 (step S120) in parallel.

[0017] In step S100, the self-location estimation device 210 executes the self-location estimation process shown in Fig. 4. In step S110, the self-location estimation device 210 acquires an image from the camera 110.

[0018] In step S120, the feature amount calculation unit 211 extracts feature points from the acquired image and calculates a feature amount for each feature point. In this embodiment, the feature amount calculation unit 211 extracts feature points using FAST (Features from Accelerated Segment Test) as a feature point extraction algorithm. Furthermore, the feature amount calculation unit 211 calculates feature amounts using SIFT (Scale Invariant Feature Transform), ORB (Oriented FAST and Rotated BRIEF), and SURF (Speeded Up Robust Features) as feature amount calculation algorithms. The feature amount calculation unit 211 is not limited to these feature amount calculation algorithms, and may also extract features using, for example, AKAZE (Accelerated KAZE), HOG (Histograms of Oriented Gradients), or CNN (Convolutional Neural Network). Furthermore, the feature amount calculation unit 211 performs image reduction, cropping, Gaussian processing, and the like as preprocessing for the above processing.

[0019] In step S130, the feature point selection unit 212 assigns a reliability to each extracted feature point according to the feature amount of that feature point. "Reliability" refers to the degree of contribution to the accuracy of self-localization estimation. For example, feature points derived from buildings have a high reliability, while feature points derived from accidentally captured oncoming vehicles or pedestrians have a low reliability. In this embodiment, the feature point selection unit 212 assigns each feature point an ID, feature point coordinates, Age, class, and feature amount information as parameters for managing reliability. "ID" refers to an identifier for each feature point used to identify feature points that appear in multiple images. The identity of feature points between different images can be identified, for example, by the feature point's coordinates in the images, the type and size of the feature amount, etc. "Age" refers to a parameter indicating the number of times each feature point has been extracted in multiple chronologically consecutive images. If a feature point that existed in a previously acquired image also exists in a newly acquired image, the feature point selection unit 212 adds "1" to the Age value of that feature point. Furthermore, "class" refers to a management parameter for managing feature points. By classifying feature points using the area, direction, length, etc. of the group of pixels (blob) that make up the feature point, the complexity of managing parameters for each feature point can be reduced. "Feature amount information" refers to information indicating the size and type of feature amount.

[0020] The Age addition process will be described in more detail using examples shown in FIGS. 5 to 7. FIGS. 5 to 7 show examples of images captured by camera 110 in a chronologically consecutive order. While an image would normally contain countless feature points, FIGS. 5 to 7 show only those feature points necessary for explanation. Since feature points FP1, FP2, and FP3 exist in image IM1 shown in FIG. 5, the Age values ​​for feature points FP1, FP2, and FP3 are each incremented by "1." Since feature points FP2 and FP3 exist in image IM2 shown in FIG. 6, the Age values ​​for feature points FP2 and FP3 are each incremented by "1." Since only feature point FP3 exists in image IM3 shown in FIG. 7, only the Age value for feature point FP3 is incremented by "1." Thus, in the examples shown in FIGS. 5 to 7, if the Age values ​​of feature points FP1, FP2, and FP3 in FIG. 5 are the same, the Age values ​​increase in the order of feature points FP3, FP2, and FP1 that appear most frequently in images IM1 to IM3.

[0021] In step S140 shown in FIG. 4, the feature point selection unit 212 selects priority feature points according to the set reliability. The "priority feature points" refer to feature points that are to be prioritized in the self-localization estimation process. In this embodiment, the feature point selection unit 212 ranks the feature points in descending order of their Age value, and selects a predetermined percentage of feature points in descending order as priority feature points. This is because if the Age value is large, that is, if a feature point appears frequently in multiple images, it is likely that the feature point is not a feature point derived from an oncoming vehicle captured accidentally, but rather a feature point derived from a building or the like that contributes to the accuracy of the self-localization estimation. Note that the ratio of the number of priority feature points to the total number of feature points extracted in step S120 is preferably set to at least 50% in order to ensure the accuracy of the self-localization estimation. In this way, by preferentially selecting priority feature points that are commonly contained in multiple images that are consecutive in time series, the number of feature points to be calculated can be reduced, thereby preventing an increase in the amount of calculation required for self-position estimation and preventing a decrease in the accuracy of self-position estimation.

[0022] In step S150, the self-location calculation unit 213 estimates the self-location of the vehicle M1 by using the selected priority feature points. More specifically, the self-location calculation unit 213 calculates the self-location of the vehicle M1 by matching the priority feature points in the image with the three-dimensional map information stored in the map information acquisition unit 220.

[0023] In step S160, the self-location estimation device 210 determines whether self-location estimation has been completed for a predetermined number of images. If the number of images has not reached the predetermined number (step S160: No), step S110 is executed again after a predetermined time interval (hereinafter also referred to as the "imaging interval") to acquire new images. In this manner, the self-location estimation device 210 executes the processes of steps S120, S130, S140, and S150 for each image until self-location estimation is completed for the predetermined number of images. Note that the "predetermined number" is determined in advance by conducting experiments or the like so as to ensure stable self-location estimation accuracy.

[0024] After the self-location estimation is completed for a preset number of images (step S160: Yes), the error correction unit 214 calculates the reprojection error of the estimated self-location for each of the multiple images and calculates the self-location with the reprojection error corrected (step S170). In this embodiment, the error correction unit 214 corrects the reprojection error of the self-location by performing bundle adjustment. Bundle adjustment refers to a technique for estimating geometric three-dimensional model parameters from input images. In bundle adjustment, the three-dimensional coordinates of each feature point are calculated based on the correspondence between each feature point in multiple images. The calculated three-dimensional coordinates of each feature point are then reprojected onto the image plane, and the reprojection error, calculated as the distance between the reprojected point and the feature point, is iteratively re-estimated to estimate more accurate three-dimensional coordinate values ​​of the feature points. Note that the self-location after the reprojection error correction calculated in this step corresponds to the "corrected self-location" in this disclosure. After completing this step, the self-location estimation device 210 terminates the self-location estimation process.

[0025] In step S200 shown in FIG. 3, the fusion processing unit 230 performs fusion processing using the 3D map information stored in the map information acquisition unit 220, the vehicle M1's own position, GNSS information, and inertial information to estimate a more reliable own position of the vehicle M1. The own position estimated in this step corresponds to the "fused own position" in this disclosure. By using information acquired in parallel from the camera 110, the GNSS receiver 120, and the inertial sensor 130 in this manner, even if information acquisition from some of these devices is interrupted, the vehicle M1's own position can be reliably estimated based on information acquired from the remaining devices. An example of a situation in which information acquisition from some devices is interrupted is when the vehicle M1 is traveling through a tunnel, in which acquisition of position information from the GNSS receiver 120 is interrupted.

[0026] In step S300, the route planning unit 240 generates a route plan for the vehicle M1 according to the vehicle's own position estimated by the fusion processing unit 230. More specifically, the route planning unit 240 generates a long-term route plan from the vehicle M1's current own position to a preset destination, and a short-term route plan of about 10 seconds according to the surrounding conditions of the vehicle M1, such as traffic restrictions and traffic signals preset in the map information and the distance to surrounding vehicles. The route plan includes information specifying vehicle control, such as the traveling speed and direction of the vehicle M1 at each point on the planned route.

[0027] In step S400, vehicle control unit 250 controls vehicle M1 in accordance with the route plan generated by route planning unit 240. Vehicle control unit 250 controls the actuators constituting engine mechanism 310 so that vehicle M1 travels at a specified travel speed. Vehicle control unit 250 also controls the actuators constituting brake mechanism 320 so that a specified braking amount is obtained at a specified timing. Vehicle control unit 250 also controls the actuators constituting steering mechanism 330 so that vehicle M1 travels along a specified traveling direction. After this step is completed, vehicle system 200 ends the vehicle control process.

[0028] According to the self-location estimation device 210 of the embodiment described above, the self-location is calculated by preferentially selecting priority feature points that are commonly included in multiple chronologically consecutive images, so that the number of feature points to be calculated can be reduced, and an increase in the amount of calculation required for self-location estimation can be suppressed, as well as a decrease in the accuracy of self-location estimation can be suppressed.

[0029] Furthermore, the feature point selection unit 212 selects priority feature points so that the ratio of the number of priority feature points to the total number of feature points extracted from the acquired image is at least 50% or more, thereby suppressing a decrease in the accuracy of self-position estimation.

[0030] B. Second embodiment: The self-location estimation device 210A of the second embodiment differs from the self-location estimation device 210 of the first embodiment in that it further includes a cropping range designation unit 215 as shown in Fig. 9 and acquires a route plan from a route planning unit 240 as shown in Fig. 8. The self-location estimation device 210A also differs from the self-location estimation device 210 of the first embodiment in that it executes steps S50 and S100A instead of step S100 in the vehicle control process shown in Fig. 10, and executes steps S132 and S142 instead of steps S130 and S140 in the self-location estimation process shown in Fig. 11. Note that the device configuration and other steps in the process of the self-location estimation device 210A of the second embodiment are the same as those of the self-location estimation device 210 of the first embodiment, and therefore the same configurations and steps are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0031] In the vehicle control process shown in FIG. 10, the self-position estimation device 210A acquires a route plan from the route planning unit 240 (step S50), and executes the self-position estimation process (step S100A).

[0032] In the self-position estimation process shown in FIG. 11, the feature amount calculation unit 211 acquires an image from the camera 110, extracts feature points from the acquired image, and calculates feature amounts, in the same manner as in steps S110 and S120 described above.

[0033] In step S132, the cropping range designation unit 215 uses the acquired route plan to designate the size of the cropping range TF and its position within the image. The "cropping range TF" refers to the range within the acquired image from which feature points are preferentially selected. In this embodiment, the cropping range designation unit 215 designates, as the cropping range TF, a rectangular area whose size is equal to or smaller than the imaging range of the image captured after the imaging interval has elapsed after capturing an image (hereinafter also referred to as the "next image range to be acquired"). The size of the next image range to be acquired can be predicted from information related to vehicle control, such as the traveling speed and direction of the vehicle M1, included in the route plan.

[0034] 12 and 13 show examples of the cropping range TF specified in images IM4 and IM5. The cropping range TF is preferably specified so that all lane markers LM painted on the road are included in the width direction of the road. More specifically, the cropping range TF is preferably specified so that the horizontal angle of view is greater than 40° and the vertical angle of view is greater than 30°. In this embodiment, as shown in FIG. 13, when the vehicle M1 is traveling around a curve, the position of the cropping range TF in image IM5 is offset according to the traveling direction of the vehicle M1. In this embodiment, the cropping range specifying unit 215 determines the offset amount based on the relationship between the traveling direction and the offset amount, which has been determined in advance through experiments or the like.

[0035] In step S142 shown in Fig. 11, the feature point selection unit 212 selects feature points within the clipping range TF as priority feature points. This is because feature points included in the area outside the clipping range TF (hereinafter also referred to as the "outer area Ar") are not included in the above-mentioned next image range to be acquired, and can be said to have a low degree of contribution to the accuracy of self-location estimation, that is, low reliability. In the example shown in Fig. 12, of the feature points FP4 to FP9 included in image IM4, feature points FP5 to FP7 included in the clipping range TF excluding the outer area Ar shown with hatching are selected as priority feature points. Furthermore, in the example shown in Fig. 13, of the feature points FP10 to FP15 included in image IM5, feature points FP13 to FP15 included in the clipping range TF are selected as priority feature points.

[0036] According to the self-location estimation device 210A of the second embodiment described above, a cutout range TF is specified using a route plan, and feature points included in the cutout range TF are preferentially selected. This makes it possible to reduce the number of feature points to be calculated, suppress an increase in the amount of calculation required for self-location estimation, and suppress a decrease in the accuracy of self-location estimation.

[0037] C. Third embodiment: As shown in Fig. 14, the self-location estimation device 210B of the third embodiment differs from the self-location estimation device 210A of the second embodiment in that, instead of a route plan, the self-location estimation device 210B acquires vehicle driving information of the vehicle M1 from a vehicle driving information acquisition device 140 included in the information acquisition device 100B. The self-location estimation device 210B also differs from the self-location estimation device 210A of the second embodiment in that, in the vehicle control process shown in Fig. 15, step S52 is executed instead of step S50. Note that the device configuration and other steps in the process of the self-location estimation device 210B of the third embodiment are the same as those of the self-location estimation device 210A of the second embodiment, and therefore the same configurations and steps are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0038] As shown in FIG. 14, the information acquisition device 100B of this embodiment further includes a vehicle driving information acquisition device 140 in addition to the configuration of the information acquisition device 100 of the first embodiment. The vehicle driving information acquisition device 140 is mounted on the vehicle M1 and measures vehicle driving information such as the driving speed and traveling direction of the vehicle M1 while it is traveling. The driving speed of the vehicle M1 can be measured using a vehicle speed sensor or an encoder. Furthermore, the traveling direction of the vehicle M1 can be predicted using the steering angle of the steering wheel and the detection value of a yaw rate sensor. Note that the vehicle driving information corresponds to "information related to the vehicle driving conditions" in this disclosure.

[0039] 15, the self-position estimation device 210B acquires vehicle driving information from the vehicle driving information acquisition device 140 (step S52). In the self-position estimation process (step S100A), the cropping range designation unit 215 predicts the above-mentioned next image acquisition range using the vehicle driving information instead of the route plan, and designates the cropping range TF.

[0040] According to the self-position estimation device 210B of the third embodiment described above, the cutting range TF is specified using vehicle driving information, so that feature points that appear in the actual traveling direction of the vehicle M1 can be preferentially selected, and the deterioration of the accuracy of self-position estimation can be further suppressed.

[0041] D. Other Embodiments: (D1) In the above embodiment, the map information acquisition unit 220 acquires map information stored in advance in the memory from the memory provided in the ECU, but the present disclosure is not limited to this. The map information acquisition unit 220 may acquire map information via communication from a device provided separately from the vehicle M1.

[0042] (D2) In the above embodiment, the clipping area designation unit 215 designates a rectangular area as the clipping area TF, but the present disclosure is not limited to this. The clipping area designation unit 215 may designate a circular area as the clipping area TF. Even in this embodiment, the same effects as those of the above embodiment are achieved.

[0043] (D3) In the above embodiment, the cropping range designation unit 215 designates the cropping range TF so that it is equal to or smaller than the next image range scheduled to be acquired, but the present disclosure is not limited to this. The cropping range designation unit 215 may set the cropping range TF to any size that is smaller than the acquired image and larger than the next image range scheduled to be acquired. Even in this embodiment, the number of feature points to be calculated can be reduced compared to when the self-position is calculated using all feature points included in the acquired image as the calculation target, thereby suppressing an increase in the amount of calculation required for self-position estimation.

[0044] The vehicle system 200 and the methods described herein may be implemented by a special-purpose computer configured with a processor and memory programmed to perform one or more functions embodied in a computer program. Alternatively, the vehicle system 200 and the methods described herein may be implemented by a special-purpose computer configured with a processor configured with one or more dedicated hardware logic circuits. Alternatively, the vehicle system 200 and the methods described herein may be implemented by one or more special-purpose computers configured with a processor configured with one or more hardware logic circuits in combination with a processor and memory programmed to perform one or more functions. Additionally, the computer program may be stored as instructions executed by a computer on a computer-readable non-transitory storage medium.

[0045] The present disclosure is not limited to the above-described embodiments and can be realized in various configurations without departing from the spirit thereof. For example, the technical features in each embodiment corresponding to the technical features in the form described in the Summary of the Invention section can be appropriately replaced or combined to solve some or all of the above-described problems or achieve some or all of the above-described effects. Furthermore, if a technical feature is not described as essential in this specification, it can be appropriately deleted. [Explanation of symbols]

[0046] 110... camera, 200... vehicle system, 210... self-position estimation device, 211... feature amount calculation unit, 212... feature point selection unit, 213... self-position calculation unit, 214... error correction unit, 220... map information acquisition unit, 230... fusion processing unit, 240... route planning unit, 250... vehicle control unit, M1... vehicle

Claims

1. A self-position estimation device (210) mounted on a vehicle (M1) having a camera (110) and estimating the position of the vehicle, a feature amount calculation unit (211) that extracts feature points from an image captured by the camera in the traveling direction of the vehicle and calculates a feature amount for each of the feature points; a feature point selection unit (212) that selects, from the feature points, feature points with high reliability as priority feature points; a self-position calculation unit (213) that calculates a self-position using the priority feature points; an error correction unit (214) that calculates a re-projection error of the self-position estimated for each of the plurality of images captured at a predetermined time interval, and calculates a corrected self-position by correcting the re-projection error; Equipped with the self-location calculation unit calculates the self-location by matching the selected priority feature points with pre-stored three-dimensional map information; The error correction unit calculating three-dimensional coordinates of each of the priority feature points based on a correspondence relationship between the priority feature points among the plurality of images; re-projecting the calculated three-dimensional coordinates of each feature point onto the image surface; calculating the reprojection error as a distance between the priority feature point and a reprojected reprojection point in the image plane; the feature point selection unit preferentially selects, as the priority feature points, feature points that are commonly included in the plurality of time-series consecutive images; the feature point selection unit selects the priority feature points so that a ratio of the number of the priority feature points to the total number of the feature points is at least 50% or more; Self-location estimation device.

2. A self-position estimation device (210) mounted on a vehicle (M1) having a camera (110) and estimating the position of the vehicle, a feature amount calculation unit (211) that extracts feature points from an image captured by the camera in the traveling direction of the vehicle and calculates a feature amount for each of the feature points; a feature point selection unit (212) that selects, from the feature points, feature points with high reliability as priority feature points; a self-position calculation unit (213) that calculates a self-position using the priority feature points; an error correction unit (214) that calculates a re-projection error of the self-position estimated for each of the plurality of images captured at a predetermined time interval, and calculates a corrected self-position by correcting the re-projection error; Equipped with the self-location calculation unit calculates the self-location by matching the selected priority feature points with pre-stored three-dimensional map information; The error correction unit calculating three-dimensional coordinates of each of the priority feature points based on a correspondence relationship between the priority feature points among the plurality of images; re-projecting the calculated three-dimensional coordinates of each feature point onto the image surface; calculating the reprojection error as a distance between the priority feature point and a reprojected reprojection point in the image plane; The self-location estimation device further includes a cutting range designation unit (215), the cropping region designation unit designates a cropping region (TF) that is an image region included in the image and that is smaller than the size of the image; the feature point selection unit preferentially selects, from the feature points, feature points that exist within the cutting range as the priority feature points; the cropping range is a range of a size equal to or smaller than an area in the image excluding an area that will be outside the imaging range in an image newly captured after the time interval has elapsed after capturing the image, Self-location estimation device.

3. A self-position estimation device (210) mounted on a vehicle (M1) having a camera (110) and estimating the position of the vehicle, a feature amount calculation unit (211) that extracts feature points from an image captured by the camera in the traveling direction of the vehicle and calculates a feature amount for each of the feature points; a feature point selection unit (212) that selects, from the feature points, feature points with high reliability as priority feature points; a self-position calculation unit (213) that calculates a self-position using the priority feature points; an error correction unit (214) that calculates a re-projection error of the self-position estimated for each of the plurality of images captured at a predetermined time interval, and calculates a corrected self-position by correcting the re-projection error; Equipped with the self-location calculation unit calculates the self-location by matching the selected priority feature points with pre-stored three-dimensional map information; The error correction unit calculating three-dimensional coordinates of each of the priority feature points based on a correspondence relationship between the priority feature points among the plurality of images; re-projecting the calculated three-dimensional coordinates of each feature point onto the image surface; calculating the reprojection error as a distance between the priority feature point and a reprojected reprojection point in the image plane; The self-location estimation device further includes a cutting range designation unit (215), the cropping region designation unit designates a cropping region (TF) that is an image region included in the image and that is smaller than the size of the image; the feature point selection unit preferentially selects, from the feature points, feature points that exist within the cutting range as the priority feature points; The size of the cutout range is set so that the horizontal angle of view is greater than 40° and the vertical angle of view is greater than 30°. Self-location estimation device.

4. A self-position estimation device (210) mounted on a vehicle (M1) having a camera (110) and estimating the position of the vehicle, a feature amount calculation unit (211) that extracts feature points from an image captured by the camera in the traveling direction of the vehicle and calculates a feature amount for each of the feature points; a feature point selection unit (212) that selects, from the feature points, feature points with high reliability as priority feature points; a self-position calculation unit (213) that calculates a self-position using the priority feature points; an error correction unit (214) that calculates a re-projection error of the self-position estimated for each of the plurality of images captured at a predetermined time interval, and calculates a corrected self-position by correcting the re-projection error; Equipped with the self-location calculation unit calculates the self-location by matching the selected priority feature points with pre-stored three-dimensional map information; The error correction unit calculating three-dimensional coordinates of each of the priority feature points based on a correspondence relationship between the priority feature points among the plurality of images; re-projecting the calculated three-dimensional coordinates of each feature point onto the image surface; calculating the reprojection error as a distance between the priority feature point and a reprojected reprojection point in the image plane; The self-location estimation device further includes a cutting range designation unit (215), the cropping region designation unit designates a cropping region (TF) that is an image region included in the image and that is smaller than the size of the image; the feature point selection unit preferentially selects, from the feature points, feature points that exist within the cutting range as the priority feature points; the cutting area designation unit sets a size and a position within the image of the cutting area by using a route plan generated by using the calculated self-position. Self-location estimation device.

5. A self-position estimation device (210) mounted on a vehicle (M1) having a camera (110) and estimating the position of the vehicle, a feature amount calculation unit (211) that extracts feature points from an image captured by the camera in the traveling direction of the vehicle and calculates a feature amount for each of the feature points; a feature point selection unit (212) that selects, from the feature points, feature points with high reliability as priority feature points; a self-position calculation unit (213) that calculates a self-position using the priority feature points; an error correction unit (214) that calculates a re-projection error of the self-position estimated for each of the plurality of images captured at a predetermined time interval, and calculates a corrected self-position by correcting the re-projection error; Equipped with the self-location calculation unit calculates the self-location by matching the selected priority feature points with pre-stored three-dimensional map information; The error correction unit calculating three-dimensional coordinates of each of the priority feature points based on a correspondence relationship between the priority feature points among the plurality of images; re-projecting the calculated three-dimensional coordinates of each feature point onto the image surface; calculating the reprojection error as a distance between the priority feature point and a reprojected reprojection point in the image plane; The self-location estimation device further includes a cutting range designation unit (215), the cropping region designation unit designates a cropping region (TF) that is an image region included in the image and that is smaller than the size of the image; the feature point selection unit preferentially selects, from the feature points, feature points that exist within the cutting range as the priority feature points; the cutout area designation unit sets the size and position of the cutout area within the image by using information on the traveling conditions of the vehicle acquired from a sensor that measures the traveling direction and traveling speed of the vehicle. Self-location estimation device.

6. A vehicle system (200), comprising: A self-position estimation device mounted on a vehicle having a camera and estimating a position of the vehicle, a feature amount calculation unit that extracts feature points from an image captured by the camera in the traveling direction of the vehicle and calculates a feature amount for each of the feature points; a feature point selection unit that selects, from the feature points, feature points with high reliability as priority feature points; a self-location calculation unit that calculates a self-location by using the priority feature point; an error correction unit that calculates a re-projection error of the self-position estimated for each of the plurality of images captured at a predetermined time interval, and calculates a corrected self-position by correcting the re-projection error; Equipped with a self-location estimation device; a map information acquisition unit (220) for acquiring map information; a fusion processing unit (230) that estimates a fusion self-position of the vehicle using the corrected self-position calculated by the self-position estimation device, the map information, the position of the vehicle acquired by receiving a GNSS (Global Navigation Satellite System) signal, and information on the running status of the vehicle acquired from a sensor provided in the vehicle; a route planning unit (240) that generates a route plan using the estimated fusion self-location; a vehicle control unit (250) that controls the vehicle according to the generated route plan; Equipped with the self-location calculation unit calculates the self-location by matching the selected priority feature points with pre-stored three-dimensional map information; The error correction unit calculating three-dimensional coordinates of each of the priority feature points based on a correspondence relationship between the priority feature points among the plurality of images; re-projecting the calculated three-dimensional coordinates of each feature point onto the image surface; calculating the reprojection error as a distance between the priority feature point and a reprojected reprojection point in the image plane; the feature point selection unit preferentially selects, as the priority feature points, feature points that are commonly included in the plurality of time-series consecutive images; the feature point selection unit selects the priority feature points so that a ratio of the number of the priority feature points to the total number of the feature points is at least 50% or more; Vehicle systems.

7. A vehicle system (200), A self-position estimation device mounted on a vehicle having a camera and estimating a position of the vehicle, a feature amount calculation unit that extracts feature points from an image captured by the camera in the traveling direction of the vehicle and calculates a feature amount for each of the feature points; a feature point selection unit that selects, from the feature points, feature points with high reliability as priority feature points; a self-location calculation unit that calculates a self-location by using the priority feature point; an error correction unit that calculates a re-projection error of the self-position estimated for each of the plurality of images captured at a predetermined time interval, and calculates a corrected self-position by correcting the re-projection error; Equipped with a self-location estimation device; a map information acquisition unit (220) for acquiring map information; a fusion processing unit (230) that estimates a fusion self-position of the vehicle using the corrected self-position calculated by the self-position estimation device, the map information, the position of the vehicle acquired by receiving a GNSS (Global Navigation Satellite System) signal, and information on the running status of the vehicle acquired from a sensor provided in the vehicle; a route planning unit (240) that generates a route plan using the estimated fusion self-location; a vehicle control unit (250) that controls the vehicle according to the generated route plan; Equipped with the self-location calculation unit calculates the self-location by matching the selected priority feature points with pre-stored three-dimensional map information; The error correction unit calculating three-dimensional coordinates of each of the priority feature points based on a correspondence relationship between the priority feature points among the plurality of images; re-projecting the calculated three-dimensional coordinates of each feature point onto the image surface; calculating the reprojection error as a distance between the priority feature point and a reprojected reprojection point in the image plane; The self-location estimation device further includes a cutting range designation unit (215), the cropping region designation unit designates a cropping region (TF) that is an image region included in the image and that is smaller than the size of the image; the feature point selection unit preferentially selects, from the feature points, feature points that exist within the cutting range as the priority feature points; the cropping range is a range of a size equal to or smaller than an area in the image excluding an area that will be outside the imaging range in an image newly captured after the time interval has elapsed after capturing the image, Vehicle systems.

8. A vehicle system (200), A self-position estimation device mounted on a vehicle having a camera and estimating a position of the vehicle, a feature amount calculation unit that extracts feature points from an image captured by the camera in the traveling direction of the vehicle and calculates a feature amount for each of the feature points; a feature point selection unit that selects, from the feature points, feature points with high reliability as priority feature points; a self-location calculation unit that calculates a self-location by using the priority feature point; an error correction unit that calculates a re-projection error of the self-position estimated for each of the plurality of images captured at a predetermined time interval, and calculates a corrected self-position by correcting the re-projection error; Equipped with a self-location estimation device; a map information acquisition unit (220) for acquiring map information; a fusion processing unit (230) that estimates a fusion self-position of the vehicle using the corrected self-position calculated by the self-position estimation device, the map information, the position of the vehicle acquired by receiving a GNSS (Global Navigation Satellite System) signal, and information on the running status of the vehicle acquired from a sensor provided in the vehicle; a route planning unit (240) that generates a route plan using the estimated fusion self-location; a vehicle control unit (250) that controls the vehicle according to the generated route plan; Equipped with the self-location calculation unit calculates the self-location by matching the selected priority feature points with pre-stored three-dimensional map information; The error correction unit calculating three-dimensional coordinates of each of the priority feature points based on a correspondence relationship between the priority feature points among the plurality of images; re-projecting the calculated three-dimensional coordinates of each feature point onto the image surface; calculating the reprojection error as a distance between the priority feature point and a reprojected reprojection point in the image plane; The self-location estimation device further includes a cutting range designation unit (215), the cropping region designation unit designates a cropping region (TF) that is an image region included in the image and that is smaller than the size of the image; the feature point selection unit preferentially selects, from the feature points, feature points that exist within the cutting range as the priority feature points; The size of the cutout range is set so that the horizontal angle of view is greater than 40° and the vertical angle of view is greater than 30°. Vehicle systems.

9. A vehicle system (200), A self-position estimation device mounted on a vehicle having a camera and estimating a position of the vehicle, a feature amount calculation unit that extracts feature points from an image captured by the camera in the traveling direction of the vehicle and calculates a feature amount for each of the feature points; a feature point selection unit that selects, from the feature points, feature points with high reliability as priority feature points; a self-location calculation unit that calculates a self-location by using the priority feature point; an error correction unit that calculates a re-projection error of the self-position estimated for each of the plurality of images captured at a predetermined time interval, and calculates a corrected self-position by correcting the re-projection error; Equipped with a self-location estimation device; a map information acquisition unit (220) for acquiring map information; a fusion processing unit (230) that estimates a fusion self-position of the vehicle using the corrected self-position calculated by the self-position estimation device, the map information, the position of the vehicle acquired by receiving a GNSS (Global Navigation Satellite System) signal, and information on the running status of the vehicle acquired from a sensor provided in the vehicle; a route planning unit (240) that generates a route plan using the estimated fusion self-location; a vehicle control unit (250) that controls the vehicle according to the generated route plan; Equipped with the self-location calculation unit calculates the self-location by matching the selected priority feature points with pre-stored three-dimensional map information; The error correction unit calculating three-dimensional coordinates of each of the priority feature points based on a correspondence relationship between the priority feature points among the plurality of images; re-projecting the calculated three-dimensional coordinates of each feature point onto the image surface; calculating the reprojection error as a distance between the priority feature point and a reprojected reprojection point in the image plane; The self-location estimation device further includes a cutting range designation unit (215), the cropping region designation unit designates a cropping region (TF) that is an image region included in the image and that is smaller than the size of the image; the feature point selection unit preferentially selects, from the feature points, feature points that exist within the cutting range as the priority feature points; the cutting area designation unit sets a size and a position within the image of the cutting area by using the route plan generated by using the calculated self-position. Vehicle systems.

10. A vehicle system (200), comprising: A self-position estimation device mounted on a vehicle having a camera and estimating a position of the vehicle, a feature amount calculation unit that extracts feature points from an image captured by the camera in the traveling direction of the vehicle and calculates a feature amount for each of the feature points; a feature point selection unit that selects, from the feature points, feature points with high reliability as priority feature points; a self-location calculation unit that calculates a self-location by using the priority feature point; an error correction unit that calculates a re-projection error of the self-position estimated for each of the plurality of images captured at a predetermined time interval, and calculates a corrected self-position by correcting the re-projection error; Equipped with a self-location estimation device; a map information acquisition unit (220) for acquiring map information; a fusion processing unit (230) that estimates a fusion self-position of the vehicle using the corrected self-position calculated by the self-position estimation device, the map information, the position of the vehicle acquired by receiving a GNSS (Global Navigation Satellite System) signal, and information on the running status of the vehicle acquired from a sensor provided in the vehicle; a route planning unit (240) that generates a route plan using the estimated fusion self-location; a vehicle control unit (250) that controls the vehicle according to the generated route plan; Equipped with the self-location calculation unit calculates the self-location by matching the selected priority feature points with pre-stored three-dimensional map information; The error correction unit calculating three-dimensional coordinates of each of the priority feature points based on a correspondence relationship between the priority feature points among the plurality of images; re-projecting the calculated three-dimensional coordinates of each feature point onto the image surface; calculating the reprojection error as a distance between the priority feature point and a reprojected reprojection point in the image plane; The self-location estimation device further includes a cutting range designation unit (215), the cropping region designation unit designates a cropping region (TF) that is an image region included in the image and that is smaller than the size of the image; the feature point selection unit preferentially selects, from the feature points, feature points that exist within the cutting range as the priority feature points; the cutout area designation unit sets the size and position of the cutout area within the image by using information on the traveling conditions of the vehicle acquired from a sensor that measures the traveling direction and traveling speed of the vehicle. Vehicle systems.

Images