Position estimation device and vehicle control system

Abstract

To restrain the amount of data storage and estimate a position of one's own vehicle precisely.SOLUTION: A position estimation device 50 includes: a feature point extraction section 112 for extracting a feature point from a camera image of a camera 1a; a storage section 12 for storing an environment map including a feature point corresponding to the outside situation in advance; a position estimation section 113 for estimating a position of one's own vehicle on the basis of a feature point extracted by the feature point extraction section 112 and an environment map stored in the storage section 12; a reliability determination section 114 for determining whether the reliability of a position of one's own vehicle estimated by the position estimation section 113 is less than a predetermined degree; an environment map generation section 115 for generating an environment map by using a feature point extracted by the feature point extraction section 112 when the reliability determination section 114 determines that the reliability is less than a predetermined degree; and a sub-position estimation section 116 for estimating a position of one's own vehicle on the basis of a feature point extracted by the feature point extraction section 112 and an environment map generated by the environment map generation section 115.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a position estimation device that estimates the position of a vehicle and a vehicle control system. [Background technology]

[0002] As this type of device, a device configured to compare feature points extracted from an image captured by a camera mounted on a terminal device such as a smartphone with a feature point map to estimate the current location of the terminal device has been known (see, for example, Patent Document 1). The device described in Patent Document 1 selects a feature point map corresponding to the imaging conditions when the captured image was acquired from a plurality of feature point maps corresponding to a plurality of imaging conditions defined by season, time, weather, etc., and estimates the current location of the terminal device using the feature point map. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-56629 Summary of the Invention [Problem to be solved by the invention]

[0004] However, preparing a feature point map for each imaging condition as in the device described in Patent Document 1 increases the amount of data stored.On the other hand, if only feature point maps corresponding to specific imaging conditions are prepared in order to suppress the increase in the amount of data stored, it becomes difficult to accurately estimate the position of the vehicle. [Means for solving the problem]

[0005] A position estimation device according to one aspect of the present invention includes a detector that detects an external situation of a vehicle; a feature point extraction unit that extracts feature points from detection data of the detector; a memory unit that stores first map information including feature points corresponding to the external situation in advance; a first estimation unit that estimates a position of the vehicle based on the feature points extracted by the feature point extraction unit and the first map information stored in the memory unit; a reliability determination unit that determines whether the reliability of the position of the vehicle estimated by the first estimation unit is less than a predetermined level; a map generation unit that generates second map information using the feature points extracted by the feature point extraction unit when the reliability determination unit determines that the reliability is less than the predetermined level; and a second estimation unit that estimates the position of the vehicle based on the feature points extracted by the feature point extraction unit and the second map information generated by the map generation unit. The first map information is map information of a road section on which the vehicle has previously traveled, and includes feature points extracted by a feature point extraction unit from detection data acquired by a detector while the vehicle was traveling on the road section. The first estimation unit estimates the current position of the vehicle traveling on the road section based on the feature points extracted by the feature point extraction unit from current detection data acquired by the detector while the vehicle was traveling on the road section and on the first map information, and the reliability determination unit determines that the reliability is less than a predetermined level when the difference between the number of feature points corresponding to a predetermined area ahead in the vehicle's traveling direction, extracted by the feature point extraction unit from the current detection data, and the number of feature points corresponding to the predetermined area among the feature points included in the first map information stored in the memory unit, is equal to or greater than a predetermined threshold.

[0006] A vehicle control system according to another aspect of the present invention includes the above-described position estimation device, a driving actuator, and a driving control unit that controls the driving actuator based on the vehicle position estimated by the first estimator or the second estimator. When the reliability determination unit determines that the reliability is less than a predetermined level while the driving control unit is controlling the driving actuator based on the vehicle position estimated by the first estimator, the driving control unit stops controlling the driving actuator based on the vehicle position estimated by the first estimator and starts controlling the driving actuator based on the vehicle position estimated by the second estimator. [Effects of the Invention]

[0007] According to the present invention, the position of the vehicle is estimated with high accuracy while suppressing an increase in the amount of data stored. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a block diagram showing an overall configuration of a vehicle control system according to an embodiment of the present invention; [Figure 2A] 2 is a diagram for explaining a position recognition process in the vehicle position recognition unit of FIG. 1; [Figure 2B]2 is a diagram for explaining a position recognition process in the vehicle position recognition unit of FIG. 1; [Figure 3] 1 is a block diagram showing a configuration of a main part of a position estimation device according to an embodiment of the present invention; [Figure 4] 4 is a flowchart showing an example of processing executed by a CPU of the controller of FIG. 3; DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings. A position estimation device according to an embodiment of the present invention can be applied to a vehicle having an automatic driving function, i.e., an automatically driven vehicle. Note that a vehicle to which a position estimation device according to the present embodiment is applied may be referred to as a host vehicle to distinguish it from other vehicles. The host vehicle may be an engine vehicle having an internal combustion engine (engine) as a driving source, an electric vehicle having a driving motor as a driving source, or a hybrid vehicle having an engine and a driving motor as driving sources. The host vehicle can travel not only in an automatic driving mode in which no driving operation by the driver is required, but also in a manual driving mode in which the driver operates the vehicle.

[0010] First, a schematic configuration of a host vehicle related to autonomous driving will be described. Fig. 1 is a block diagram showing a schematic overall configuration of a vehicle control system 100 of a host vehicle having a position estimation device according to this embodiment. As shown in Fig. 1, the vehicle control system 100 mainly includes a controller 10, an external sensor group 1 and an internal sensor group 2, each of which is communicatively connected to the controller 10, an input / output device 3, a positioning unit 4, a map database 5, a navigation device 6, a communication unit 7, and a driving actuator AC.

[0011] The external sensor group 1 is a collective term for a plurality of sensors (external sensors) that detect the external situation, which is information about the surroundings of the host vehicle. For example, the external sensor group 1 includes a lidar that measures the distance from the host vehicle to surrounding obstacles by measuring scattered light in response to light irradiated in all directions of the host vehicle, a radar that detects other vehicles and obstacles around the host vehicle by irradiating electromagnetic waves and detecting reflected waves, and a camera that is mounted on the host vehicle and has an imaging element (image sensor) such as a CCD or CMOS that captures images of the surroundings (front, rear, and sides) of the host vehicle.

[0012] The internal sensor group 2 is a collective term for a plurality of sensors (internal sensors) that detect the driving state of the vehicle. For example, the internal sensor group 2 includes an inertial measurement unit (IMU) that detects the rotational angular velocity around three axes of the center of gravity of the vehicle, namely the vertical direction, the front-rear direction (travel direction), and the left-right direction (vehicle width direction), as well as the acceleration in three axes. The internal sensor group 2 also includes sensors that detect the driver's driving operations in manual driving mode, such as operation of the accelerator pedal, operation of the brake pedal, operation of the steering wheel, etc.

[0013] The input / output device 3 is a general term for devices that input commands from the driver and output information to the driver. For example, the input / output device 3 includes various switches through which the driver inputs various commands by operating operating members, a microphone through which the driver inputs commands by voice, a display that provides information to the driver via displayed images, a speaker that provides information to the driver by voice, etc.

[0014] The positioning unit (GNSS unit) 4 has a positioning sensor that receives positioning signals transmitted from positioning satellites. The positioning satellites are artificial satellites such as GPS satellites and quasi-zenith satellites. The positioning unit 4 measures the current position (latitude, longitude, altitude) of the vehicle using the positioning information received by the positioning sensor.

[0015] The map database 5 is a device that stores general map information used in the navigation device 6, and is configured with, for example, a hard disk or semiconductor elements. The map information includes road position information, road shape information (curvature, etc.), and position information of intersections and branch points. Note that the map information stored in the map database 5 is different from the highly accurate map information stored in the memory unit 12 of the controller 10.

[0016] The navigation device 6 is a device that searches for a target route on roads to a destination input by the driver and provides guidance along the target route. The input of the destination and guidance along the target route are performed via the input / output device 3. The target route is calculated based on the current position of the vehicle measured by the positioning unit 4 and map information stored in the map database 5. The current position of the vehicle can also be measured using detection values ​​from the external sensor group 1, and the target route can be calculated based on this current position and high-precision map information stored in the memory unit 12.

[0017] The communication unit 7 communicates with various servers (not shown) via networks including wireless communication networks such as the Internet and mobile phone networks, and acquires map information, driving history information, traffic information, and the like from the servers periodically or at any timing. In addition to acquiring driving history information, the vehicle's driving history information may also be transmitted to the server via the communication unit 7. Networks include not only public wireless communication networks but also closed communication networks established for each predetermined management area, such as wireless LAN, Wi-Fi (registered trademark), Bluetooth (registered trademark), and the like. The acquired map information is output to the map database 5 and the storage unit 12, where the map information is updated.

[0018] Actuators AC are driving actuators for controlling the driving of the host vehicle. When the driving source is an engine, actuators AC include a throttle actuator that adjusts the opening of the engine's throttle valve (throttle opening). When the driving source is a driving motor, actuators AC include the driving motor. Actuators AC also include a brake actuator that operates the host vehicle's braking device and a steering actuator that drives the steering device.

[0019] The controller 10 is composed of an electronic control unit (ECU). More specifically, the controller 10 includes a computer having an arithmetic unit 11 such as a CPU (microprocessor), a storage unit 12 such as a ROM and RAM, and other peripheral circuits (not shown) such as an I / O interface. Note that although multiple ECUs with different functions, such as an engine control ECU, a traction motor control ECU, and a braking device ECU, can be provided separately, for convenience, the controller 10 is shown in FIG. 1 as a collection of these ECUs.

[0020] The memory unit 12 stores highly accurate, detailed map information (referred to as high-accuracy map information). The high-accuracy map information includes road position information, road shape (curvature, etc.), road gradient information, intersection and branch point position information, types of road dividing lines such as white lines and their positions, the number of lanes, lane width and position information for each lane (information on the center position of the lane and boundary lines of the lane positions), position information of landmarks (buildings, traffic lights, signs, etc.) as map markers, and road surface profile information such as road surface irregularities. In the embodiment, center lines, lane boundary lines, outer lane lines, etc. are collectively referred to as road dividing lines. The high-accuracy map information stored in the memory unit 12 includes map information acquired from outside the vehicle via the communication unit 7 (referred to as external map information), and a map (referred to as internal map information) created by the vehicle itself using detection values ​​from the external sensor group 1 or detection values ​​from the external sensor group 1 and the internal sensor group 2.

[0021] The external map information is, for example, information on a map (referred to as a cloud map) acquired via a cloud server, and the internal map information is information on a map (referred to as an environmental map) made up of three-dimensional point cloud data generated by mapping using a technology such as SLAM (Simultaneous Localization and Mapping). The external map information is shared between the vehicle and other vehicles, whereas the internal map information is map information unique to the vehicle (for example, map information that the vehicle possesses independently). For roads on which the vehicle has not yet traveled or newly constructed roads, the vehicle itself creates an environmental map. The internal map information may be provided to a server device or other vehicles via the communication unit 7. In addition to the high-precision map information described above, the storage unit 12 also stores information such as the vehicle's travel trajectory information, various control programs, and thresholds used in the programs.

[0022] The calculation unit 11 has, as functional components, a vehicle position recognition unit 13, an external environment recognition unit 14, a behavior plan generation unit 15, a driving control unit 16, and a map generation unit 17.

[0023] The vehicle position recognition unit 13 recognizes (or may call it estimates) the position of the vehicle on the map (the vehicle's own position) based on the vehicle's own position information obtained by the positioning unit 4 and the map information in the map database 5. The vehicle's own position may be recognized (estimated) using high-precision map information stored in the memory unit 12 and information about the vehicle's surroundings detected by the external sensor group 1, thereby enabling the vehicle's own position to be recognized with high precision. The vehicle's own position can also be recognized by calculating the vehicle's movement information (movement direction, movement distance) based on the detection values ​​of the internal sensor group 2. Note that when the vehicle's own position can be measured by an external sensor installed on or beside the road, the vehicle's own position can also be recognized by communicating with the sensor via the communication unit 7.

[0024] The external environment recognition unit 14 recognizes the external situation around the host vehicle based on signals from the external sensor group 1, such as a lidar, radar, and camera. For example, it recognizes the positions, speeds, and accelerations of surrounding vehicles (vehicles ahead and vehicles behind) traveling around the host vehicle, the positions of surrounding vehicles stopped or parked around the host vehicle, and the positions and states of other objects. Examples of other objects include signs, traffic lights, markings such as road dividing lines and stop lines, buildings, guardrails, utility poles, billboards, pedestrians, and bicycles. Examples of the states of other objects include the color of traffic lights (red, green, yellow), the movement speed and direction of pedestrians and bicycles, and the like. Some of the stationary objects among the other objects constitute landmarks that serve as indicators of locations on a map, and the external environment recognition unit 14 also recognizes the positions and types of the landmarks.

[0025] The behavior plan generation unit 15 generates a traveling trajectory (target trajectory) of the host vehicle from the current time to a predetermined time ahead based on, for example, a target route calculated by the navigation device 6, high-precision map information stored in the memory unit 12, the host vehicle position recognized by the host vehicle position recognition unit 13, and the external situation recognized by the external environment recognition unit 14. When there are multiple trajectories that are candidates for the target trajectory on the target route, the behavior plan generation unit 15 selects an optimal trajectory from among them that satisfies criteria such as compliance with laws and regulations and efficient and safe traveling, and sets the selected trajectory as the target trajectory. The behavior plan generation unit 15 then generates a behavior plan according to the generated target trajectory. The behavior plan generation unit 15 generates various behavior plans corresponding to overtaking driving to overtake a preceding vehicle, lane-changing driving to change the traveling lane, following driving to follow a preceding vehicle, lane-keeping driving to maintain the traveling lane without deviating from the traveling lane, decelerating driving, accelerating driving, etc. When generating the target trajectory, the behavior plan generation unit 15 first determines a driving mode and generates the target trajectory based on the driving mode.

[0026] In the autonomous driving mode, the driving control unit 16 controls the actuators AC so that the vehicle travels along the target trajectory generated by the behavior plan generation unit 15. More specifically, in the autonomous driving mode, the driving control unit 16 calculates a required driving force for achieving the target acceleration per unit time calculated by the behavior plan generation unit 15, taking into account the driving resistance determined by the road gradient, etc. Then, for example, the driving control unit 16 feedback-controls the actuators AC so that the actual acceleration detected by the internal sensor group 2 becomes the target acceleration. In other words, the driving control unit 16 controls the actuators AC so that the vehicle travels at the target vehicle speed and target acceleration. Note that in the manual driving mode, the driving control unit 16 controls the actuators AC in response to driving commands (such as steering operations) from the driver acquired by the internal sensor group 2.

[0027] While driving in manual driving mode, the map generation unit 17 uses detection values ​​detected by the external sensor group 1 to generate an environmental map of the area around the road on which the vehicle has traveled as internal map information. For example, edges indicating the contours of objects are extracted from multiple frames of camera images acquired by a camera based on brightness and color information for each pixel, and feature points are extracted using the edge information. Feature points are, for example, intersections of edges, and correspond to corners of buildings or corners of road signs. The map generation unit 17 estimates the position and orientation of the camera according to a SLAM technology algorithm so that identical feature points converge to a single point across multiple frames of camera images, and calculates the three-dimensional positions of the feature points. By performing this calculation process for each of the multiple feature points, an environmental map consisting of three-dimensional point cloud data is generated. Note that instead of a camera, data acquired by radar or lidar may be used to extract feature points of objects around the vehicle and generate an environmental map.

[0028] The vehicle position recognition unit 13 may perform a process of recognizing the position of the vehicle based on the environmental map generated by the map generation unit 17 and feature points extracted from the camera image. The vehicle position recognition unit 13 may also perform the process of recognizing the position of the vehicle in parallel with the map generation process by the map generation unit 17. The map generation process and the position recognition (estimation) process are performed simultaneously according to an algorithm of SLAM technology. The map generation unit 17 can generate an environmental map not only when driving in manual driving mode, but also when driving in autonomous driving mode. If an environmental map has already been generated and stored in the memory unit 12, the map generation unit 17 may update the environmental map based on feature points (which may also be referred to as new feature points) newly extracted from newly acquired camera images.

[0029] 2A and 2B are diagrams for explaining the position recognition processing in the vehicle position recognition unit 13. FIG. 2A shows how the vehicle position is recognized based on the environmental map generated by the map generation unit 17 and feature points extracted from the camera image. FIG. 2A also shows a schematic representation of part of the point cloud data (seven feature points FP) included in the environmental map. The camera image IM t-3 ,IM t-2 ,IM t-1 ,IM t are frame image data (camera images) acquired by the camera 1a at times t-3, t-2, t-1, and t. The black dots fp in each camera image schematically represent feature points extracted from the camera images. The vehicle position recognition unit 13 estimates the position and attitude of the vehicle (specifically, the camera 1a) on the environmental map by finding a feature point fp (hereinafter referred to as a corresponding point) that corresponds to the feature point FP on the environmental map from among the group of feature points in the camera images. A dashed line connecting the feature point FP and the feature point fp indicates that the feature point FP and the feature point fp correspond to each other. The vehicle position recognition unit 13 adjusts (corrects) the estimated vehicle position and attitude based on multiple frames of camera images to improve the accuracy of position estimation. For example, when the camera image IM t The position and attitude of the vehicle at time t estimated based on the camera image IM t-3 ,IMt-2 ,IM t-1 The vehicle position recognition unit 13 tracks the position and attitude of the vehicle at each time estimated in this way, thereby recognizing the position of the vehicle on the environmental map while it is traveling.

[0030] However, when the time of day when the environmental map was generated, the brightness around the vehicle, the weather (climate), and other conditions (hereinafter referred to as environmental conditions) are different from when the camera image was acquired, the feature point FP corresponding to the feature point fp extracted from the camera image may not exist on the environmental map, or the point corresponding to the feature point FP on the environmental map may not exist in the camera image. FIG. 2B shows an example of a camera image acquired under environmental conditions different from when the environmental map was generated. For example, when the time of day when the environmental map was generated is daytime and the time of day when the camera image was acquired is nighttime, the appearance of objects such as structures around the vehicle differs between daytime and nighttime, so the camera image IM in FIG. 2B t-2 In some cases, the feature point corresponding to the feature point FP on the environmental map may not be extracted. t-2 The dotted circle in the figure indicates the feature point FP on the environmental map that corresponds to the feature point in the camera image IM. t-2 10 is a schematic diagram showing a state in which feature points are not extracted from the map. In such a case, the accuracy of matching feature points between the environmental map and the camera image decreases, and the vehicle position cannot be recognized accurately. Therefore, in order to address this problem, in this embodiment, the position estimation device is configured as follows.

[0031] Fig. 3 is a block diagram showing the configuration of the main parts of a position estimation device 50 according to this embodiment. This position estimation device 50 constitutes a part of the vehicle control system 100 shown in Fig. 1. As shown in Fig. 3, the position estimation device 50 includes a controller 10, a camera 1a, and an illuminance sensor 1b.

[0032] Camera 1a is a monocular camera having an imaging element (image sensor) such as a CCD or CMOS, and constitutes part of the external sensor group 1 in Fig. 1. Camera 1a is attached, for example, to a predetermined position in the front of the vehicle, continuously captures images of the space ahead of the vehicle at a predetermined frame rate, and sequentially outputs frame image data (camera images) as detection information to controller 10. Camera 1a may also be a stereo camera.

[0033] Illuminance sensor 1b has a light-receiving element and detects the brightness (illuminance) of light incident on the light-receiving element. Illuminance sensor 1b is installed outside (for example, on the roof) or inside (on the dashboard) of the vehicle so that it can detect the illuminance (brightness) around the vehicle. Illuminance sensor 1b outputs a detected value (detected data) to controller 10. Note that light passing through the windshield is attenuated to some extent by the glass, so if illuminance sensor 1b is installed inside the vehicle, the sensor value may be corrected taking this attenuation amount into account.

[0034] The controller 10 includes a calculation unit 11 and a storage unit 12. The calculation unit 11 has, as functional components, an information acquisition unit 111, a feature point extraction unit 112, a position estimation unit 113, a reliability determination unit 114, an environmental map generation unit 115, and a secondary position estimation unit 116. The storage unit 12 stores map information (environmental map) of roads on which the vehicle has traveled in the past.

[0035] The feature point extraction unit 112 and the environmental map generation unit 115 are included in, for example, the map generation unit 17 in Fig. 1. The position estimation unit 113, the reliability determination unit 114, and the auxiliary position estimation unit 116 are included in, for example, the vehicle position recognition unit 13 in Fig. 1.

[0036] The information acquisition unit 111 acquires information about past and current weather, time, and ambient brightness for the road section on which the vehicle has traveled. More specifically, the information acquisition unit 111 acquires past or current weather information for the road section on which the vehicle has traveled, via the communication unit 7, from an external server (not shown) that provides past or current weather information. At this time, the information acquisition unit 111 acquires past weather information for the road section on which the vehicle has traveled from the external server based on time information stored in the storage unit 12 in association with the environmental map. The time information records information indicating the time (image capture time) when the camera image from which the feature point was extracted was acquired by the camera 1a for each feature point included in the environmental map.

[0037] Furthermore, the information acquisition unit 111 acquires detection data from the illuminance sensor 1b as current brightness information for the road section on which the vehicle has traveled. The information acquisition unit 111 acquires past brightness information for the road section on which the vehicle has traveled from the storage unit 12. The storage unit 12 stores the brightness information together with time information in association with the environmental map. The brightness information records, for each feature point included in the environmental map, detection data acquired by the illuminance sensor 1b at the time the camera image from which the feature point was extracted was captured.

[0038] The feature point extraction unit 112 extracts feature points from camera images acquired by the camera 1a while the host vehicle is traveling on a road. The position estimation unit 113 estimates the host vehicle position based on the feature points extracted by the feature point extraction unit 112 and an environmental map stored in the storage unit 12. The position estimation unit 113 estimates the current position of the host vehicle on the road on which the host vehicle is traveling based on the feature points extracted by the feature point extraction unit 112 from camera images acquired by the camera 1a while the host vehicle is traveling on the road and the environmental map stored in the storage unit 12. The environmental map stored in the storage unit 12 includes a group of feature points (three-dimensional point cloud data) extracted by the feature point extraction unit 112 when the host vehicle traveled on that road in the past.

[0039] The reliability determination unit 114 determines whether the reliability of the host vehicle position estimated by the position estimation unit 113 is less than a predetermined level. The reliability determination unit 114 determines that the reliability is less than the predetermined level when the difference between the number of feature points corresponding to a predetermined area ahead of the host vehicle in the traveling direction extracted from the current camera image by the feature point extraction unit 112 and the number of feature points corresponding to the predetermined area among the feature points included in the environmental map stored in the storage unit 12 is equal to or greater than a predetermined threshold. The predetermined threshold may be changed depending on the number of feature points corresponding to the predetermined area included in the environmental map stored in the storage unit 12. For example, the greater the number of feature points corresponding to the predetermined area included in the environmental map stored in the storage unit 12, the greater the value set for the predetermined threshold. The predetermined area is the current imaging range of the camera 1a.

[0040] The reliability determination unit 114 also compares the environmental conditions at the time when the environmental map stored in the storage unit 12 was generated (hereinafter referred to as the map generation time) with the current environmental conditions. The environmental conditions include at least one of the travel time, weather, and brightness around the host vehicle. The reliability determination unit 114 recognizes the time (time period) when the host vehicle previously traveled to the current location based on time information associated with point cloud data corresponding to a predetermined area included in the environmental map stored in the storage unit 12. At this time, the information acquisition unit 111 acquires, as past weather information, weather information corresponding to the time (time period) when the host vehicle previously traveled to the current location from an external server. The information acquisition unit 111 also acquires, as past brightness information, brightness information corresponding to the time (time period) when the host vehicle previously traveled to the current location from the storage unit 12. The comparison of the environmental conditions is performed every time a camera image is acquired by the camera 1a, i.e., at a predetermined interval based on the frame rate of the camera 1a. The comparison of the environmental conditions may be performed for each frame or every time a predetermined number of camera images are acquired. The comparison of the environmental conditions may also be performed at other times, for example, every time the vehicle travels a predetermined distance or a predetermined time. When the comparison result indicates that there is a predetermined difference between the environmental conditions, the reliability determination unit 114 determines that the reliability of the vehicle position estimated by the position estimation unit 113 is less than a predetermined level.

[0041] For example, if the weather at the time of map generation is clear and the current weather is bad (rain, fog, snow, etc.) that reduces visibility ahead of the vehicle, corresponding points may not be extracted from the camera image, as shown in the example of FIG. 2B. Also, if the map is generated during the day and the current time is a time period or brightness that makes it difficult to extract the edges of objects included in the image capture range of camera 1a from the camera image, corresponding points may not be extracted from the camera image. In such cases, the accuracy of the vehicle position estimated based on the feature points extracted from the camera image and the environmental map stored in memory unit 12 may be reduced. Therefore, when there is a difference between the environmental conditions at the time of map generation and the current environmental conditions as described above, the reliability of the vehicle position is determined to be below a predetermined level. An example of a time period in which it is difficult to extract object edges is nighttime. An example of brightness in which it is difficult to extract object edges is illuminance below a predetermined value.

[0042] In a backlit scene (hereinafter referred to as a backlit scene) in which sunlight or light from lighting (such as the headlights of an oncoming vehicle) enters the camera 1a from behind the subject, it becomes difficult to extract the edges of objects included in the imaging range of the camera 1a from the camera image. Therefore, even if the imaging environment at the time of map generation was not a backlit scene but the current imaging environment is a backlit scene, it may be determined that there is a predetermined difference between the environmental conditions at the time of map generation and the current environmental conditions. Whether the imaging environment is a backlit scene may be determined based on the brightness of the camera image (brightness of the entire image), or based on the position and attitude of the vehicle (camera 1a) and the positions of the sun and lighting, or by other methods.

[0043] When the reliability determination unit 114 determines that the reliability is less than a predetermined level, the environmental map generation unit 115 starts generating an environmental map using the feature points extracted by the feature point extraction unit 112. The environmental map generation unit 115 continues generating the environmental map while the reliability determination unit 114 determines that the reliability is less than a predetermined level. The secondary position estimation unit 116 estimates the position of the host vehicle on the environmental map based on the feature points extracted by the feature point extraction unit 112 and the environmental map generated by the environmental map generation unit 115. The generation of the environmental map by the environmental map generation unit 115 and the estimation of the host vehicle position by the secondary position estimation unit 116 are performed simultaneously according to an algorithm of SLAM technology. Note that the environmental map (three-dimensional point cloud data) generated by the environmental map generation unit 115 may be used not only to estimate the host vehicle position by the secondary position estimation unit 116 but also to update the environmental map stored in the memory unit 12.

[0044] The driving control unit 16 controls the actuators AC based on the vehicle position estimated by the position estimation unit 113 or the secondary position estimation unit 116. More specifically, the driving control unit 16 controls the actuators AC so that the vehicle travels along the target trajectory generated by the behavior plan generation unit 15 based on the vehicle position estimated by the position estimation unit 113 or the secondary position estimation unit 116.

[0045] While the reliability determination unit 114 determines that the reliability is equal to or greater than a predetermined level, the driving control unit 16 controls the actuators AC based on the vehicle position estimated by the position estimation unit 113. Thereafter, when the reliability determination unit 114 determines that the reliability is less than the predetermined level and the generation of an environmental map by the environmental map generation unit 115 and the estimation of the vehicle position by the secondary position estimation unit 116 start, the driving control unit 16 starts controlling the actuators AC based on the vehicle position estimated by the secondary position estimation unit 116.

[0046] On the other hand, when control of actuators AC is being executed based on the vehicle position estimated by the secondary position estimation unit 116, if the reliability determination unit 114 determines that the reliability is equal to or higher than a predetermined level, the driving control unit 16 resumes control of actuators AC based on the vehicle position estimated by the position estimation unit 113. At this time, generation of the environmental map by the environmental map generation unit 115 and estimation of the vehicle position by the secondary position estimation unit 116 are stopped.

[0047] Fig. 4 is a flowchart showing an example of processing executed by the CPU of the controller 10 in Fig. 3 in accordance with a predetermined program. The processing shown in this flowchart is executed at a predetermined interval while the host vehicle is traveling in autonomous driving mode, for example.

[0048] First, in step S1, controller 10 acquires a camera image from camera 1a. In step S2, controller 10 extracts feature points from the camera image using feature point extraction unit 112. In step S3, controller 10 determines whether there is a difference between the number of feature points extracted from the camera image in step S2 and the number of feature points in point cloud data (hereinafter referred to as target point cloud data) corresponding to a predetermined area (imaging range of camera 1a) included in the environmental map stored in storage unit 12. Specifically, controller 10 determines whether the difference in the number of feature points between them is equal to or greater than a predetermined threshold.

[0049] If the result in step S3 is negative, it is determined in step S4 whether there is a difference in the external environment. Specifically, it is determined whether there is a predetermined difference between the environmental conditions at the time when the target point cloud data was generated (at the time when the map was generated) and the environmental conditions at the time when the camera image was acquired in step S1, i.e., the current environmental conditions.

[0050] If the result in step S4 is negative, then in step S5, the vehicle's position on the environmental map is estimated based on the feature points extracted in step S2 and the environmental map stored in memory unit 12. On the other hand, if the result in step S3 or step S4 is positive, then in step S61, an environmental map is generated using the feature points extracted in step S2. In step S62, the vehicle's position on the environmental map is estimated based on the feature points extracted in step S2 and the environmental map generated in step S61. The generation of the environmental map in step S61 and the estimation of the vehicle's position in step S62 are performed simultaneously according to the algorithm of SLAM technology.

[0051] The operation of the position estimation device 50 according to this embodiment can be summarized as follows. First, when a camera image is acquired by the camera 1a while the vehicle is traveling on a road (S1), feature points are extracted from the camera image (S2). Then, it is determined whether there is a difference between the number of feature points extracted from the camera image and the number of feature points in the target point cloud data (point cloud data corresponding to the imaging range of the camera 1a) included in the environmental map stored in the storage unit 12 (S3). If there is no difference between the numbers of feature points, it is further determined whether there is a difference between the external environment at the time of map generation and the current external environment (S4). If there is no difference between the external environments, it is determined that the reliability of the vehicle position estimated based on the feature points extracted from the camera image and the environmental map stored in the storage unit 12 is equal to or higher than a predetermined level. Then, the vehicle position is estimated based on the feature points extracted from the camera image and the environmental map stored in the storage unit 12 until it is determined that the reliability is less than the predetermined level (S1 to S5).

[0052] Subsequently, if the weather worsens and feature points (corresponding points) corresponding to the target point cloud data cannot be extracted from the camera image, resulting in a discrepancy between the number of feature points extracted from the camera image and the number of feature points in the target point cloud data, the reliability is determined to be below a predetermined level (S1, S3). If the weather worsens and a discrepancy occurs between the external environment at the time of map generation and the current external environment, the reliability is determined to be below a predetermined level even if there is no discrepancy between the number of feature points extracted from the camera image and the number of feature points in the target point cloud data (S1, S3, S4). If it is determined that the reliability is below a predetermined level, estimation of the vehicle position based on the feature points extracted from the camera image and the environmental map stored in memory unit 12 is stopped, and generation of an environmental map using the feature points extracted from the camera image and estimation of the vehicle position based on the environmental map are started (S61, S62). Thereafter, when the weather improves and feature points (corresponding points) corresponding to the target point cloud data can be extracted with high accuracy from the camera image, it is determined that the reliability has reached a predetermined level or higher, and estimation of the vehicle position based on the feature points extracted from the camera image and the environmental map stored in the memory unit 12 is resumed (S1 to S5).

[0053] According to the embodiment described above, the following effects can be obtained. (1) The position estimation device 50 includes a camera 1a as a detector for detecting the external situation of the vehicle; a feature point extraction unit 112 that extracts feature points from the detection data (camera image) of the camera 1a; a memory unit 12 that stores an environmental map as first map information including feature points corresponding to the external situation in advance; a position estimation unit 113 that estimates the position of the vehicle based on the feature points extracted by the feature point extraction unit 112 and the environmental map stored in the memory unit 12; a reliability determination unit 114 that determines whether the reliability of the position of the vehicle estimated by the position estimation unit 113 is less than a predetermined level; an environmental map generation unit 115 that generates an environmental map as second map information using the feature points extracted by the feature point extraction unit 112 when the reliability determination unit 114 determines that the reliability is less than the predetermined level; and a secondary position estimation unit 116 that estimates the position of the vehicle based on the feature points extracted by the feature point extraction unit 112 and the environmental map generated by the environmental map generation unit 115. The environmental map stored in the memory unit 12 is map information of a road section on which the host vehicle has previously traveled, and includes feature points extracted by the feature point extraction unit 112 from camera images captured by the camera 1a while the host vehicle was traveling on that road section. The position estimation unit 113 estimates the current position of the host vehicle traveling on that road section based on the feature points extracted by the feature point extraction unit 112 from the current camera image captured by the camera 1a while the host vehicle was traveling on that road section and the environmental map stored in the memory unit 12. This allows the host vehicle position to be accurately estimated continuously even when feature points cannot be accurately extracted from camera images due to, for example, worsening weather, or when the external environment at the time of map generation differs from the current external environment. Furthermore, since there is no need to prepare multiple environmental maps corresponding to different imaging conditions, an increase in the amount of data stored can be suppressed. Furthermore, this driving assistance technology can further improve traffic safety and convenience. Furthermore, it can contribute to the development of sustainable transportation systems.

[0054] (2) When the difference between the number of feature points corresponding to a predetermined area ahead of the vehicle in the traveling direction extracted by the feature point extraction unit 112 from the current camera image and the number of feature points corresponding to the predetermined area among the feature points included in the environmental map stored in the memory unit 12 is equal to or greater than a predetermined threshold, the reliability determination unit 114 determines that the reliability of the vehicle's position estimated by the position estimation unit 113 is less than a predetermined level. The greater the number of feature points corresponding to the predetermined area included in the environmental map stored in the memory unit 12, the larger the predetermined threshold value is set. This makes it possible to accurately recognize changes in the external environment that have occurred between the time the map was generated and the current time.

[0055] (3) The position estimation device 50 further includes an illuminance sensor 1b that detects the brightness around the vehicle, and an information acquisition unit that acquires past environmental information and current environmental information, including at least one of weather information from past and current travels on the road section, time information, and brightness information including data detected by the illuminance sensor 1b. The reliability determination unit 114 determines whether the reliability is below a predetermined level based on the past environmental information and current environmental information acquired by the information acquisition unit 111. This allows the vehicle position to be accurately estimated even when there is a risk of a decrease in the accuracy of matching feature points between the environmental map and the camera image due to factors such as worsening weather, glare caused by light sources entering the camera, or evaporation caused by headlights of oncoming vehicles.

[0056] (4) The vehicle control system 100 further includes a position estimation device 50, actuators AC for driving, and a driving control unit 16 that controls the actuators AC based on the position of the host vehicle estimated by the position estimation unit 113 or the secondary position estimation unit 116. When the driving control unit 16 is controlling the actuators AC based on the position of the host vehicle estimated by the position estimation unit 113, if the reliability determination unit 114 determines that the reliability is less than a predetermined level, the driving control unit 16 stops controlling the actuators AC based on the position of the host vehicle estimated by the position estimation unit 113 and starts controlling the actuators AC based on the position of the host vehicle estimated by the secondary position estimation unit 116. This allows the host vehicle to drive smoothly in autonomous driving mode. Furthermore, when the driving control unit 16 is controlling the actuators AC based on the position of the host vehicle estimated by the secondary position estimation unit 116, if the reliability determination unit 114 determines that the reliability is equal to or greater than a predetermined level, the driving control unit 16 stops controlling the actuators AC based on the position of the host vehicle estimated by the secondary position estimation unit 116 and resumes controlling the actuators AC based on the position of the host vehicle estimated by the position estimation unit 113. This allows the vehicle to run smoothly in autonomous driving mode.

[0057] The above embodiment can be modified in various ways. Modifications will be described below. In the above embodiment, when the reliability determination unit 114 determines that the reliability is less than a predetermined level while the vehicle is traveling on a road, the controller 10 stops the position estimation unit 113 from estimating the vehicle's position and starts the environmental map generation unit 115 to generate an environmental map and the secondary position estimation unit 116 to estimate the vehicle's position. However, the controller 10 may output a stop instruction to the position estimation unit 113 (as the first estimation unit) and the secondary position estimation unit 116 (as the second estimation unit) to stop estimating the vehicle's position when the number of times the reliability determination unit 114 determines that the reliability is less than the predetermined level exceeds a predetermined number. In this way, when the vehicle's position is repeatedly lost, the controller 10 (as the stop control unit) interrupts the estimation of the vehicle's position, thereby reducing the processing load on the position estimation device 50.

[0058] The stop control unit may output a stop instruction to the position estimation unit 113 and the auxiliary position estimation unit 116 based on the driving state of the host vehicle. In this case, the controller 10 also functions as a state acquisition unit that acquires vehicle state information indicating the state of the host vehicle. The stop control unit determines whether the host vehicle can continue driving based on the vehicle state information acquired by the state acquisition unit. If the stop control unit determines that the host vehicle cannot continue driving, it outputs a stop instruction to the position estimation unit 113 and the auxiliary position estimation unit 116. The vehicle state information includes information indicating whether or not a wheel (tire) is punctured, acceleration information indicating the degree of vehicle body shaking (vertical and lateral shaking), and the like. For example, if the stop control unit determines that a wheel is punctured based on the vehicle state information, it determines that the host vehicle cannot continue driving. Furthermore, if the vehicle state information (acceleration information) indicates that the acceleration of the vehicle body in the vertical or lateral direction is equal to or greater than a predetermined value, it determines that the road surface conditions have deteriorated and determines that the host vehicle cannot continue driving.

[0059] Depending on the degree of deterioration of the road surface conditions, it may be possible to continue driving. However, even if driving is possible, if the road surface conditions deteriorate, the position and attitude of the camera 1a may change due to the shaking of the vehicle body, and therefore, feature points (corresponding points) corresponding to the target point cloud data may not be extracted from the camera image. Therefore, the road surface conditions of the road on which the vehicle is traveling may be included in the environmental conditions. In this case, if the current road surface conditions have deteriorated and are different from the road surface conditions at the time of map generation, it is determined in step S4 that there is a difference between the environmental conditions at the time of map generation and the current environmental conditions.

[0060] In the above embodiment, as in the example of FIG. 2B , when there is a possibility that feature points corresponding to feature points included in the environmental map stored in the storage unit 12 are not extracted from the camera image, the reliability determination unit 114 determines that the reliability of the vehicle position estimated by the position estimation unit 113 is less than a predetermined level. However, when the weather at the time of map generation is bad weather such as rain and the current weather is sunny, or when the map generation is at night and the current time is daytime, there is a possibility that feature points corresponding to feature points extracted from the camera image are not present in the environmental map stored in the storage unit 12. In such cases, the accuracy of the vehicle position estimated by the position estimation unit 113 may be reduced. Therefore, even in such cases, it may be determined that there is a predetermined difference between the environmental conditions at the time of map generation and the current environmental conditions, and the reliability of the vehicle position estimated by the position estimation unit 113 may be determined to be less than a predetermined level.

[0061] In the above embodiment, it is determined whether there is a difference between the number of feature points extracted from the camera image and the number of feature points in the target point cloud data included in the environmental map stored in the storage unit 12 (S3), and then it is determined whether there is a difference between the external environment at the time of map generation and the current external environment (S4). However, the determination in step S3 may be performed after the determination in step S4. In the above embodiment, an example is shown in which, for each feature point included in the environmental map, information indicating the capture time of the camera image from which the feature point was extracted is recorded as the time information. However, each time the host vehicle travels a predetermined distance or a predetermined time, the capture time of the camera image captured at that time may be recorded as the time information. In the above embodiment, an example is shown in which, for each feature point included in the environmental map, detection data acquired by the illuminance sensor 1b at the capture time of the camera image from which the feature point was extracted is recorded as the brightness information. However, similar to the time information, each time the host vehicle travels a predetermined distance or a predetermined time, detection data acquired by the illuminance sensor 1b at that time may be recorded as the brightness information.

[0062] In the above embodiment, the camera 1a serving as the first detector detects the external environment of the vehicle. However, the first detector may be a device other than a camera, such as a radar or a lidar. In the above embodiment, the illuminance sensor 1b serving as the second detector detects the brightness of the surroundings of the vehicle. However, the camera 1a may also serve as the second detector and detect the brightness of the surroundings of the vehicle based on a camera image.

[0063] Furthermore, in the above embodiment, the position estimation device 50 is applied to an autonomous vehicle, but the position estimation device 50 can also be applied to vehicles other than autonomous vehicles. For example, the position estimation device 50 can also be applied to a manually driven vehicle equipped with an ADAS (Advanced Driver-Assistance Systems).

[0064] The above description is merely an example, and the present invention is not limited to the above-described embodiment and modifications as long as the features of the present invention are not impaired. One or more of the above-described embodiment and modifications can be arbitrarily combined, and modifications can also be combined with each other. [Explanation of symbols]

[0065] 1a camera, 1b illuminance sensor, 10 controller, 11 calculation unit, 12 memory unit, 16 driving control unit, 50 position estimation device, 111 information acquisition unit, 112 feature point extraction unit, 113 position estimation unit, 114 reliability determination unit, 115 environmental map generation unit, 116 sub-position estimation unit

Claims

1. a detector for detecting an external situation of the vehicle; a feature point extraction unit that extracts feature points from the detection data of the detector; a storage unit that stores in advance first map information including feature points corresponding to the external environment; a first estimation unit that estimates a position of the vehicle based on the feature points extracted by the feature point extraction unit and the first map information stored in the storage unit; a reliability determination unit that determines whether the reliability of the vehicle position estimated by the first estimation unit is less than a predetermined level; a map generation unit that generates second map information using the feature points extracted by the feature point extraction unit when the reliability determination unit determines that the reliability is less than the predetermined level; a second estimation unit that estimates a position of the vehicle based on the feature points extracted by the feature point extraction unit and the second map information generated by the map generation unit, the first map information is map information of a road section on which the vehicle has traveled in the past, and includes the feature points extracted by the feature point extraction unit from the detection data acquired by the detector while the vehicle was traveling on the road section; the first estimation unit estimates a current position of the vehicle traveling on the road section based on the feature points extracted by the feature point extraction unit from the current detection data acquired by the detector while the vehicle is traveling on the road section and based on the first map information; a reliability determination unit that determines that the reliability is less than the predetermined level when a difference between the number of feature points corresponding to a predetermined area ahead in the direction of travel of the vehicle, extracted from the current detection data by the feature point extraction unit, and the number of feature points corresponding to the predetermined area among the feature points included in the first map information stored in the memory unit, is equal to or greater than a predetermined threshold.

2. 2. The position estimation device according to claim 1, The position estimation device is characterized in that the predetermined threshold value is set to a larger value as the number of feature points corresponding to the predetermined area included in the first map information increases.

3. 2. The position estimation device according to claim 1, the detector is a first detector; a second detector that detects the brightness of the surroundings of the vehicle; an information acquisition unit that acquires past environmental information and current environmental information, including at least one of weather information, time information, and brightness information including detection data from the second detector when traveling on the road section in the past and at the current time; A position estimation device characterized in that the reliability determination unit determines whether the reliability is less than the predetermined level based on the past environmental information and the current environmental information acquired by the information acquisition unit.

4. 2. The position estimation device according to claim 1, a stop control unit that outputs a stop instruction to the first estimation unit and the second estimation unit to stop estimating the position of the vehicle when the number of times that the reliability determination unit determines that the reliability is less than the predetermined level exceeds a predetermined number while the vehicle is traveling on the road section.

5. 2. The position estimation device according to claim 1, a status acquisition unit that acquires a vehicle status of the vehicle; a stop control unit that outputs a stop instruction to the first estimation unit and the second estimation unit to stop estimating the position of the vehicle based on the vehicle state acquired by the state acquisition unit.

6. A position estimation device according to any one of claims 1 to 5; A traveling actuator; a travel control unit that controls the travel actuator based on the position of the vehicle estimated by the first estimation unit or the second estimation unit, a vehicle control system characterized in that, when the reliability determination unit determines that the reliability is less than the predetermined level while the driving control unit is controlling the driving actuator based on the vehicle position estimated by the first estimation unit, the driving control unit stops controlling the driving actuator based on the vehicle position estimated by the first estimation unit and starts controlling the driving actuator based on the vehicle position estimated by the second estimation unit.

7. 7. The vehicle control system according to claim 6, a vehicle control system characterized in that, when the driving control unit is controlling the driving actuators based on the vehicle position estimated by the second estimation unit and the reliability determination unit determines that the reliability is equal to or higher than the predetermined level, the driving control unit stops controlling the driving actuators based on the vehicle position estimated by the second estimation unit and resumes controlling the driving actuators based on the vehicle position estimated by the first estimation unit.

Images