Position estimation device and vehicle control system

Abstract

To accurately estimate a traveling position of an own vehicle.SOLUTION: A position estimation device 50 estimates a position of a vehicle based on first feature points of objects around the vehicle contained in a camera image of a camera 1a that detects a situation around the vehicle and second feature points of objects contained in map information. The position estimation device includes: a feature point extraction part 112 that extracts a first feature point from a camera image; a storage part 12 that stores type information indicating a type of an object corresponding to the second feature point together with map information; a type recognition part 113 that recognizes a type of the object corresponding to the first feature point based on the camera image; a search part 114 that searches for the second feature point corresponding to the first feature point from the map information based on the first feature point, a type of the object corresponding to the first feature point, and the type information stored in the storage part 12; and a position estimation part 115 that estimates a position of the own vehicle based on the first feature point and the second feature point searched for by the search part 114.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present invention relates to a position estimation device for estimating the position of a host vehicle and a vehicle control system.

Background Art

[0002] In recent years, a vehicle control system that contributes to improving traffic safety and the development of a sustainable transportation system has been desired. Conventionally, as this type of device, a device configured to detect the distance to an object existing around a moving body and match the detection data with map data to estimate the self-position of the moving body is known (see, for example, Patent Document 1). In the device described in Patent Document 1, by resetting a reference position for starting the estimation of the self-position when the deviation amount of the estimated self-position becomes equal to or greater than a predetermined threshold value, a decrease in estimation accuracy that may occur when the environment around the moving body is different from the map data is suppressed.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in a method of matching detection data with map data, as in the device described in Patent Document 1 above, when there are objects with similar shapes, such as road markings or crosswalks, around the moving body, mis-matching between the detection data and the map data may occur, and the estimation accuracy of the self-position may decrease.

Means for Solving the Problems

[0005] A position estimation device according to one aspect of the present invention is a position estimation device that estimates the position of a vehicle based on a first feature point of an object around the vehicle included in detection data of an in-vehicle detector that detects the situation around the vehicle and a second feature point of the object included in map information. The position estimation device includes a feature point extraction unit that extracts the first feature point from the detection data of the in-vehicle detector, a storage unit that stores type information indicating the type of the object corresponding to the second feature point together with the map information, a recognition unit that recognizes the type of the object corresponding to the first feature point extracted by the feature point extraction unit based on the detection data of the in-vehicle detector, a search unit that searches for the second feature point corresponding to the first feature point from the map information based on the first feature point extracted by the feature point extraction unit, the type of the object corresponding to the first feature point recognized by the recognition unit, and the type information stored in the storage unit, and a position estimation unit that estimates the position of the vehicle based on the first feature point extracted by the feature point extraction unit and the second feature point searched by the search unit.

[0006] A vehicle control system according to another aspect of the present invention includes the above-described position estimation device, a traveling actuator, and a traveling control unit that controls the traveling actuator based on the position of the vehicle estimated by the position estimation unit.

Advantages of the Invention

[0007] According to the present invention, the traveling position of the host vehicle can be accurately estimated.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3A

Figure 3B

Figure 4A

Figure 4B

Figure 5

Figure 6

Figure 7

Figure 8A

Figure 8B

Figure 9

Embodiments for Carrying Out the Invention

[0009] Hereinafter, embodiments of the invention will be described with reference to the drawings. The position estimation device according to the embodiment of the present invention can be applied to a vehicle having an automatic driving function, that is, an autonomous vehicle. Note that the vehicle to which the position estimation device according to the present embodiment is applied may be referred to as the host vehicle to distinguish it from other vehicles. The host vehicle may be any of an engine vehicle having an internal combustion engine (engine) as a driving source for traveling, an electric vehicle having a driving motor as a driving source for traveling, and a hybrid vehicle having an engine and a driving motor as driving sources for traveling. The host vehicle can travel not only in an automatic driving mode that does not require a driving operation by a driver but also in a manual driving mode by a driving operation of the driver.

[0010] First, the general configuration of the host vehicle related to autonomous driving will be described. FIG. 1 is a block diagram schematically showing the overall configuration of a vehicle control system 100 of a host vehicle having a position estimation device according to the present embodiment. As shown in FIG. 1, the vehicle control system 100 mainly includes a controller 10, an external sensor group 1, an internal sensor group 2, an input / output device 3, a positioning unit 4, a map database 5, a navigation device 6, a communication unit 7, and a traveling actuator AC, which are communicably connected to the controller 10 respectively.

[0011] The external sensor group 1 is a general term for a plurality of sensors (external sensors) that detect the external situation, which is the surrounding information of the host vehicle. For example, the external sensor group 1 includes a lidar that measures the reflected light of the omnidirectional irradiation light of the host vehicle to measure the distance from the host vehicle to surrounding obstacles, a radar that irradiates electromagnetic waves and detects reflected waves to detect other vehicles and obstacles around the host vehicle, and a camera mounted on the host vehicle and having an image sensor (image sensor) such as a CCD or a CMOS to image the surrounding (front, rear, and side) of the host vehicle.

[0012] The internal sensor group 2 is a general term for a plurality of sensors (internal sensors) that detect the traveling state of the host vehicle. For example, the internal sensor group 2 includes an inertial measurement unit (IMU) that detects the rotational angular velocity around three axes and the acceleration in three axial directions of the vertical direction, the front-rear direction (traveling direction), and the left-right direction (vehicle width direction) of the center of gravity of the host vehicle. Sensors that detect the driving operations of the driver in the manual driving mode, such as the operations of the accelerator pedal, the brake pedal, and the steering wheel, are also included in the internal sensor group 2.

[0013] The input / output device 3 is a general term for devices through which commands are input from the driver or information is output to the driver. For example, the input / output device 3 includes various switches through which the driver inputs various commands by operating operation members, a microphone through which the driver inputs commands by voice, a display that provides information to the driver via a display image, a speaker that provides information to the driver by voice, and the like.

[0014] The positioning unit (GNSS unit) 4 has a positioning sensor that receives positioning signals transmitted from positioning satellites. The positioning satellites are artificial satellites such as GPS satellites and quasi-zenith satellites. The positioning unit 4 measures the current position (latitude, longitude, altitude) of the host vehicle using the positioning information received by the positioning sensor.

[0015] The map database 5 is a device that stores general map information used in the navigation device 6, and is composed of, for example, a hard disk or a semiconductor element. The map information includes road position information, road shape (curvature, etc.) information, and intersection and branch point position information. Note that the map information stored in the map database 5 is different from the high-precision map information stored in the storage unit 12 of the controller 10.

[0016] The navigation device 6 is a device that searches for a target route on the road to a destination input by the driver and provides guidance along the target route. The input of the destination and the guidance along the target route are performed via the input / output device 3. The target route is calculated based on the current position of the host vehicle measured by the positioning unit 4 and the map information stored in the map database 5. It is also possible to measure the current position of the host vehicle using the detection values of the external sensor group 1, and calculate the target route based on this current position and the high-precision map information stored in the storage unit 12.

[0017] The communication unit 7 communicates with various servers (not shown) via a network including a wireless communication network typified by the Internet or a mobile phone network, and periodically or at an arbitrary timing acquires map information, driving history information, traffic information, etc. from the server. Not only acquire the driving history information, but also transmit the driving history information of the host vehicle to the server via the communication unit 7. The network includes not only a public wireless communication network but also a closed communication network provided for each predetermined management area, such as a wireless LAN, Wi-Fi (registered trademark), Bluetooth (registered trademark), etc. The acquired map information is output to the map database 5 and the storage unit 12, and the map information is updated.

[0018] The actuator AC is a driving actuator for controlling the running of the host vehicle. When the driving power source is an engine, the actuator AC includes a throttle actuator for adjusting the opening degree of the throttle valve of the engine (throttle opening degree). When the driving power source is a driving motor, the driving motor is included in the actuator AC. The actuator AC also includes a brake actuator for operating the braking device of the host vehicle and a steering actuator for driving the steering device.

[0019] The controller 10 is constituted by an electronic control unit (ECU). More specifically, the controller 10 includes a computer having an arithmetic unit 11 such as a CPU (microprocessor), a storage unit 12 such as a ROM and a RAM, and other peripheral circuits (not shown) such as an I / O interface. Although a plurality of ECUs with different functions such as an engine control ECU, a driving motor control ECU, and a braking device ECU can be provided separately, in FIG. 1, for the sake of convenience, the controller 10 is shown as a collection of these ECUs.

[0020] The storage unit 12 stores highly accurate detailed map information (referred to as highly accurate map information). The highly accurate map information includes road position information, road shape (curvature, etc.) information, road gradient information, intersection and branch point position information, type and position information of road division lines such as white lines, number of lanes information, lane width and position information for each lane (information on the center position of the lane and the boundary line of the lane position), position information of landmarks (buildings, traffic lights, signs, etc.) as marks on the map, and road surface profile information such as road surface unevenness. In the embodiment, the center line, lane boundary line, outside lane line, etc. are collectively referred to as road division lines. The highly accurate map information stored in the storage unit 12 includes map information acquired from outside the host vehicle via the communication unit 7 (referred to as external map information) and a map created by the host vehicle itself using the detection values of the external sensor group 1 or the detection values of the external sensor group 1 and the internal sensor group 2 (referred to as internal map information).

[0021] The external map information is, for example, information on a map (referred to as a cloud map) obtained via a cloud server, and the internal map information is information on a map (referred to as an environmental map) composed of three-dimensional point cloud data generated by mapping using a technique such as SLAM (Simultaneous Localization and Mapping). The external map information is shared between the host vehicle and other vehicles, while the internal map information is unique map information of the host vehicle (for example, map information exclusively possessed by the host vehicle). For roads not traveled by the host vehicle, newly constructed roads, etc., an environmental map is created by the host vehicle itself. Note that the internal map information may be provided to a server device or other vehicles via the communication unit 7. The storage unit 12 stores, in addition to the above-described high-precision map information, information such as the travel trajectory information of the host vehicle, various control programs, and thresholds used in the programs.

[0022] The arithmetic unit 11 functionally includes a host vehicle position recognition unit 13, an external environment recognition unit 14, a behavior plan generation unit 15, a travel control unit 16, and a map generation unit 17.

[0023] The host vehicle position recognition unit 13 recognizes (which may also be referred to as estimating) the position of the host vehicle on the map (host vehicle position) based on the position information of the host vehicle obtained by the positioning unit 4 and the map information in the map database 5. The host vehicle position may be recognized (estimated) using the high-precision map information stored in the storage unit 12 and the peripheral information of the host vehicle detected by the external sensor group 1, whereby the host vehicle position can be recognized with high precision. The movement information (movement direction, movement distance) of the host vehicle can also be calculated based on the detection values of the internal sensor group 2, and thereby the host vehicle position can be recognized. When the host vehicle position can be measured by a sensor installed outside the road or beside the road, the host vehicle position can also be recognized by communicating with the sensor via the communication unit 7.

[0024] The external situation recognition unit 14 recognizes the external situation around the host vehicle based on signals from an external sensor group 1 such as a lidar, radar, camera, etc. For example, it recognizes the positions, speeds, and accelerations of surrounding vehicles (front vehicles and rear vehicles) traveling around the host vehicle, the positions of surrounding vehicles parked or stopped around the host vehicle, and the positions and states of other objects. Other objects include signs, traffic lights, markings such as road lane lines and stop lines, buildings, guardrails, utility poles, billboards, pedestrians, bicycles, etc. The states of other objects include the colors of traffic lights (red, blue, yellow), the moving speeds and directions of pedestrians and bicycles, etc. A part of the stationary objects among other objects constitutes landmarks that are indicators of positions on the map, and the external situation recognition unit 14 also recognizes the positions and types of landmarks.

[0025] The action plan generation unit 15 generates a driving trajectory (target trajectory) of the host vehicle from the current time to a predetermined time in the future based on, for example, the target route calculated by the navigation device 6, the high-precision map information stored in the storage unit 12, the host vehicle position recognized by the host vehicle position recognition unit 13, and the external situation recognized by the external situation recognition unit 14. When there are a plurality of trajectories that are candidates for the target trajectory on the target route, the action plan generation unit 15 selects the optimal trajectory that complies with the laws and regulations and meets criteria such as efficient and safe driving from among them, and sets the selected trajectory as the target trajectory. Then, the action plan generation unit 15 generates an action plan corresponding to the generated target trajectory. The action plan generation unit 15 generates various action plans corresponding to passing driving for overtaking a preceding vehicle, lane change driving for changing the driving lane, following driving for following a preceding vehicle, lane keep driving for maintaining the lane so as not to deviate from the driving lane, deceleration driving, or acceleration driving, etc. When generating the target trajectory, the action plan generation unit 15 first determines the driving mode and generates the target trajectory based on the driving mode.

[0026] In the automatic driving mode, the travel control unit 16 controls each actuator AC so that the host vehicle travels along the target trajectory generated by the action plan generation unit 15. More specifically, in the automatic driving mode, the travel control unit 16 calculates a required driving force for obtaining the target acceleration per unit time calculated by the action plan generation unit 15 in consideration of the running resistance determined by the road gradient or the like. Then, for example, the actuator AC is feedback-controlled so that the actual acceleration detected by the internal sensor group 2 becomes the target acceleration. That is, the actuator AC is controlled so that the host vehicle travels at the target vehicle speed and the target acceleration. In the manual driving mode, the travel control unit 16 controls each actuator AC in response to a travel command (such as a steering operation) from the driver acquired by the internal sensor group 2.

[0027] While traveling in the manual driving mode, the map generation unit 17 generates an environmental map around the road on which the host vehicle has traveled as internal map information using the detection values detected by the external sensor group 1. For example, from a plurality of frames of camera images acquired by a camera, an edge or a characteristic region (blob) indicating the contour of an object is extracted based on the luminance and color information of each pixel, and feature points are extracted using the information of the edge and the blob. The feature points are, for example, intersections of edges and correspond to corners of buildings, corners of road signs, and the like. The map generation unit 17 calculates the three-dimensional position of the feature points while estimating the position and orientation of the camera so that the same feature point converges to one point among a plurality of frames of camera images according to the algorithm of the SLAM technique. By performing this calculation process for each of a plurality of feature points, an environmental map composed of three-dimensional point cloud data is generated. Note that, instead of the camera, data acquired by a radar or a lidar may be used to extract feature points of the objects around the host vehicle and generate an environmental map.

[0028] The own-vehicle position recognition unit 13 may perform the position recognition process of the own vehicle based on the environmental map generated by the map generation unit 17 and the feature points extracted from the camera image. Further, the own-vehicle position recognition unit 13 may perform the position recognition process of the own vehicle in parallel with the map creation process by the map generation unit 17. The map creation process and the position recognition (estimation) process are performed simultaneously according to the algorithm of the SLAM technology. The map generation unit 17 can generate the environmental map in the same manner not only when traveling in the manual driving mode but also when traveling in the automatic driving mode. When the environmental map has already been generated and stored in the storage unit 12, the map generation unit 17 may update the environmental map based on the feature points newly extracted from the newly acquired camera image (which may be called new feature points).

[0029] By the way, when the own-vehicle position recognition unit 13 recognizes (estimates) the position of the own vehicle based on the environmental map and the feature points extracted from the camera image, it searches for the feature points corresponding to the feature points extracted from the camera image from the environmental map (3D point cloud data). Then, the own-vehicle position recognition unit 13 solves the PNP (Perspective-n-point) problem based on the correspondence between the feature points extracted from the camera image and the searched feature points, thereby estimating the position and orientation of the camera (own vehicle). Note that the method for estimating the position and orientation of the camera is not limited to this, and other methods such as SfM (Shape from Motion) that restores the shape of an object from a plurality of camera images obtained from a moving camera may be used to estimate the position and orientation of the camera. The own-vehicle position recognition unit 13 further improves the estimation accuracy by adjusting the position and orientation of the own vehicle obtained by solving the PNP problem and the like by the bundle adjustment method using the camera images acquired from a plurality of viewpoints.

[0030] FIG. 2 is a diagram for explaining the association of feature points. In FIG. 2, the image IM schematically represents the camera image of the camera CA. The black circles fp11 to fp17 in the camera image IM schematically represent the feature points of the subject (object OB) extracted from the camera image. The black circles FP11 to FP17 schematically represent a part of the point cloud data included in the environmental map, specifically, the feature points constituting the point cloud corresponding to the object OB. The double-headed arrows in the figure schematically represent that the feature point fp11 extracted from the camera image is associated with the feature point FP11 among the feature points FP11 to FP17 on the environmental map. When the feature points in the camera image and the feature points on the environmental map are correctly associated as in the feature point fp11 in FIG. 2, the position and orientation of the host vehicle can be accurately estimated using PNP. Further, by using the bundle adjustment method using a plurality of frames of camera images in combination with PNP, a higher-precision position and orientation can be estimated.

[0031] FIG. 3A is a diagram showing an example of a camera image acquired by an in-vehicle camera while the host vehicle is traveling. FIG. 3A shows an example of a camera image in front of the host vehicle. FIG. 3B is a diagram showing an example of an environmental map corresponding to the imaging range of the camera image of FIG. 3A. As shown in FIG. 3A, various objects such as a traffic signal SG, a crosswalk CW, a lane line DL, and a stop line SL exist on and around the road on which the host vehicle travels. Among these objects, there are objects whose partial shapes are similar to each other. Therefore, when extracting feature points from the camera image of FIG. 3A and searching for the feature points corresponding to those feature points from the environmental map of FIG. 3B, so-called false matching may occur, in which feature points different from the feature points that should originally be searched for are acquired as search results. FIGS. 4A and 4B are diagrams showing examples of false matching. Since the shape of the corner of the lane line and the corner of the line of the crosswalk are similar, as in the example of FIG. 4A, the feature point fp21 of the lane line DL in the camera image may be erroneously associated with the feature point FP21 of the point cloud PC_ST (point cloud corresponding to the line ST of the crosswalk CW) on the environmental map. In addition, for feature points of a portion where the same shape continues, such as the edge of the lane line, multiple candidates for the corresponding feature points may be extracted from the environmental map. Even in such a case, there is a possibility of false matching occurring. The feature points FP22_1 to 22_3 in FIG. 4B schematically represent candidates for the feature points corresponding to the feature point fp22 of the edge of the lane line DL extracted from the environmental map. In the example of FIG. 4B, the feature point fp22 may be erroneously associated with other feature points (for example, the feature point FP22_1 or the feature point FP22_3) instead of the feature point (for example, the feature point FP22_2) that should originally be associated with the point cloud PC_DL (point cloud corresponding to the lane line DL) on the environmental map. In order to suppress such false matching, in the present embodiment, the position estimation device 50 is configured as follows.

[0032] FIG. 5 is a block diagram showing a main configuration of a position estimation device 50 according to the present embodiment. This position estimation device 50 constitutes a part of the vehicle control system 100 in FIG. 1. As shown in FIG. 5, the position estimation device 50 includes a controller 10 and a camera 1a.

[0033] Camera 1a is a monocular camera having an imaging device (image sensor) such as a CCD or a CMOS, and constitutes a part of the external sensor group 1 in FIG. 1. Camera 1a detects the situation around the host vehicle. Camera 1a is attached, for example, at a predetermined position in the front part of the host vehicle, continuously images the front space of the host vehicle at a predetermined frame rate, and sequentially outputs frame image data (camera image) as detection information to the sequencer 10. Note that camera 1a may be a stereo camera.

[0034] The controller 10 includes an arithmetic unit 11 and a storage unit 12. The arithmetic unit 11 functionally includes an information acquisition unit 111, a feature point extraction unit 112, a type recognition unit 113, a search unit 114, a position estimation unit 115, and a surrounding map generation unit 116. The storage unit 12 stores map information (surrounding map) of the road on which the host vehicle has traveled in the past. In addition, the storage unit 12 stores type information indicating the type of the corresponding object (lane line, stop line, road surface, traffic signal, etc.) for each feature point included in the surrounding map.

[0035] The surrounding map generation unit 116 is included, for example, in the map generation unit 17 in FIG. 1. The feature point extraction unit 112, the type recognition unit 113, the search unit 114, and the position estimation unit 115 are included, for example, in the host vehicle position recognition unit 13 in FIG. 1.

[0036] The information acquisition unit 111 acquires camera images from the camera 1a. The feature point extraction unit 112 extracts feature points from the camera images acquired by the information acquisition unit 111 while the host vehicle is traveling on a road. The type recognition unit 113 recognizes the type of object corresponding to the feature points extracted by the feature point extraction unit 112, based on the camera image from the camera 1a. Specifically, the type recognition unit 113 classifies the areas of the camera image by object type (e.g., lane markings, crosswalks, road surfaces, traffic lights, etc.) using a segmentation technique that utilizes machine learning or the like. The type recognition unit 113 then determines to which area each feature point extracted by the feature point extraction unit 112 belongs, thereby recognizing the type of object corresponding to each feature point. FIG. 6 is a diagram schematically illustrating a camera image (the camera image of FIG. 3A) divided into areas by object type. In FIG. 6, the shading applied to each area indicates the type of object corresponding to each area. Specifically, the area shaded with dots corresponds to the road surface, the area shaded with diagonal lines going down to the right corresponds to lane markings, the area shaded with horizontal stripes corresponds to crosswalks, the area shaded with diagonal lines going down to the left corresponds to stop lines, and the area shaded with diagonal grid corresponds to traffic lights.

[0037] The search unit 114 searches the environmental map for feature points (hereinafter referred to as corresponding feature points) that correspond to feature points (hereinafter referred to as extracted feature points) extracted by the feature point extraction unit 112. At this time, if the type of the extracted feature point recognized by the type recognition unit 113 differs from the type of the corresponding feature point indicated by the type information stored in the storage unit 12, the search unit 114 excludes the pair of the extracted feature point and the corresponding feature point from the search result. For example, as shown in FIG. 4A , if the type of the extracted feature point fp21 (a lane marking) differs from the type of the corresponding feature point FP21 (a stop line), the pair of the extracted feature point fp21 and the corresponding feature point FP21 is excluded from the search result.

[0038] The position estimation unit 115 estimates the position and orientation of the camera 1a based on the feature points (extracted feature points) extracted by the feature point extraction unit 112 and the corresponding feature points searched by the search unit 114. Note that the position estimation unit 115 does not use the pairs of corresponding feature points and extracted feature points excluded from the search results by the search unit 114 for estimating the position and orientation of the camera 1a.

[0039] Here, the processing of the position estimation unit 115 will be described. The position estimation unit 115 solves the PNP problem based on the correspondence between the two-dimensional coordinates of the extracted feature points (position coordinates on the camera image) extracted by the feature point extraction unit 112 and the three-dimensional coordinates of the corresponding feature points (position coordinates on the environmental map) searched by the search unit 114, thereby estimating the position and orientation of the camera 1a. Solving the PNP problem means calculating the position and orientation of the camera 1a such that the error between the two-dimensional coordinates of the extracted feature points and the two-dimensional coordinates obtained by projecting the three-dimensional coordinates of the corresponding feature points onto the camera image is minimized. Note that the method for estimating the position and orientation of the camera 1a is not limited to this, and the position estimation unit 115 may estimate the position and orientation of the camera 1a using other methods such as SfM. The estimated values of the position and orientation of the camera 1a obtained by solving the PNP problem or the like are referred to as initial estimated values. Since the camera 1a is attached to the host vehicle as described above, the position and orientation of the camera 1a are equivalent to the position and orientation of the host vehicle. Therefore, hereinafter, the position and orientation of the camera 1a may be expressed as the position and orientation of the host vehicle, or simply the self-position.

[0040] FIG. 7 is a diagram for explaining the estimation error of the self-position. FIG. 7 shows an example of a camera image in which feature points on the environmental map are projected. The black circles in FIG. 7 schematically represent the feature points on the environmental map (specifically, the feature points corresponding to the edges on the right side (the right side in the figure) of the traveling direction of the division line DL) projected onto the camera image based on the initial estimated values of the position and orientation of the host vehicle. When the feature points on the environmental map corresponding to the edges of the division line DL are projected onto the camera image based on the position and orientation of the host vehicle, those feature points are projected onto or near the edges of the division line DL in the camera image. However, since the initial estimated value of the self-position obtained by solving the PNP problem or the like includes an error, as shown in FIG. 7, the projection position may deviate from the edge of the division line DL. Therefore, in order to reduce the estimation error of the self-position as described above, the position estimation unit 115 performs bundle adjustment using a plurality of camera images with different viewpoints. Note that the position estimation unit 115 performs the above bundle adjustment by adding a predetermined constraint condition so as to further reduce the estimation error of the self-position. The predetermined constraint condition is defined such that when the corresponding feature points included in the environmental map are projected onto the camera image of camera 1a based on the self-position estimated by the position estimation unit 115, the perpendicular distance (hereinafter referred to as the projection error) between the projected corresponding feature points and the object corresponding to the corresponding feature points is minimized on the camera image. FIGS. 8A and 8B are diagrams for explaining the constraint conditions added to the bundle adjustment. In FIG. 8A, for the sake of simplification of the drawing, only four feature points fp22_1 to fp22_4 are shown as the feature points corresponding to the edges of the division line DL. In the case of FIG. 8A, the predetermined constraint condition is represented by the following formula (i). d_n represents the perpendicular distance from the feature point fp22_n (n = 1, 2, 3,...) to the edge of the division line DL.

[0041] minΣ(d_n)^2 ···(i)

[0042] By performing the bundle adjustment with the above-described constraint conditions added for each edge of the partition line DL, as shown in FIG. 8B, the partition line DL on the camera image and the point cloud PC_DL on the environmental map corresponding to the partition line DL are matched as a plane. As a result, even when using the point cloud data of a flat object, such as a partition line, for which it is difficult to associate the feature points in the camera image with the feature points on the environmental map for self-position estimation, the self-position can be accurately estimated without causing the mis-matching as shown in FIG. 4B.

[0043] When the information (point cloud data) regarding the road on which the host vehicle is traveling is not included in the environmental map stored in the storage unit 12, the environmental map generation unit 116 generates an environmental map corresponding to that road. Specifically, when the host vehicle is traveling on a road for which the environmental map has not been generated (a road not traveled by the host vehicle or a newly constructed road), the environmental map generation unit 116 generates point cloud data corresponding to that road based on the feature points (extracted feature points) extracted by the feature point extraction unit 112, and adds the generated point cloud data to the environmental map. At this time, the environmental map generation unit 116 stores, in the storage unit 12, in association with each feature point, the recognition result (type information) of the type of the object for each feature point obtained by the type recognition unit 113. On the other hand, when the information regarding the road on which the host vehicle is traveling is included in the environmental map, that is, when the host vehicle is traveling on a road for which the environmental map has already been generated, the environmental map generation unit 116 updates the environmental map stored in the storage unit 12 based on the feature points (extracted feature points) extracted by the feature point extraction unit 112. Further, the environmental map generation unit 116 updates the type information stored in the storage unit 12 based on the recognition result of the type of the object for each feature point obtained by the type recognition unit 113.

[0044] FIG. 9 is a flowchart showing an example of the processing executed by the CPU of the controller 10 in FIG. 5 according to a predetermined program. The processing shown in this flowchart is executed, for example, at a predetermined cycle while the host vehicle is traveling in the automatic driving mode.

[0045] First, in step S1, the controller 10 acquires a camera image from the camera 1a. In step S2, the controller 10 extracts feature points from the camera image acquired in step S1. In step S31, the controller 10 performs segmentation (region division) on the camera image acquired in step S1. Specifically, the camera image is divided into regions for each type of object. In step S32, the controller 10 matches each feature point (extracted feature point) extracted in step S2 with a feature point on the environmental map. Specifically, the feature point (corresponding feature point) corresponding to each extracted feature point is searched for from the environmental map stored in the storage unit 12. In step S33, the controller 10 determines which region among the regions obtained by dividing the camera image in step S31 each extracted feature point belongs to, and based on the determination result, recognizes the type of object associated with each extracted feature point. Then, for each extracted feature point for which a corresponding feature point is found in the matching in step S32, that is, for each pair of the extracted feature point and the corresponding feature point, the controller 10 compares the type of object associated with the extracted feature point with the type of object associated with the corresponding feature point. As a result of the comparison, pairs with different object types are excluded from the matching result (search result).

[0046] In step S34, the controller 10 estimates its own position based on each extracted feature point and the corresponding feature point corresponding to each extracted feature point. More specifically, first, the controller 10 calculates an initial estimated value of its own position by solving the PNP problem or the like based on the correspondence between the two-dimensional coordinates of the extracted feature point (position coordinates on the camera image) and the three-dimensional coordinates of the corresponding feature point (position coordinates on the environmental map). Next, the controller 10 performs bundle adjustment using a plurality of camera images with different viewpoints, adding the constraint conditions defined by the above formula (i). Thereby, the error included in the initial estimated value is minimized, and the final estimated value of the own position is calculated.

[0047] Also, in parallel with the processes of steps S31 to S34, the controller 10 executes the processes of steps S41 to S42. In step S41, based on the feature points (extracted feature points) extracted in step S2, the controller 10 generates a surrounding environment map corresponding to the road on which the host vehicle is traveling and stores it in the storage unit 12. In step S42, the recognition result of the object type for each of the extracted feature points obtained in step S33 is stored in the storage unit 12 as type information.

[0048] According to the embodiment described above, the following operational effects can be obtained. (1) The position estimation device 50 estimates the position of the host vehicle based on the feature points (first feature points) of the objects around the host vehicle included in the camera image of the camera 1a that detects the situation around the host vehicle and the feature points (second feature points) of the objects included in the surrounding environment map. The position estimation device 50 includes a feature point extraction unit 112 that extracts the feature points of the objects around the host vehicle from the camera image of the camera 1a, a storage unit 12 that stores, together with the surrounding environment map, type information indicating the type of the object corresponding to each feature point included in the surrounding environment map, a type recognition unit 113 as a recognition unit that recognizes the type of the object corresponding to the feature points (extracted feature points) extracted by the feature point extraction unit 112 based on the camera image of the camera 1a, a search unit 114 that searches for the feature points (corresponding feature points) corresponding to the extracted feature points from the surrounding environment map based on the extracted feature points, the type of the object corresponding to the extracted feature points recognized by the type recognition unit 113, and the type information stored in the storage unit 12, and a position estimation unit 115 that estimates the position of the host vehicle based on the extracted feature points and the corresponding feature points searched by the search unit 114. Thereby, the traveling position of the host vehicle can be accurately estimated.

[0049] (2) The position estimation unit 115 minimizes the error in the position of the host vehicle estimated based on the feature points (extracted feature points) extracted by the feature point extraction unit 112 and the feature points (corresponding feature points) searched by the search unit 114 by bundle adjustment. At this time, the position estimation unit 115 adds a predetermined constraint condition and executes bundle adjustment. The predetermined constraint condition is defined such that when the corresponding feature points included in the environmental map are projected onto the camera image of the camera 1a based on the position of the host vehicle estimated by the position estimation unit 115, the vertical distance between the corresponding feature points on the camera image and the object corresponding to the corresponding feature points becomes minimum. Thereby, even when the point cloud data of a flat object, such as a lane line, for which it is difficult to associate the feature points in the camera image with the feature points on the environmental map, is used for estimating the traveling position of the host vehicle, the traveling position of the host vehicle can be accurately estimated without causing false matching.

[0050] (3) The vehicle control system 100 further includes a position estimation device 50, an actuator AC for traveling, and a traveling control unit 16 that controls the actuator AC based on the position of the host vehicle estimated by the position estimation unit 115. Thereby, the host vehicle can travel well in the automatic driving mode.

[0051] The above embodiment can be modified into various forms. Hereinafter, modification examples will be described. (Modification Example 1)

[0052] In the above embodiment, the information acquisition unit 111 is configured to acquire the detection data (camera image) of the camera 1a as an in-vehicle detector. However, the in-vehicle detector may be other than a camera, such as a radar or a lidar, and the information acquisition unit may acquire the detection data of the radar or the lidar. (Modification Example 2)

[0053] Also, in the above embodiment, the controller 10 is configured to execute self-position estimation processing (S31 to S34) at a predetermined cycle while the host vehicle is traveling in the automatic driving mode. However, the controller 10 may further function as a reliability determination unit that determines whether the reliability of the position of the host vehicle estimated by the position estimation unit 115 is less than a predetermined level. The reliability determination unit determines that the reliability of the position of the host vehicle estimated by the position estimation unit 115 is less than a predetermined level when the difference between the number of feature points extracted by the feature point extraction unit 112 from the current camera image corresponding to a predetermined region (for example, the current imaging range of the camera 1a) in front of the traveling direction of the host vehicle and the number of feature points corresponding to the predetermined region among the feature points included in the environmental map stored in the storage unit 12 is equal to or greater than a predetermined threshold. Further, when the host vehicle is traveling on a road and the reliability determination unit determines that the reliability is less than a predetermined level, or when the number of times the reliability determination unit determines that the reliability is less than a predetermined level exceeds a predetermined number, the controller 10 may further function as a stop control unit that outputs a stop instruction to the position estimation unit 115 to stop the estimation of the position of the host vehicle. In this way, when the loss of the host vehicle position is continuous, the processing load of the position estimation device 50 can be reduced by interrupting the estimation of the host vehicle position.

[0054] In this modification example, the stop control unit may output a stop instruction to the position estimation unit 115 based on the driving state of the host vehicle. In this case, the controller 10 also functions as a state acquisition unit that acquires vehicle state information indicating the state of the host vehicle. The stop control unit determines whether the host vehicle can continue to drive based on the vehicle state information acquired by the state acquisition unit. When the stop control unit determines that it is impossible to continue driving, the stop control unit outputs a stop instruction to the position estimation unit 115. The vehicle state information includes information indicating the presence or absence of a puncture in the wheels (tires), acceleration information indicating the degree of shaking of the vehicle body (vertical and lateral shaking), and the like. For example, when the stop control unit determines that a wheel is punctured based on the vehicle state information, the stop control unit determines that it is impossible to continue driving. Further, when the acceleration in the vertical direction or the lateral direction of the vehicle body indicated by the vehicle state information (acceleration information) is equal to or greater than a predetermined value, it is determined that the road surface condition has deteriorated, and it is determined that it is impossible to continue driving. (Modification Example 3)

[0055] Incidentally, when the environmental map is generated, if the time zone, the brightness around the host vehicle, the weather (climate), and other conditions (hereinafter referred to as environmental conditions) are different from those when the camera image is acquired, the feature points (corresponding feature points) corresponding to the feature points (extracted feature points) extracted from the camera image may not exist on the environmental map, or the corresponding points of the feature points on the environmental map may not exist in the camera image. In this case, the matching accuracy of the feature points between the environmental map and the camera image decreases, and the position of the host vehicle cannot be accurately recognized. Therefore, in order to address such problems, the storage unit 12 may store a plurality of environmental maps respectively generated in different external environments, in association with environmental information indicating the external environment at the time of generating the environmental map. In this case, the environmental map generation unit 116 acquires information regarding the weather, time, and ambient brightness at the time of generating the environmental map. More specifically, the environmental map generation unit 116 acquires the weather information of the vicinity of the traveling position of the host vehicle from an external server (not shown) that provides weather information via the communication unit 7. Further, the environmental map generation unit 116 detects (acquires) the ambient brightness around the host vehicle based on the camera image of the camera 1a, and acquires the imaging time of the camera image. The environmental map generation unit 116 stores, as environmental information, the information indicating the weather, time, and brightness acquired at the time of generating the environmental map, together with the environmental map, in the storage unit 12. Similarly, the search unit 114 acquires information regarding the current external environment (weather, time, and ambient brightness). Based on the acquired information and the environmental information stored in the storage unit 12, the search unit 114 reads out from the storage unit 12 the environmental map corresponding to the current external environment, and searches for the feature points (corresponding feature points) corresponding to the feature points (extracted feature points) extracted from the camera image by the feature point extraction unit 112 from the read environmental map.

[0056] Furthermore, in the above embodiment, the position estimation device 50 is applied to the autonomous vehicle, but the position estimation device 50 can also be applied to vehicles other than the autonomous vehicle. For example, the position estimation device 50 can also be applied to a manually driven vehicle equipped with ADAS (Advanced driver-assistance systems).

[0057] The above description is merely an example, and the present invention is not limited by the above-described embodiments and modified examples as long as the features of the present invention are not impaired. It is also possible to arbitrarily combine one or more of the above-described embodiments and modified examples, and it is also possible to combine the modified examples with each other.

Explanation of Reference Numerals

[0058] 1a Camera, 10 Controller, 11 Arithmetic Unit, 12 Storage Unit, 16 Travel Control Unit, 50 Position Estimation Device, 111 Information Acquisition Unit, 112 Feature Point Extraction Unit, 113 Type Recognition Unit, 114 Search Unit, 115 Position Estimation Unit, 116 Surrounding Map Generation Unit

Claims

1. A position estimation device that estimates the position of a vehicle based on a first feature point of an object around the vehicle included in detection data of an in-vehicle detector that detects the situation around the vehicle and a second feature point of the object included in map information, comprising: A feature point extraction unit that extracts the first feature point from the detection data of the in-vehicle detector; A storage unit that stores type information indicating the type of the object corresponding to the second feature point together with the map information; A recognition unit that recognizes the type of the object corresponding to the first feature point extracted by the feature point extraction unit based on the detection data of the in-vehicle detector; A search unit that searches for the second feature point corresponding to the first feature point from the map information based on the first feature point extracted by the feature point extraction unit, the type of the object corresponding to the first feature point recognized by the recognition unit, and the type information stored in the storage unit; A position estimation unit that estimates the position of the vehicle based on the first feature point extracted by the feature point extraction unit and the second feature point searched by the search unit. The position estimation device is characterized by comprising the above components.

2. In the position estimation device according to Claim 1, the position estimation unit minimizes an error in the position of the vehicle estimated based on the first feature point extracted by the feature point extraction unit and the second feature point searched by the search unit by bundle adjustment. The position estimation device is characterized by this.

3. In the position estimation device according to Claim 2, the position estimation unit performs the bundle adjustment by adding a predetermined constraint condition, wherein the predetermined constraint condition is defined such that when the second feature point included in the map information is projected onto an image indicated by the detection data of the in-vehicle detector based on the position of the vehicle estimated by the position estimation unit, the vertical distance between the second feature point on the image and the object corresponding to the second feature point is minimized. The position estimation device is characterized by this.

4. In the position estimation device according to Claim 1, the storage unit stores a plurality of the map information respectively generated in different external environments in association with environment information representing the external environment at the time of map generation. The search unit reads out, from the storage unit, the map information corresponding to the external environment when the detection data of the in-vehicle detector is acquired based on the environment information, and searches for the second feature point corresponding to the first feature point extracted from the detection data by the feature point extraction unit from the read map information. A position estimation device characterized by the above.

5. In the position estimation device according to Claim 1, a reliability determination unit that determines whether or not the reliability of the position of the vehicle estimated by the position estimation unit is less than a predetermined level; a stop control unit that outputs a stop instruction to stop the estimation of the position of the vehicle when the number of times the reliability determination unit determines that the reliability is less than the predetermined level exceeds a predetermined number while the vehicle is traveling on a road included in the map information. A position estimation device characterized by further comprising:

6. In the position estimation device according to Claim 1, a state acquisition unit that acquires the vehicle state of the vehicle; a stop control unit that outputs a stop instruction to stop the estimation of the position of the vehicle to the position estimation unit based on the vehicle state acquired by the state acquisition unit. A position estimation device characterized by further comprising:

7. The position estimation device according to any one of Claims 1 to 6, a traveling actuator; A vehicle control system comprising: a traveling control unit that controls the traveling actuator based on the position of the vehicle estimated by the position estimation unit.

Images