Position estimation device and vehicle control system

Abstract

A position estimation device (50) estimates the position of a vehicle on the basis of a first feature point of an object around the vehicle included in detection data of an in-vehicle detector (1a) for detecting the situation around the vehicle and a second feature point of an object included in map information. A storage unit (12) that stores both map information and category information indicating the category of the object corresponding to the second feature point; a category recognition unit (113) that recognizes the category of the object corresponding to the first feature point on the basis of the detection data; a search unit (114) that searches for a second feature point (114) corresponding to the first feature point from map information according to the category of the object corresponding to the first feature point and the category information stored in the storage unit (12); and a position estimation unit (115) that estimates the position of the host vehicle on the basis of the first feature point and the second feature point searched by the search unit (114).

Description

Technical Field

[0001] The present invention relates to a position estimation device for estimating the position of the vehicle itself and a vehicle control system. Background Art

[0002] In recent years, there has been a demand for a vehicle control system that promotes traffic safety and contributes to the sustainable development of the transportation system. As such a device, there has been conventionally known a device that detects the distance to an object existing around a moving body and matches the detection data with map data to estimate the own position of the moving body (for example, refer to Patent Document 1). In the device described in Patent Document 1, when the deviation amount of the estimated own position reaches a predetermined threshold or more, the reference position for starting the estimation of the own position is reset, thereby suppressing a decrease in the estimation accuracy that may occur when the environment around the moving body is different from the map data.

[0003] However, in the method of matching the detection data with the map data as in the device described in Patent Document 1, when there are some objects with similar shapes such as road markings or crosswalks around the moving body, there may be a mis-match between the detection data and the map data, resulting in a decrease in the estimation accuracy of the own position.

[0004] Prior Art Documents

[0005] Patent Documents

[0006] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2021-176052 (JP2021-176052A). Summary of the Invention

[0007] A position estimation device according to one aspect of the present invention is a position estimation device that estimates the vehicle position based on a first feature point of an object around the vehicle included in detection data of an in-vehicle detector that detects the surrounding conditions of the vehicle and a second feature point of the object included in map information, and includes: a feature point extraction unit that extracts the first feature point from the detection data of the in-vehicle detector; a storage unit that stores the map information and category information indicating the category of the object corresponding to the second feature point together; an identification unit that identifies the category of the object corresponding to the first feature point extracted by the feature point extraction unit based on the detection data of the in-vehicle detector; a search unit that searches for the second feature point corresponding to the first feature point from the map information based on the first feature point extracted by the feature point extraction unit, the category of the object corresponding to the first feature point identified by the identification unit, and the category information stored in the storage unit; and a position estimation unit that estimates the vehicle position based on the first feature point extracted by the feature point extraction unit and the second feature point searched by the search unit.

[0008] Another technical solution of the vehicle control system of the present invention includes: the above-mentioned position estimation device, a traveling actuator, and a traveling control unit that controls the traveling actuator according to the position of the vehicle estimated by the position estimation unit. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The object, features, and advantages of the present invention will be further clarified by the following description of the embodiments related to the drawings.

[0010] Figure 1 is a block diagram schematically showing the overall configuration of the vehicle control system according to an embodiment of the present invention;

[0011] Figure 2 is a diagram for explaining the correspondence relationship of feature points;

[0012] Figure 3A is a diagram showing an example of a camera image obtained by an in-vehicle camera during the traveling of the vehicle;

[0013] Figure 3B shows Figure 3A an example of an environmental map corresponding to the shooting range of the camera image;

[0014] Figure 4A is a diagram showing an example of a false match;

[0015] Figure 4B is a diagram showing another example of a false match;

[0016] Figure 5 is a block diagram showing the main part configuration of the position estimation device according to the embodiment of the present invention;

[0017] Figure 6 is a diagram schematically showing a camera image in which regions are segmented by object category;

[0018] Figure 7 is a diagram for explaining the estimation error of the position and posture of the vehicle;

[0019] Figure 8A is a diagram for explaining the constraint conditions added to bundle adjustment;

[0020] Figure 8B is a diagram for explaining the constraint conditions added to bundle adjustment;

[0021] Figure 9 represents Figure 5 an example of a flowchart of the processing executed by the CPU of the controller; DETAILED DESCRIPTION OF THE EMBODIMENTS

[0022] Next, embodiments of the invention will be described with reference to the accompanying drawings. The position estimation device according to the embodiments of the present invention can be applied to a vehicle having an autonomous driving function, that is, it can be applied to an autonomous driving vehicle. It should be noted that the vehicle to which the position estimation device of this embodiment is applied is sometimes distinguished from other vehicles and referred to as the present vehicle. The present vehicle can be any one of an engine vehicle having an internal combustion engine (engine) as a driving source for traveling, an electric vehicle having a driving motor as a driving source for traveling, and a hybrid vehicle having an engine and a driving motor as driving sources for traveling. The present vehicle can not only travel in an autonomous driving mode without the driver performing driving operations, but also travel in a manual driving mode based on the driver's driving operations.

[0023] First, a schematic configuration of the present vehicle related to autonomous driving will be described. Figure 1 FIG. is a block diagram schematically showing the overall configuration of a vehicle control system 100 of the present vehicle having the position estimation device according to the present embodiment. As Figure 1 shown, the vehicle control system 100 mainly includes a controller 10, an external sensor group 1, an internal sensor group 2, an input / output device 3, a positioning unit 4, a map database 5, a navigation device 6, a communication unit 7, and a driving actuator AC, which are respectively communicably connected to the controller 10.

[0024] The external sensor group 1 is a general term for a plurality of sensors (external sensors) that detect the external situation as the surrounding information of the present vehicle. For example, the external sensor group 1 includes a lidar that measures the distance from the present vehicle to surrounding obstacles by irradiating light omnidirectionally to the present vehicle and measuring the reflected light, a radar that detects other vehicles and obstacles around the present vehicle by irradiating electromagnetic waves and detecting the reflected waves, a camera mounted on the present vehicle and having an imaging element (image sensor) such as a CCD (charge-coupled device) or a CMOS (complementary metal oxide semiconductor) to capture the surroundings (front, rear, and sides) of the present vehicle, and the like.

[0025] The internal sensor group 2 is a general term for a plurality of sensors (internal sensors) that detect the driving state of the present vehicle. For example, the internal sensor group 2 includes an inertial measurement unit (IMU), etc., which detects the rotational angular velocity and the acceleration in the three axial directions of the vertical direction, the front-rear direction (traveling direction), and the left-right direction (vehicle width direction) around the center of gravity of the present vehicle. Sensors that detect the driving operations of the driver in the manual driving mode, such as operations on the accelerator pedal, the brake pedal, and the steering wheel, are also included in the internal sensor group 2.

[0026] The input / output device 3 is a general term for devices that allow a driver to input commands or output information to the driver. For example, the input / output device 3 includes various switches that allow a driver to input various commands by operating an operation member, a microphone that allows a driver to input commands by voice, a display that provides information to the driver by displaying an image, a speaker that provides information to the driver by sound, and the like.

[0027] The positioning unit (GNSS unit) 4 has a positioning sensor that receives positioning signals transmitted from positioning satellites. The positioning satellites are artificial satellites such as GPS satellites and quasi-zenith satellites. The positioning unit 4 measures the current position (latitude, longitude, altitude) of the vehicle using the positioning information received by the positioning sensor.

[0028] The map database 5 is a device that stores general map information used by the navigation device 6, and is composed of, for example, a hard disk and semiconductor elements. The map information includes the position information of roads, information on the shape of roads (such as curvature), and the position information of intersections or branch points. It should be noted that the map information stored in the map database 5 is different from the high-precision map information stored in the storage unit 12 of the controller 10.

[0029] The navigation device 6 is a device that searches for a target route on the road to a destination input by the driver and guides along the target route. The input of the destination and the guidance along the target route are performed via the input / output device 3. The target route is calculated based on the current position of the vehicle measured by the positioning unit 4 and the map information stored in the map database 5. It is also possible to use the detection values of the external sensor group 1 to measure the current position of the vehicle, and the target route can also be calculated based on this current position and the high-precision map information stored in the storage unit 12.

[0030] The communication unit 7 communicates with various servers (not shown) via a network including a wireless communication network represented by the Internet, a mobile phone network, etc., and obtains map information, driving history information, traffic information, etc. from the server at regular intervals or at any time. Not only can driving history information be obtained, but the driving history information of the vehicle can also be sent to the server via the communication unit 7. The network includes not only a public wireless communication network but also a closed communication network set for each specified management area, such as a wireless LAN, Wi-Fi (registered trademark), Bluetooth (registered trademark), etc. The obtained map information is output to the map database 5 and the storage unit 12, and the map information is updated.

[0031] The actuator AC is a driving actuator for controlling the running of the vehicle. When the driving power source is an engine, the actuator AC includes a throttle actuator for adjusting the opening degree of the throttle valve (throttle opening degree) of the engine. When the driving power source is a driving motor, the driving motor is included in the actuator AC. The braking actuator for driving the braking device of the vehicle and the steering actuator for driving the steering device are also included in the actuator AC.

[0032] The controller 10 is composed of an electronic control unit (ECU). More specifically, the controller 10 includes a computer having an arithmetic unit 11 such as a CPU (microprocessor), a storage unit 12 such as a ROM (read-only memory) and a RAM (random access memory), and other peripheral circuits (not shown) such as an I / O interface. It should be noted that although multiple ECUs with different functions such as an engine control ECU, a driving motor control ECU, and a braking device ECU can be set separately, in Figure 1 this case, for convenience, the controller 10 is shown as a collection of these ECUs.

[0033] High-precision detailed map information (referred to as high-precision map information) is stored in the storage unit 12. The high-precision map information includes road position information, road shape (curvature, etc.) information, road slope information, intersection or branch point position information, category or position information of road markings such as white lines, lane number information, lane width and position information of each lane (information on the center position of the lane or the boundary line of the lane position), position information of landmarks (buildings, traffic lights, signs, etc.) as marks on the map, and road surface profile information such as road surface unevenness. In the embodiment, the center line, lane boundary line, lane outer line, etc. are collectively referred to as road markings. The high-precision map information stored in the storage unit 12 includes map information obtained from outside the vehicle via the communication unit 7 (referred to as external map information), and a map generated by the vehicle itself using the detection values of the external sensor group 1 or the detection values of the external sensor group 1 and the internal sensor group 2 (referred to as internal map information).

[0034] The external map information is, for example, information of a map (referred to as a cloud map) obtained through a cloud server, and the internal map information is information of a map (referred to as an environmental map) composed of three-dimensional point cloud data generated by mapping using technologies such as SLAM (Simultaneous Localization and Mapping). The external map information is shared between this vehicle and other vehicles, while the internal map information is the unique map information of this vehicle (for example, the map information exclusively owned by this vehicle). In roads where this vehicle has not traveled, newly established roads, etc., this vehicle generates the environmental map by itself. It should be noted that the internal map information can also be provided to the server device and other vehicles via the communication unit 7. In addition to the above-mentioned high-precision map information, the storage unit 12 stores information such as the driving trajectory information of this vehicle, programs for various controls, and thresholds used in the programs.

[0035] The arithmetic unit 11 has a vehicle position recognition unit 13, an external environment recognition unit 14, an action plan generation unit 15, a driving control unit 16, and a map generation unit 17 as functional structures.

[0036] The vehicle position recognition unit 13 recognizes (which can also be referred to as estimating) the position of this vehicle (the vehicle position) on the map based on the position information of this vehicle obtained by the positioning unit 4 and the map information of the map database 5. It is also possible to use the high-precision map information stored in the storage unit 12 and the surrounding information of this vehicle detected by the external sensor group 1 to recognize (estimate) the vehicle position, thereby enabling the vehicle position to be recognized with high precision. The movement information (movement direction, movement distance) of this vehicle is calculated based on the detection values of the internal sensor group 2, and thereby the vehicle position can also be recognized. It should be noted that when it is possible to measure the vehicle position using sensors externally installed on or beside the road, the vehicle position can also be recognized by communicating with the sensor via the communication unit 7.

[0037] The external environment recognition unit 14 recognizes the external conditions around this vehicle based on signals from the external sensor group 1 such as lidar, radar, and cameras. For example, it recognizes the positions, speeds, or accelerations of surrounding vehicles (front vehicles, rear vehicles) traveling around this vehicle, the positions of surrounding vehicles parked or stationary around this vehicle, and the positions or states of other objects. Other objects include signs, traffic lights, road markings or stop lines, buildings, guardrails, utility poles, signs, pedestrians, bicycles, etc. The states of other objects include the colors of traffic lights (red, blue, yellow), the moving speeds or directions of pedestrians or bicycles, etc. A part of the stationary objects among other objects constitutes landmarks that are indicators of positions on the map, and the external environment recognition unit 14 also recognizes the positions and categories of the landmarks.

[0038] The action plan generation unit 15 generates a driving track (target track) of the host vehicle after a specified time from the current time point, based on, for example, the target path calculated by the navigation device 6, the high-precision map information stored in the storage unit 12, the position of the host vehicle identified by the host vehicle position recognition unit 13, and the external conditions identified by the external recognition unit 14. When there are multiple tracks that are candidates for the target track on the target path, the action plan generation unit 15 selects the best track that meets criteria such as compliance with laws and regulations and efficient and safe driving from among them, and sets the selected track as the target track. Then, the action plan generation unit 15 generates an action plan corresponding to the generated target track. The action plan generation unit 15 generates various action plans corresponding to, for example, overtaking driving for overtaking a preceding vehicle, lane change driving for changing the driving lane, following driving for following a preceding vehicle, lane keeping driving for maintaining the lane without deviating from the driving lane, decelerating driving, or accelerating driving. When generating the target track, the action plan generation unit 15 first determines the driving mode and generates the target track according to the driving mode.

[0039] The driving control unit 16 controls each actuator AC in the autonomous driving mode so that the host vehicle travels along the target track generated by the action plan generation unit 15. More specifically, the driving control unit 16 considers the driving resistance determined by, for example, the road gradient in the autonomous driving mode, and calculates the required driving force for obtaining the target acceleration per unit time calculated by the action plan generation unit 15. And, for example, feedback control is performed on the actuator AC so that the actual acceleration detected by the internal sensor group 2 becomes the target acceleration. That is, the actuator AC is controlled so that the host vehicle travels at the target vehicle speed and the target acceleration. It should be noted that in the manual driving mode, the driving control unit 16 controls each actuator AC according to the driving instruction (steering operation, etc.) from the driver obtained by the internal sensor group 2.

[0040] While traveling in the manual driving mode, the map generation unit 17 generates an environmental map of the periphery of the road traveled by the host vehicle as internal map information, using the detection values detected by the external sensor group 1. For example, from multiple frames of camera images obtained by the camera, edges and feature regions (BLOBs) showing the outlines of objects are extracted based on the brightness and color information of each pixel, and feature points are extracted using the information of the edges and BLOBs. Feature points are, for example, intersections of edges, corresponding to corners of buildings, corners of road traffic signs, etc. The map generation unit 17 calculates the three-dimensional positions for the feature points while estimating the position and orientation of the camera, in accordance with the algorithm of the SLAM technology, so that the same feature point converges to one point among multiple frames of camera images. By performing this calculation process for each of a plurality of feature points, an environmental map composed of three-dimensional point cloud data is generated. Note that, instead of the camera, data obtained by a radar or lidar may be used to extract feature points of objects around the host vehicle to generate an environmental map.

[0041] The host vehicle position recognition unit 13 may also perform the host vehicle position recognition process based on the environmental map generated by the map generation unit 17 and the feature points extracted from the camera images. In addition, the host vehicle position recognition unit 13 may perform the host vehicle position recognition process in parallel with the map creation process of the map generation unit 17. The map generation process and the position recognition (estimation) process are performed simultaneously in accordance with the algorithm of the SLAM technology. The map generation unit 17 can generate an environmental map not only when traveling in the manual driving mode but also when traveling in the autonomous driving mode. In the case where an environmental map has already been generated and stored in the storage unit 12, the map generation unit 17 may update the environmental map based on the feature points newly extracted from newly obtained camera images (which may also be referred to as new feature points).

[0042] However, when the vehicle position recognition unit 13 recognizes (estimates) the position of the vehicle based on the environmental map and the feature points extracted from the camera image, it searches for the feature points corresponding to the feature points extracted from the camera image in the environmental map (three-dimensional point cloud data). Then, the vehicle position recognition unit 13 solves the PNP (Perspective-n-point) problem based on the correspondence between the feature points extracted from the camera image and the searched feature points, thereby estimating the position and posture of the camera (the vehicle). It should be noted that the method for estimating the position and posture of the camera is not limited to this, and other methods such as SfM (Shape from Motion) that restores the shape of an object from multiple camera images obtained from a self-moving camera can also be used to estimate the position and posture of the camera. The vehicle position recognition unit 13 also adjusts the position and posture of the vehicle obtained by solving the PNP problem, etc., by using bundle adjustment of camera images acquired from multiple viewpoints, thereby improving the estimation accuracy.

[0043] Figure 2 is a diagram for explaining the correspondence of feature points. In Figure 2 it, the image IM schematically shows the camera image of the camera CA. The black circles fp11 to fp17 in the camera image IM schematically show the feature points of the subject (object OB) extracted from the camera image. The black circles FP11 to FP17 schematically show a part of the point cloud data included in the environmental map, specifically, schematically show the feature points constituting the point cloud corresponding to the object OB. The double arrows in the figure schematically show that the feature point fp11 extracted from the camera image establishes a correspondence with the feature point FP11 among the feature points FP11 to FP17 on the environmental map. As Figure 2 shown by the feature point fp11, when the feature points in the camera image are correctly corresponded to the feature points on the environmental map, the position and posture of the vehicle can be estimated with high precision using PNP. In addition, by using bundle adjustment of multiple-frame camera images in combination with PNP, a more accurate position and posture can be further estimated.

[0044] Figure 3A is a diagram showing an example of a camera image acquired by an in-vehicle camera during the driving of the vehicle. In Figure 3A it, an example of a camera image in front of the vehicle is shown. Figure 3B is a diagram showing an example of the environmental map corresponding to the shooting range of the Figure 3A camera image. As Figure 3A shown, on the road where the vehicle is traveling or around the road, there are various objects such as a traffic signal SG, a crosswalk CW, road markings DL, a stop line SL, etc. Among these objects, there are some objects with similar shapes to each other. Therefore, in from Figure 3Aextracting feature points from the camera image of Figure 3B When searching for feature points corresponding to the feature points on the environmental map of Figure 4A and Figure 4B are diagrams showing examples of false matches. Since the shapes of the corners of road markings and the corners of crosswalk lines are similar, as in Figure 4A In the example of, the feature point fp21 of the road marking DL in the camera image may erroneously establish a correspondence with the feature point FP21 of the point cloud PC_ST (point cloud corresponding to the line ST of the crosswalk CW) on the environmental map. In addition, for feature points in continuous parts with the same shape such as the edges of road markings, sometimes multiple candidates for corresponding feature points are extracted from the environmental map. In such a case, false matches may also occur. Figure 4B The feature points FP22_1 to 22_3 of Figure 4B schematically represent candidates for feature points corresponding to the feature point fp22 of the edge of the road marking DL extracted from the environmental map. In

[0045] Figure 5 is a block diagram showing the main part structure of the position estimation device 50 of the present embodiment. The position estimation device 50 constitutes Figure 1 a part of the vehicle control system 100 of Figure 5 As shown in, the position estimation device 50 includes a controller 10 and a camera 1a.

[0046] The camera 1a is a monocular camera having a photographing element (image sensor) such as a CCD or a CMOS, and constitutes Figure 1 a part of the external sensor group 1 of

[0047] The controller 10 includes a computing unit 11 and a storage unit 12. The computing unit 11 includes an information acquisition unit 111, a feature point extraction unit 112, a category recognition unit 113, a search unit 114, a position estimation unit 115, and an environment map generation unit 116 as functional structures. The storage unit 12 stores map information (environmental map) of roads that the vehicle has traveled in the past. In addition, the storage unit 12 stores category information indicating the category of the corresponding object (road marking, stop line, road surface, signal, etc.) for each feature point included in the environment map.

[0048] The environment map generation unit 116 includes, for example, Figure 1 The feature point extraction unit 112, the category identification unit 113, the search unit 114, and the position estimation unit 115 are included in the map generation unit 17. Figure 1 In the vehicle position recognition unit 13.

[0049] The information acquisition unit 111 acquires a camera image from the camera 1a. The feature point extraction unit 112 extracts feature points from the camera image acquired by the information acquisition unit 111 while the vehicle is traveling on the road. The category recognition unit 113 recognizes the category of the object corresponding to the feature point extracted by the feature point extraction unit 112 based on the camera image of the camera 1a. Specifically, the category recognition unit 113 classifies the area of the camera image according to the category of the object (road marking, crosswalk, road surface, traffic light, etc.) using a segmentation technology that uses machine learning, etc. Then, the category recognition unit 113 determines to which area each feature point extracted by the feature point extraction unit 112 belongs, thereby identifying the category of the object corresponding to each feature point. Figure 6 is a schematic diagram showing a camera image segmented by object category ( Figure 3A The camera image of Figure 6 The shading applied to each area in the figure indicates the type of object corresponding to each area. Specifically, the area shaded with dots corresponds to the road surface, the area shaded with right-down diagonal lines corresponds to the road markings, the area shaded with horizontal stripes corresponds to the crosswalk, the area shaded with left-down diagonal lines corresponds to the stop line, and the area shaded with diagonal grids corresponds to the traffic light.

[0050] The search unit 114 searches the environment map for feature points (hereinafter referred to as corresponding feature points) corresponding to the feature points (hereinafter referred to as extracted feature points) extracted by the feature point extraction unit 112. At this time, when the category of the extracted feature points identified by the category identification unit 113 is different from the category of the corresponding feature points represented by the category information stored in the storage unit 12, the search unit 114 excludes the pair of the extracted feature points and the corresponding feature points from the search results. For example, Figure 4AAs shown, when the category of the extracted feature point fp21 (road marking) is different from the category of the corresponding feature point FP21 (stop line), the pair of the extracted feature point fp21 and the corresponding feature point FP21 is excluded from the search results.

[0051] The position estimation unit 115 estimates the position and orientation of the camera 1a based on the feature points (extracted feature points) extracted by the feature point extraction unit 112 and the corresponding feature points searched by the search unit 114. It should be noted that the position estimation unit 115 does not use the pairs of the corresponding feature points and the extracted feature points excluded from the search results by the search unit 114 for the estimation of the position and orientation of the camera 1a.

[0052] Here, the processing of the position estimation unit 115 will be described. The position estimation unit 115 solves the PNP problem based on the correspondence between the two-dimensional coordinates (position coordinates on the camera image) of the extracted feature points extracted by the feature point extraction unit 112 and the three-dimensional coordinates (position coordinates on the environmental map) of the corresponding feature points searched by the search unit 114, thereby estimating the position and orientation of the camera 1a. Solving the PNP problem is to calculate the position and orientation of the camera 1a that minimizes the error between the two-dimensional coordinates of the extracted feature points and the two-dimensional coordinates obtained by projecting the three-dimensional coordinates of the corresponding feature points onto the camera image. It should be noted that the method for estimating the position and orientation of the camera 1a is not limited to this, and the position estimation unit 115 can also use other methods such as SfM to estimate the position and orientation of the camera 1a. The estimated values of the position and orientation of the camera 1a obtained by solving the PNP problem etc. are referred to as initial estimated values. It should be noted that since the camera 1a is installed on the vehicle as described above, the position and orientation of the camera 1a are the same as the position and orientation of the vehicle. Therefore, hereinafter, the position and orientation of the camera 1a are sometimes expressed as the position and orientation of the vehicle, or simply as its own position.

[0053] Figure 7 is a diagram for explaining the estimation error of the own position. Figure 7 Shows an example of a camera image on which the feature points on the environmental map are projected. Figure 7 The black circles schematically represent the feature points on the environmental map (specifically, the feature points corresponding to the right edge (right side in the figure) of the traveling direction of the road marking DL) projected onto the camera image according to the initial estimated values of the position and orientation of the vehicle. When the feature points on the environmental map corresponding to the edge of the road marking DL are projected onto the camera image according to the position and orientation of the vehicle, these feature points are projected onto the edge of the road marking DL on the camera image or near it. However, since the initial estimated values of the own position obtained by solving the PNP problem etc. contain errors, as Figure 7As shown, the projection position may deviate from the edge of the road marking DL. Therefore, in order to reduce the estimation error of its own position as described above, the position estimation unit 115 performs bundle adjustment using a plurality of camera images with different viewpoints. It should be noted that the position estimation unit 115 performs the above-mentioned bundle adjustment with additional specified constraint conditions to further reduce the estimation error of its own position. The specified constraint condition is defined as follows: when the corresponding feature points included in the environmental map are projected onto the camera image of the camera 1a according to the own position estimated by the position estimation unit 115, the vertical distance (hereinafter referred to as the projection error) between the projected corresponding feature points and the object corresponding to the corresponding feature points is minimized on the camera image. Figure 8A And Figure 8B is a diagram for explaining the constraint conditions added to the bundle adjustment. It should be noted that in Figure 8A , in order to simplify the drawing, only four feature points fp22_1 to fp22_4 are shown as the feature points corresponding to the edge of the road marking DL. In Figure 8A 's case, the specified constraint condition is represented by the following formula (i). d_n represents the vertical distance from the feature point fp22_n (n = 1, 2, 3,...) to the edge of the road marking DL.

[0054] min∑(d_n)^2(i)

[0055] As described above, by performing bundle adjustment with constraint conditions added to each edge of the road marking DL, as Figure 8B shown, the road marking DL on the camera image and the point cloud PC_DL on the environmental map corresponding to the road marking DL are matched as planes. Thus, even when using the point cloud data of a flat object such as a road marking, where it is difficult to establish a correspondence between the feature points in the camera image and the feature points on the environmental map, for the estimation of its own position, it is possible to accurately estimate its own position without generating Figure 4B such mismatches.

[0056] When the information (point cloud data) related to the road on which the vehicle is traveling is not included in the environmental map stored in the storage unit 12, the environmental map generation unit 116 generates an environmental map corresponding to the road. Specifically, when the vehicle is traveling on a road for which an environmental map has not been generated (a road on which the vehicle has not traveled, a newly built road), the environmental map generation unit 116 generates point cloud data corresponding to the road based on the feature points (extracted feature points) extracted by the feature point extraction unit 112, and appends the generated point cloud data to the environmental map. At this time, the environmental map generation unit 116 stores, in the storage unit 12, the recognition result (category information) of the object category of each feature point obtained by the category recognition unit 113 in correspondence with each feature point. On the other hand, when the information related to the road on which the vehicle is traveling is included in the environmental map, that is, when the vehicle is traveling on a road for which an environmental map has been generated, the environmental map generation unit 116 updates the environmental map stored in the storage unit 12 based on the feature points (extracted feature points) extracted by the feature point extraction unit 112. In addition, the environmental map generation unit 116 updates the category information stored in the storage unit 12 according to the recognition result of the object category of each feature point obtained by the category recognition unit 113.

[0057] Figure 9 is a flowchart showing an example of the processing executed by the CPU of the controller 10 according to a predetermined program. The processing shown in this flowchart is executed, for example, at a predetermined cycle during the process of the vehicle traveling in the autonomous driving mode. Figure 5

[0058] First, in step S1, the controller 10 acquires a camera image from the camera 1a. In step S2, the controller 10 extracts feature points from the camera image acquired in step S1. In step S31, the controller 10 performs segmentation (region segmentation) on the camera image acquired in step S1. Specifically, the camera image is segmented into regions for each object category. In step S32, the controller 10 matches each of the feature points (extracted feature points) extracted in step S2 with the feature points on the environmental map. Specifically, the corresponding feature points (corresponding feature points) corresponding to each of the extracted feature points are searched for in the environmental map stored in the storage unit 12. In step S33, the controller 10 determines which region among the regions obtained by segmenting the camera image in step S31 each of the extracted feature points belongs to, and identifies the category of the object corresponding to each of the extracted feature points based on the determination result. Then, for each of the extracted feature points for which corresponding feature points are found in the matching in step S32, that is, for each pair of the extracted feature point and the corresponding feature point, the controller 10 compares the category of the object corresponding to the extracted feature point with the category of the object corresponding to the corresponding feature point. Based on the comparison result, pairs with different object categories are excluded from the matching result (search result).

[0059] In step S34, the controller 10 estimates its own position based on each extracted feature point and the corresponding feature point corresponding to each extracted feature point. More specifically, first, the controller 10 solves the PNP problem or the like based on the correspondence between the two-dimensional coordinates of the extracted feature points (position coordinates on the camera image) and the three-dimensional coordinates of the corresponding feature points (position coordinates on the environmental map), thereby calculating an initial estimated value of its own position. Next, the controller 10 adds the constraint condition defined by the above formula (i) and performs bundle adjustment using multiple camera images with different viewpoints. Thereby, the error included in the initial estimated value is minimized, and a final estimated value of its own position is calculated.

[0060] In addition, in parallel with the processing of steps S31 to S34, the controller 10 performs the processing of steps S41 to S42. In step S41, the controller 10 generates an environmental map corresponding to the road on which the vehicle is traveling based on the feature points (extracted feature points) extracted in step S2, and stores it in the storage unit 12. In step S42, the recognition result of the object category for each extracted feature point obtained in step S33 is stored in the storage unit 12 as category information.

[0061] By adopting the embodiment described above, the following effects can be obtained.

[0062] (1) The position estimation device 50 estimates the position of the vehicle based on the feature points (first feature points) of the objects around the vehicle included in the camera image of the camera 1a that detects the situation around the vehicle and the feature points (second feature points) of the objects included in the environmental map. The position estimation device 50 includes: a feature point extraction unit 112 that extracts feature points of the objects around the vehicle from the camera image of the camera 1a; a storage unit 12 that stores the environmental map and category information showing the category of the object corresponding to the feature point for each feature point included in the environmental map; a category recognition unit 113 that, as a recognition unit, recognizes the category of the object corresponding to the feature points (extracted feature points) extracted by the feature point extraction unit 112 based on the camera image of the camera 1a; a search unit 114 that searches for the corresponding feature points (corresponding feature points) corresponding to the extracted feature points from the environmental map based on the extracted feature points, the category of the object corresponding to the extracted feature points recognized by the category recognition unit 113, and the category information stored in the storage unit 12; and a position estimation unit 115 that estimates the position of the vehicle based on the extracted feature points and the corresponding feature points searched by the search unit 114. Thereby, the traveling position of the vehicle can be estimated with high accuracy.

[0063] (2) The position estimation unit 115 minimizes the error of the position of the host vehicle estimated based on the feature points (extracted feature points) extracted by the feature point extraction unit 112 and the feature points (corresponding feature points) searched by the search unit 114 through bundle adjustment. At this time, the position estimation unit 115 adds a prescribed constraint condition and performs bundle adjustment. The prescribed constraint condition is defined such that when the corresponding feature points included in the environmental map are projected onto the camera image of the camera 1a based on the position of the host vehicle estimated by the position estimation unit 115, the vertical distance between the corresponding feature points on the camera image and the object corresponding to the corresponding feature points is minimized. Thereby, even when the point cloud data of a flat object such as a road marking, for which it is difficult to establish a correspondence between the feature points in the camera image and the feature points on the environmental map, is used for estimating the traveling position of the host vehicle, the traveling position of the host vehicle can be estimated with high accuracy without causing a false match.

[0064] (3) The vehicle control system 100 further includes a position estimation device 50, a driving actuator AC, and a driving control unit 16 that controls the actuator AC based on the position of the host vehicle estimated by the position estimation unit 115. Thereby, the host vehicle can travel well in the autonomous driving mode.

[0065] The above-described embodiment can be modified in various ways. Hereinafter, modification examples will be described.

[0066] (Modification Example 1)

[0067] In the above-described embodiment, the information acquisition unit 111 acquires the detection data (camera image) of the camera 1a as an in-vehicle detector. However, the in-vehicle detector may be a device other than a camera, such as a radar or a lidar, and the information acquisition unit may acquire the detection data of the radar or the lidar.

[0068] (Modification Example 2)

[0069] In addition, in the above-described embodiment, the controller 10 executes its own position estimation process (S31 to S34) at a predetermined cycle while the vehicle is traveling in the autonomous driving mode. However, the controller 10 may also function as a reliability determination unit that determines whether the reliability of the position of the vehicle estimated by the position estimation unit 115 is less than a predetermined level. Note that the reliability determination unit determines that the reliability of the position of the vehicle estimated by the position estimation unit 115 is less than a predetermined level when the difference between the number of feature points corresponding to a predetermined area in front of the traveling direction of the vehicle (for example, the shooting range at the current time of the camera 1a) extracted by the feature point extraction unit 112 from the camera image at the current time point and the number of feature points corresponding to the predetermined area among the feature points included in the environmental map stored in the storage unit 12 is equal to or greater than a predetermined threshold. In addition, the controller 10 may also function as a stop control unit that outputs a stop instruction to stop the estimation of the position of the vehicle to the position estimation unit 115 when the reliability determination unit determines that the reliability is less than a predetermined level during the travel of the vehicle on the road, or when the number of times the reliability determination unit determines that the reliability is less than a predetermined level exceeds a predetermined number. In this way, in the case where the position of the vehicle is continuously lost, by interrupting the estimation of the position of the vehicle, the processing load of the position estimation device 50 can be reduced.

[0070] Note that, in this modification, the stop control unit may also output a stop instruction to the position estimation unit 115 according to the traveling state of the vehicle. In this case, the controller 10 also functions as a state acquisition unit that acquires vehicle state information indicating the state of the vehicle. The stop control unit determines whether the vehicle can continue to travel based on the vehicle state information acquired by the state acquisition unit. When it is determined that the vehicle cannot continue to travel, the stop control unit outputs a stop instruction to the position estimation unit 115. The vehicle state information includes information indicating whether a wheel (tire) has a flat tire, acceleration information indicating the degree of shaking (vertical and lateral shaking) of the vehicle body, and the like. For example, when it is determined from the vehicle state information that a wheel has a flat tire, the stop control unit determines that the vehicle cannot continue to travel. In addition, when the acceleration in the vertical direction or the lateral direction of the vehicle body indicated by the vehicle state information (acceleration information) is equal to or greater than a predetermined value, it is determined that the road surface condition has deteriorated and that the vehicle cannot continue to travel.

[0071] (Modification 3)

[0072] However, when the environmental conditions such as the time period for generating the environmental map, the brightness around the vehicle, and the weather (meteorology) (hereinafter referred to as environmental conditions) are different from those at the time of acquiring the camera image, sometimes the feature points (corresponding feature points) corresponding to the feature points (extracted feature points) extracted from the camera image do not exist on the environmental map, or the corresponding points of the feature points on the environmental map do not exist in the camera image. In this case, the matching accuracy of the feature points between the environmental map and the camera image decreases, and the position of the vehicle cannot be identified with high precision. Therefore, in order to address such a problem, the storage unit 12 may also store, in correspondence, multiple environmental maps generated in different external environments and environmental information indicating the external environment at the time of generating the environmental map. In this case, the environmental map generation unit 116 acquires information related to the meteorology, time, and surrounding brightness at the time of generating the environmental map. More specifically, the environmental map generation unit 116 acquires the meteorological information near the traveling position of the vehicle from an external server (not shown) that provides meteorological information via the communication unit 7. In addition, the environmental map generation unit 116 detects (acquires) the brightness around the vehicle based on the camera image of the camera 1a and acquires the shooting time of the camera image. The environmental map generation unit 116 uses the information indicating the meteorology, time, and brightness acquired at the time of generating the environmental map as environmental information, and stores it in the storage unit 12 together with the environmental map. Similarly, the search unit 114 acquires information related to the external environment (meteorology, time, and surrounding brightness) at the current time point. The search unit 114 reads out from the storage unit 12 the environmental map corresponding to the external environment at the current time point based on the acquired information and the environmental information stored in the storage unit 12, and searches for the feature points (corresponding feature points) corresponding to the feature points (extracted feature points) extracted from the camera image by the feature point extraction unit 112 in the read environmental map. Figure 1 and stores it in the storage unit 12.

[0073] Furthermore, in the above-described embodiment, the position estimation device 50 is applied to an autonomous driving vehicle, but the position estimation device 50 can also be applied to vehicles other than autonomous driving vehicles. For example, the position estimation device 50 can also be applied to a manually driven vehicle equipped with ADAS (Advanced Driver-Assistance Systems).

[0074] The above description is ultimately an example, and the present invention is not limited by the above-described embodiment and modification examples as long as the features of the present invention are not impaired. One or more of the above-described embodiment and modification examples can be arbitrarily combined, and the modification examples can also be combined with each other.

[0075] By adopting the present invention, the traveling position of the vehicle can be estimated with high precision.

[0076] The present invention has been described above in connection with preferred embodiments, but those skilled in the art should understand that various modifications and changes can be made without departing from the scope of disclosure of the claims.

Claims

1. A position estimating device (50) that estimates the position of a vehicle based on a first feature point of an object around the vehicle included in detection data of an in-vehicle detector (1a) that detects the surrounding conditions of the vehicle and a second feature point of the object included in map information, characterized in that, Comprising: A feature point extraction unit (112) that extracts the first feature points from the detection data of the vehicle detector (1a); A storage unit (12) that stores the map information and the category information indicating the category of the object corresponding to the second feature points together; An identification unit (113) that identifies the category of the object corresponding to the first feature points extracted by the feature point extraction unit (112) based on the detection data of the vehicle detector (1a); A search unit (114) that searches for the second feature points corresponding to the first feature points from the map information based on the first feature points extracted by the feature point extraction unit (112), the category of the object corresponding to the first feature points identified by the identification unit (113), and the category information stored in the storage unit (12); And A position estimation unit (115) that estimates the position of the vehicle based on the first feature points extracted by the feature point extraction unit (112) and the second feature points searched by the search unit (114).

2. The position estimation device according to claim 1, characterized in that The position estimation unit (115) minimizes the error of the position of the vehicle estimated based on the first feature points extracted by the feature point extraction unit (112) and the second feature points searched by the search unit (114) through bundle adjustment.

3. The position estimation device according to claim 2, characterized in that The position estimation unit (115) performs the bundle adjustment by adding prescribed constraint conditions, The prescribed constraint conditions are defined as follows: when projecting the second feature points included in the map information onto the image represented by the detection data of the vehicle detector (1a) at the position of the vehicle estimated by the position estimation unit (115), the vertical distance between the second feature points on the image and the object corresponding to the second feature points is minimized.

4. The position estimation device according to claim 1, characterized in that The storage unit (12) stores by establishing a correspondence relationship between a plurality of the map information respectively generated in different external environments and the environment information indicating the external environment at the time of map generation, The search unit (114) reads out the map information corresponding to the external environment at the time of obtaining the detection data of the vehicle detector (1a) from the storage unit (12) according to the environment information, and searches for the second feature points corresponding to the first feature points extracted from the detection data by the feature point extraction unit (112) from the read-out map information.

5. The position estimation device according to claim 1, wherein Further comprising: A reliability determination unit that determines whether the reliability of the position of the vehicle estimated by the position estimation unit (115) is less than a prescribed level; and A stop control unit that, during the travel of the vehicle on a road included in the map information, outputs a stop instruction to stop estimating the position of the vehicle to the position estimation unit (115) when the number of times the reliability determination unit determines that the reliability is less than the specified level exceeds a specified number.

6. The position estimation device according to claim 1, characterized in that, It further includes: A state acquisition unit that acquires the vehicle state of the vehicle; and A stop control unit that outputs a stop instruction to stop estimating the position of the vehicle to the position estimation unit according to the vehicle state acquired by the state acquisition unit.

7. The position estimation device according to claim 6, wherein the vehicle state includes at least any one of information indicating whether a wheel has a flat tire and acceleration information indicating the degree of shaking of the vehicle body, the stop control unit determines whether the vehicle can continue to travel according to the vehicle state, and when it determines that the vehicle cannot continue to travel, outputs the stop instruction to the position estimation unit.

8. The position estimation device according to any one of claims 1 to 7, characterized in that, It includes: A travel actuator (AC); and A travel control unit (16) that controls the travel actuator (AC) according to the position of the vehicle estimated by the position estimation unit (115).

Images