Map Generator

Abstract

To appropriately generate a map that is needed for safe vehicle control.SOLUTION: A map generation device 60 comprises: an extraction unit 171 that extracts feature points from detection information detected by an onboard detector 1a that detects the surrounding state of the host vehicle; a selection unit 172 that selects the feature point from a plurality of extracted feature points, that is used for calculation by a calculation unit 173; the calculation unit 173 that calculates, for each of a plurality of different feature points selected by the selection unit 172, the three-dimensional position of the same feature point included in a plurality of pieces of detection information, on the basis of the plurality of pieces of detection information, and using the position and attitude of the onboard detector 1a; and a generation unit 174 that generates a map that includes information regarding each three-dimensional position, using the calculated three-dimensional position of the plurality of different feature points. The selection unit 172 selects a feature point on an object so as to reduce a bias in number between a plurality of feature points on an object located in a first distance range on the onboard detector 1a side and a plurality of feature points on an object located in a second distance range that is farther than the first distance range.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a map generating device that generates a map used to estimate the position of a vehicle. [Background technology]

[0002] Conventionally, as this type of device, a device configured to create a map using feature points extracted from a captured image acquired by a camera mounted on a traveling vehicle is known (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2020-153956 Summary of the Invention [Problem to be solved by the invention]

[0004] With conventional technology, the accuracy of map information can be compromised depending on whether the feature points on the image being tracked are mostly located near or far from the camera. [Means for solving the problem]

[0005] A map generating device according to one aspect of the present invention includes an extraction unit that extracts feature points from detection information detected by an on-board detector that detects the situation around the vehicle; a selection unit that selects feature points to be used for calculation by a calculation unit described later from the plurality of feature points extracted by the extraction unit; a calculation unit that calculates, based on the plurality of detection information and using the position and orientation of the on-board detector, the three-dimensional position of the same feature point included in the plurality of detection information for each of the plurality of different feature points selected by the selection unit; and a generation unit that generates a map including information on each of the three-dimensional positions using the three-dimensional positions of the plurality of different feature points calculated by the calculation unit, wherein the selection unit selects feature points so as to reduce a bias in the number of feature points on a plurality of objects located in a first distance range on the on-board detector side and feature points on a plurality of objects located in a second distance range that is farther away than the first distance range., the feature points are thinned out from the distance region having more feature points than the average value of the number of feature points in the first distance region and the number of feature points in the second distance region so as to bring the number of feature points closer to the average value, and the feature points remaining after thinning out are Select the feature points. [Effects of the Invention]

[0006] According to the present invention, it is possible to appropriately generate a map required for safe vehicle control. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a block diagram showing an overall configuration of a vehicle control system according to an embodiment of the present invention; [Figure 2] FIG. 1 is a block diagram showing a configuration of a main part of a map generating device according to an embodiment. [Figure 3A] FIG. 10 is a diagram showing an example of a camera image. [Figure 3B] FIG. 10 is a diagram illustrating extracted feature points. [Figure 3C] FIG. 10 is a diagram showing feature points associated with a plurality of distance ranges. [Figure 4A] FIG. 10 is a diagram illustrating an example of the distribution of feature points. [Figure 4B] A diagram of a camera image divided into multiple regions. [Figure 5A] 6 is a flowchart illustrating an example of processing by a program executed by a controller. [Figure 5B] 6 is a flowchart illustrating an example of processing by a program executed by a controller. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. A map generating device according to an embodiment of the present invention can be applied to a vehicle having an automatic driving function, i.e., an automatic driving vehicle. The vehicle to which the map generating device according to the embodiment is applied may be referred to as the host vehicle to distinguish it from other vehicles. The host vehicle may be an engine vehicle having an internal combustion engine (engine) as a driving source, an electric vehicle having a driving motor as a driving source, or a hybrid vehicle having an engine and a driving motor as driving sources. The host vehicle can run not only in an automatic driving mode in which no driving operation by the driver is required, but also in a manual driving mode in which the driver operates the vehicle.

[0009] First, a schematic configuration of a host vehicle related to autonomous driving will be described. Fig. 1 is a block diagram showing a schematic overall configuration of a vehicle control system 100 of the host vehicle having a map generation device according to an embodiment. As shown in Fig. 1, the vehicle control system 100 mainly includes a controller 10, an external sensor group 1 and an internal sensor group 2, each of which is communicatively connected to the controller 10, an input / output device 3, a positioning unit 4, a map database 5, a navigation device 6, a communication unit 7, and a driving actuator AC.

[0010] The external sensor group 1 is a collective term for a plurality of sensors (external sensors) that detect the external situation, which is information about the surroundings of the host vehicle. For example, the external sensor group 1 includes a lidar that measures the distance from the host vehicle to surrounding obstacles by measuring scattered light in response to light irradiated in all directions of the host vehicle, a radar that detects other vehicles and obstacles around the host vehicle by irradiating electromagnetic waves and detecting reflected waves, and a camera that is mounted on the host vehicle and has an imaging element (image sensor) such as a CCD or CMOS that captures images of the surroundings (front, rear, and sides) of the host vehicle.

[0011] The internal sensor group 2 is a collective term for a plurality of sensors (internal sensors) that detect the driving state of the host vehicle. For example, the internal sensor group 2 includes a vehicle speed sensor that detects the vehicle speed of the host vehicle, an acceleration sensor that detects the longitudinal acceleration and the lateral acceleration (lateral acceleration) of the host vehicle, a rotation speed sensor that detects the rotation speed of the driving source, a yaw rate sensor that detects the rotation angular velocity around the vertical axis of the center of gravity of the host vehicle, etc. The internal sensor group 2 also includes sensors that detect the driver's driving operations in manual driving mode, such as operation of the accelerator pedal, operation of the brake pedal, operation of the steering wheel, etc.

[0012] The input / output device 3 is a general term for devices that input commands from the driver and output information to the driver. For example, the input / output device 3 includes various switches through which the driver inputs various commands by operating operating members, a microphone through which the driver inputs commands by voice, a display that provides information to the driver via displayed images, a speaker that provides information to the driver by voice, etc.

[0013] The positioning unit (GNSS unit) 4 has a positioning sensor that receives positioning signals transmitted from positioning satellites. The positioning satellites are artificial satellites such as GPS satellites and quasi-zenith satellites. The positioning unit 4 measures the current position (latitude, longitude, altitude) of the vehicle using the positioning information received by the positioning sensor.

[0014] The map database 5 is a device that stores general map information used in the navigation device 6, and is configured with, for example, a hard disk or semiconductor elements. The map information includes road position information, road shape information (curvature, etc.), and position information of intersections and branch points. Note that the map information stored in the map database 5 is different from the highly accurate map information stored in the memory unit 12 of the controller 10.

[0015] The navigation device 6 is a device that searches for a target route on roads to a destination input by the driver and provides guidance along the target route. The input of the destination and guidance along the target route are performed via the input / output device 3. The target route is calculated based on the current position of the vehicle measured by the positioning unit 4 and map information stored in the map database 5. The current position of the vehicle can also be measured using detection values ​​from the external sensor group 1, and the target route can be calculated based on this current position and high-precision map information stored in the memory unit 12.

[0016] The communication unit 7 communicates with various servers (not shown) via networks including wireless communication networks such as the Internet and mobile phone networks, and acquires map information, driving history information, traffic information, and the like from the servers periodically or at any timing. In addition to acquiring driving history information, the communication unit 7 may also transmit driving history information of the vehicle to the server. Networks include not only public wireless communication networks but also closed communication networks established for each predetermined management area, such as wireless LAN, Wi-Fi (registered trademark), Bluetooth (registered trademark), and the like. The acquired map information is output to the map database 5 and the storage unit 12, where the map information is updated.

[0017] Actuators AC are driving actuators for controlling the driving of the host vehicle. When the driving source is an engine, actuators AC include a throttle actuator that adjusts the opening of the engine's throttle valve (throttle opening). When the driving source is a driving motor, actuators AC include the driving motor. Actuators AC also include a brake actuator that operates the host vehicle's braking device and a steering actuator that drives the steering device.

[0018] The controller 10 is composed of an electronic control unit (ECU). More specifically, the controller 10 includes a computer having an arithmetic unit 11 such as a CPU (microprocessor), a storage unit 12 such as a ROM and RAM, and other peripheral circuits (not shown) such as an I / O interface. Note that although multiple ECUs with different functions, such as an engine control ECU, a traction motor control ECU, and a braking device ECU, can be provided separately, for convenience, the controller 10 is shown in FIG. 1 as a collection of these ECUs.

[0019] The storage unit 12 stores highly accurate, detailed map information (referred to as high-accuracy map information). The high-accuracy map information includes road position information, road shape (curvature, etc.), road gradient information, intersection and branch point position information, types of road dividing lines such as white lines and their positions, the number of lanes, lane width and position information for each lane (information on the center position of the lane and boundary lines of the lane positions), position information of landmarks (buildings, traffic lights, signs, etc.) as markers on the map, and road surface profile information such as road surface irregularities. In the embodiment, center lines, lane boundary lines, outer lane lines, etc. are collectively referred to as road dividing lines. The high-precision map information stored in the memory unit 12 includes map information obtained from outside the vehicle via the communication unit 7 (referred to as external map information), and a map created by the vehicle itself using detection values ​​from the external sensor group 1 or detection values ​​from the external sensor group 1 and the internal sensor group 2 (referred to as internal map information).

[0020] The external map information is, for example, information on a map (called a cloud map) acquired via a cloud server, and the internal map information is information on a map (called an environmental map) made up of three-dimensional point cloud data generated by mapping using a technology such as SLAM (Simultaneous Localization and Mapping). The external map information is shared between the vehicle and other vehicles, whereas the internal map information is map information unique to the vehicle (for example, map information that is solely possessed by the vehicle). For roads that the vehicle has not yet traveled on, newly constructed roads, etc., the vehicle itself creates an environmental map. The internal map information may be provided to a server device or other vehicles via the communication unit 7. In addition to the high-precision map information described above, the storage unit 12 also stores information such as the vehicle's travel trajectory information, various control programs, and threshold values ​​used in the programs.

[0021] The calculation unit 11 has, as functional components, a vehicle position recognition unit 13, an external environment recognition unit 14, a behavior plan generation unit 15, a driving control unit 16, and a map generation unit 17.

[0022] The vehicle position recognition unit 13 recognizes (may also be called estimates) the position of the vehicle on the map (vehicle position) based on the position information of the vehicle obtained by the positioning unit 4 and the map information of the map database 5. The vehicle position may be recognized (estimated) using high-precision map information stored in the memory unit 12 and information about the surroundings of the vehicle detected by the external sensor group 1, thereby enabling the vehicle position to be recognized with high precision. The vehicle's position can also be recognized by calculating the vehicle's movement information (movement direction, movement distance) based on the detection values ​​of the internal sensor group 2. When the vehicle's position can be measured by an external sensor installed on the road or beside the road, the vehicle's position can also be recognized by communicating with the sensor via the communication unit 7.

[0023] The external environment recognition unit 14 recognizes the external situation around the host vehicle based on signals from the external sensor group 1, such as a lidar, radar, and camera. For example, it recognizes the positions, speeds, and accelerations of surrounding vehicles (vehicles ahead and vehicles behind) traveling around the host vehicle, the positions of surrounding vehicles stopped or parked around the host vehicle, and the positions and states of other objects. Examples of other objects include signs, traffic lights, markings such as road dividing lines and stop lines, buildings, guardrails, utility poles, billboards, pedestrians, and bicycles. Examples of the states of other objects include the color of traffic lights (red, green, yellow), the movement speed and direction of pedestrians and bicycles, and the like. Some of the stationary objects among the other objects constitute landmarks that serve as indicators of locations on a map, and the external environment recognition unit 14 also recognizes the positions and types of the landmarks.

[0024] The behavior plan generation unit 15 generates a traveling trajectory (target trajectory) of the host vehicle from the current time to a predetermined time ahead based on, for example, a target route calculated by the navigation device 6, high-precision map information stored in the memory unit 12, the host vehicle position recognized by the host vehicle position recognition unit 13, and the external situation recognized by the external environment recognition unit 14. When there are multiple trajectories that are candidates for the target trajectory on the target route, the behavior plan generation unit 15 selects an optimal trajectory from among them that satisfies criteria such as compliance with laws and regulations and efficient and safe traveling, and sets the selected trajectory as the target trajectory. The behavior plan generation unit 15 then generates a behavior plan according to the generated target trajectory. The behavior plan generation unit 15 generates various behavior plans corresponding to overtaking driving to overtake a preceding vehicle, lane-changing driving to change the traveling lane, following driving to follow a preceding vehicle, lane-keeping driving to maintain the traveling lane without deviating from the traveling lane, decelerating driving, accelerating driving, etc. When generating the target trajectory, the behavior plan generation unit 15 first determines a driving mode and generates the target trajectory based on the driving mode.

[0025] In the autonomous driving mode, the driving control unit 16 controls the actuators AC so that the vehicle travels along the target trajectory generated by the behavior plan generation unit 15. More specifically, in the autonomous driving mode, the driving control unit 16 calculates a required driving force for achieving the target acceleration per unit time calculated by the behavior plan generation unit 15, taking into account the driving resistance determined by the road gradient, etc. Then, for example, the driving control unit 16 feedback-controls the actuators AC so that the actual acceleration detected by the internal sensor group 2 becomes the target acceleration. In other words, the driving control unit 16 controls the actuators AC so that the vehicle travels at the target vehicle speed and target acceleration. Note that in the manual driving mode, the driving control unit 16 controls the actuators AC in response to driving commands (such as steering operations) from the driver acquired by the internal sensor group 2.

[0026] While driving in manual driving mode, the map generation unit 17 uses detection values ​​detected by the external sensor group 1 to generate an environmental map of the area around the road on which the vehicle has traveled as internal map information. For example, edges that indicate the contours of objects are extracted from multiple frames of camera images acquired by a camera based on brightness and color information for each pixel, and feature points are extracted using the edge information. Feature points are, for example, intersections of edges, and correspond to corners of buildings or corners of road signs. The map generation unit 17 estimates the position and orientation of the camera according to the SLAM technology algorithm so that identical feature points converge to a single point across multiple frames of camera images, and calculates the three-dimensional position of the feature points. By performing this calculation process for each of the multiple feature points, an environmental map consisting of three-dimensional point cloud data is generated. It should be noted that instead of using a camera, data acquired by a radar or a lidar may be used to extract feature points of objects around the vehicle and generate an environmental map. Furthermore, when generating the environmental map, if the map generation unit 17 determines by pattern matching processing or the like that a predetermined feature (e.g., road dividing lines, traffic lights, signs, etc.) having feature points that were not used in calculating the above-mentioned three-dimensional positions is included in the camera image, the map generation unit 17 adds the position information of the points corresponding to the feature points of the feature based on the camera image to the environmental map and records it in the memory unit 12.

[0027] The vehicle position recognition unit 13 performs a process of recognizing the position of the vehicle in parallel with the map creation process by the map generation unit 17. That is, the vehicle position is estimated based on changes in the positions of feature points over time. The map creation process and the position recognition (estimation) process are performed simultaneously according to the algorithm of SLAM technology. The map generation unit 17 can generate an environmental map not only when driving in manual driving mode, but also when driving in automatic driving mode. If an environmental map has already been generated and stored in the memory unit 12, the map generation unit 17 may update the environmental map based on newly extracted feature points (which may also be called new feature points) from newly acquired camera images.

[0028] The more feature points used to generate an environmental map using SLAM technology, the more accurate the matching between the environmental map and camera images, and the more accurate the estimation of the vehicle's position. However, if there are many feature points near the vehicle, the estimated vehicle position will be accurate, but the estimated vehicle orientation will be inaccurate. Conversely, if there are many feature points far from the vehicle, the estimated vehicle orientation will be accurate, but the estimated vehicle position will be inaccurate. If either the estimated orientation or estimated position of the vehicle is inaccurate, the accuracy of the generated environmental map will also be compromised. Therefore, it is desirable to distribute the feature points used to generate the environmental map over a wide range in the camera image, from near to far from the vehicle. In this embodiment, the same feature points are tracked across multiple frames of camera images according to the SLAM technology algorithm, and before calculating the three-dimensional position of the feature points, the distances to the feature points extracted from the camera images are estimated, and the number of feature points is adjusted so that the distances to the feature points are distributed over a wide range in the camera images, from near to far from the vehicle.

[0029] The map generating device that performs the above processing will now be described in more detail. Fig. 2 is a block diagram showing the configuration of a main part of a map generating device 60 according to an embodiment. The map generating device 60 is used to control the traveling operation of the host vehicle, and constitutes a part of the vehicle control system 100 shown in Fig. 1. As shown in Fig. 2, the map generating device 60 has a controller 10, a camera 1a, a radar 1b, and a lidar 1c.

[0030] The camera 1a constitutes part of the external sensor group 1 in Fig. 1. The camera 1a may be a monocular camera or a stereo camera, and captures images of the surroundings of the vehicle. The camera 1a is attached, for example, to a predetermined position in the front of the vehicle, continuously captures images of the space ahead of the vehicle at a predetermined frame rate, and sequentially outputs frame image data (simply referred to as camera images) as detection information to the controller 10. 3A is a diagram showing an example of a camera image of a certain frame captured by camera 1a. Camera image IM includes another vehicle V1 traveling ahead of the host vehicle, another vehicle V2 traveling in the right lane of the host vehicle, traffic lights SG around the host vehicle, pedestrians PE, traffic signs TS1 and TS2, buildings BL1, BL2, and BL3 around the host vehicle, outer roadway lines OL, and lane boundary lines SL.

[0031] The radar 1b in FIG. 2 is mounted on the host vehicle and detects other vehicles, obstacles, etc. around the host vehicle by emitting electromagnetic waves and detecting reflected waves. The radar 1b outputs detection values ​​(detection data) as detection information to the controller 10. The lidar 1c is mounted on the host vehicle and measures scattered light in response to irradiated light in all directions from the host vehicle to detect the distance from the host vehicle to surrounding obstacles. The lidar 1c outputs detection values ​​(detection data) as detection information to the controller 10.

[0032] The controller 10 includes a calculation unit 11 and a storage unit 12. The calculation unit 11 has, as functional components, an information acquisition unit 141, an extraction unit 171, a selection unit 172, a calculation unit 173, a generation unit 174, and a vehicle position recognition unit 13. The information acquisition unit 141 is included in, for example, the external environment recognition unit 14 in Fig. 1. The extraction unit 171, the selection unit 172, the calculation unit 173, and the generation unit 174 are included in, for example, the map generation unit 17 in Fig. 1.

[0033] The information acquisition unit 141 acquires information used to control the traveling operation of the vehicle from the storage unit 12. More specifically, the information acquisition unit 141 reads landmark information included in the environmental map from the storage unit 12, and further acquires, from the landmark information, information indicating the positions of the lane markings on the road on which the vehicle is traveling and the extension direction of those lane markings (hereinafter referred to as lane marking information). When the lane marking information does not include information indicating the direction in which the lane marks extend, the information acquisition unit 141 may calculate the direction in which the lane marks extend based on the positions of the lane marks. Alternatively, the information acquisition unit 141 may acquire information indicating the positions and directions in which the lane marks of the road on which the host vehicle is traveling from road map information or a white line map (information indicating the positions of white, yellow, and other lane markings) stored in the storage unit 12.

[0034] The extraction unit 171 extracts edges that indicate the contour of an object from the camera image IM (illustrated in FIG. 3A) captured by the camera 1a, and extracts feature points using the edge information. As described above, feature points are, for example, intersections of edges. FIG. 3B is a diagram illustrating feature points extracted by the extraction unit 171 based on the camera image IM of FIG. 3A. Black circles in the diagram indicate feature points.

[0035] The selection unit 172 selects feature points for calculating three-dimensional positions from among the feature points extracted by the extraction unit 171. In this embodiment, unique feature points that are easily distinguishable from other feature points are selected.

[0036] First, the selection unit 172 associates each feature point extracted by the extraction unit 171 with a plurality of distance ranges based on distance information from the camera 1a to an object that includes the feature point. FIG. 3C is a diagram showing, as an example, feature points associated with three distance ranges. The white circles in the diagram indicate feature points associated with the first distance range on the camera 1a side. The hatched circles in the diagram indicate feature points associated with the second distance range that is farther than the first distance range. The black circles in the diagram indicate feature points associated with the third distance range that is farther than the second distance range.

[0037] The distance information from the camera 1a to the object including the feature point is estimated by using machine learning (DNN (Deep Neural Network) etc.) technology based on the position of the object captured in the camera image IM, and the distance in the depth direction from the camera 1a to the object including the feature point is calculated. Note that the distance from the vehicle to the object may be calculated based on the detection value of the radar 1b or the lidar 1c.

[0038] Next, the selection unit 172 selects feature points so as to reduce the bias among the number of feature points on the object located in the first distance range, the number of feature points on the object located in the second distance range, and the number of feature points on the object located in the third distance range. For example, the number of feature points in the other distance ranges is thinned out to reduce the number of feature points so as to approach the number of feature points in the distance range with the fewest number of feature points among the multiple distance ranges. Note that the number of feature points in all distance ranges does not need to be the same, and it is sufficient to reduce the bias among the number of feature points between the distance ranges.

[0039] FIG. 4A is a diagram illustrating an example of the distribution of feature points extracted by extraction unit 171. The horizontal axis indicates the distance from camera 1a to an object containing feature points, and the vertical axis indicates the number of feature points. The number of distance zones and the number of feature points shown in FIG. 4A are from a camera image different from the camera image IM illustrated in FIG. 3C. In FIG. 4A, 32 feature points are associated with nine distance zones (first to ninth distance zones). The horizontal dashed lines in the diagram indicate the average value of the number of feature points on an object located in each distance zone. The selection unit 172 reduces the number of feature points in the second, fourth, fifth, seventh, and ninth distance regions, which have more feature points than the average number of feature points, by, for example, a total of eight feature points indicated by the reference numerals 21, 41, 51, 52, 71, and 91 to 93, thereby bringing the number of feature points in these five regions closer to the average number. Then, it selects 24 feature points remaining in the first to ninth distance regions after the reduction.

[0040] Note that the selection unit 172 may thin out the feature points 21, 41, 51, 52, 71, 91 to 93 to be thinned out so that they are not concentrated in a certain area of ​​the camera image IM. Fig. 4B is a diagram in which pixel data constituting the camera image IM is divided into a plurality of groups (for example, 35 rectangular areas). In Fig. 4B, the circles denoted by the reference numerals 21, 41, 51, 52, 71, 91 to 93 indicate the groups that contain the feature points that are the target of thinning out in Fig. 4A. By targeting the feature points 21, 41, 51, 52, 71, 91 to 93 that are each included in a different group as the target of thinning out, it becomes possible to thin out the feature points so that the number of feature points to be thinned out is not biased across the groups (not biased across the areas of the camera image IM).

[0041] 2 estimates the position and orientation of the camera 1a so that the same feature points converge to one point across multiple frames of camera images IM, and calculates the three-dimensional positions of the feature points. The calculation unit 173 calculates the three-dimensional positions of each of the multiple different feature points selected by the selection unit 172.

[0042] The generating unit 174 uses the three-dimensional positions of the different feature points calculated by the calculating unit 173 to generate an environmental map made up of three-dimensional point cloud data including information on each three-dimensional position.

[0043] The vehicle position recognition unit 13 estimates the vehicle position on the environmental map based on the environmental map stored in the storage unit 12. First, the vehicle position recognition unit 13 estimates the position of the host vehicle in the vehicle width direction. Specifically, the vehicle position recognition unit 13 uses machine learning technology to recognize road lane lines included in the camera image IM newly acquired by the camera 1a. The vehicle position recognition unit 13 recognizes the positions and extension directions of the lane lines included in the camera image IM on the environmental map based on lane line information acquired from landmark information included in the environmental map stored in the memory unit 12. The vehicle position recognition unit 13 then estimates the relative positional relationship between the host vehicle and the lane lines in the vehicle width direction (positional relationship on the environmental map) based on the positions and extension directions of the lane lines on the environmental map. In this way, the position of the host vehicle in the vehicle width direction on the environmental map is estimated.

[0044] Next, the vehicle position recognition unit 13 estimates the position of the vehicle in the traveling direction. In detail, the vehicle position recognition unit 13 recognizes a landmark (e.g., building BL1) from a camera image IM (FIG. 3A) newly acquired by the camera 1a by processing such as pattern matching, and recognizes a feature point on the landmark from among the feature points extracted by the extraction unit 171. Furthermore, the vehicle position recognition unit 13 estimates the distance from the vehicle to the landmark in the traveling direction based on the position of the feature point of the landmark captured in the camera image IM. Note that the distance from the vehicle to the landmark may be calculated based on detection values ​​from the radar 1b and the lidar 1c.

[0045] The vehicle position recognition unit 13 searches for feature points corresponding to the landmarks in the environmental map stored in the storage unit 12. In other words, feature points that match the feature points of the landmarks recognized from the newly acquired camera image IM are recognized from among the multiple feature points (point cloud data) that make up the environmental map. Next, the vehicle position recognition unit 13 estimates the position of the vehicle on the environmental map in the direction of travel based on the position of the feature point on the environmental map that corresponds to the feature point of the landmark and the distance in the direction of travel from the vehicle to the landmark. As described above, the vehicle position recognition unit 13 recognizes the position of the vehicle on the environmental map based on the estimated position of the vehicle on the environmental map in the vehicle width direction and the traveling direction.

[0046] The storage unit 12 stores information about the environmental map generated by the generation unit 174. The storage unit 12 also stores information indicating the travel path of the host vehicle. The travel path is represented as the host vehicle position on the environmental map, recognized by the host vehicle position recognition unit 13 while the host vehicle is traveling, for example.

[0047] <Explanation of the flowchart> An example of processing executed by the controller 10 of Fig. 2 in accordance with a predetermined program will be described with reference to the flowcharts of Fig. 5A and Fig. 5B. Fig. 5A shows the process of creating an environmental map, which is started, for example, in manual driving mode and repeated at a predetermined interval. Fig. 5B shows the details of step S30 of Fig. 5A.

[0048] In step S10 of FIG. 5A, the controller 10 acquires a camera image IM as detection information from the camera 1a, and the process proceeds to step S20.

[0049] In step S20, the controller 10 extracts feature points from the camera image IM using the extraction unit 171, and then the process proceeds to step S30.

[0050] In step S30, the controller 10 selects a feature point using the selection unit 172, and proceeds to step S40.

[0051] In step S40, the controller 10 causes the calculation unit 173 to calculate the three-dimensional positions of the plurality of different feature points, and then proceeds to step S50.

[0052] In step S50, the controller 10 generates, by the generation unit 174, an environmental map made up of three-dimensional point cloud data including information on the three-dimensional positions of a plurality of different feature points, and then proceeds to step S60.

[0053] In step S60, if the controller 10 recognizes that the position where the vehicle is traveling is on the past traveling trajectory, it corrects the three-dimensional position information contained in the environmental map by loop closing processing, and proceeds to step S70. A brief explanation of loop closing processing is provided below. Generally, with SLAM technology, errors accumulate because the vehicle recognizes its own position while moving. For example, when the vehicle travels around a road that is closed in the shape of a square, the accumulated errors cause the start and end points to not match. Therefore, when the vehicle's current position is recognized as being on the previous travel path, loop closing processing is performed to set the vehicle's position recognized using feature points (called new feature points) extracted from a camera image newly acquired at the same travel point as before to the same coordinates as the vehicle's position previously recognized using feature points extracted from a camera image acquired during previous travel.

[0054] In step S70, the controller 10 records the information of the environmental map in the storage unit 12, and ends the processing in FIG. 5A.

[0055] 5B, the selection unit 172 acquires the distances to the feature points extracted by the extraction unit 171, and the process proceeds to step S32. As described above, the distances to the feature points can be estimated as the distance in the depth direction from the camera 1a to the object containing the feature points. Alternatively, the distances can be calculated based on the detection values ​​of the radar 1b and the lidar 1c.

[0056] In step S32, the selection unit 172 associates each of the feature points extracted by the extraction unit 171 with a plurality of distance ranges based on the distance of each feature point, and the process proceeds to step S33.

[0057] In step S33, the selection unit 172 calculates the average value of the number of feature points associated with each distance range (in other words, the feature points on the object located in each distance range), and the process proceeds to step S34.

[0058] In step S34, the selection unit 172 thins out feature points from distance regions that have more feature points than the average number of feature points, and the process proceeds to step S35.

[0059] In step S35, the selection unit 172 selects the feature points remaining in each of the isolated regions after thinning, and ends the processing in FIG. 5B.

[0060] According to the embodiment described above, the following effects can be obtained. (1) The map generating device 60 includes an extraction unit 171 that extracts feature points from a camera image IM as detection information detected by a camera 1a that serves as an on-board detector for detecting the situation around the vehicle; a selection unit 172 that selects feature points to be used for calculation by a calculation unit 173 from the plurality of feature points extracted by the extraction unit 171; a calculation unit 173 that calculates the three-dimensional position of the same feature point included in the camera image IM of multiple frames for each of multiple different feature points selected by the selection unit 172 based on the camera image IM of multiple frames using the position and attitude of the camera 1a; and a generation unit 174 that generates an environmental map including information on each of the three-dimensional positions using the three-dimensional positions of the multiple different feature points calculated by the calculation unit 173, wherein the selection unit 172 selects feature points so as to reduce the imbalance in the number of feature points on multiple objects located in a first distance range on the side of the camera 1a and the number of feature points on multiple objects located in a second distance range that is farther away than the first distance range. With this configuration, the distances of the feature points used to generate the environmental map (distances to objects from which feature points are extracted by the extraction unit 171) are distributed over a wide range in the camera image IM, from near to far from the vehicle, and bias between distance ranges is suppressed. This ensures the accuracy of the estimated attitude and estimated position of the camera 1a (vehicle) when calculating the three-dimensional positions of the feature points, and an accurate environmental map is generated. In this way, it becomes possible to appropriately generate the environmental map necessary for safe vehicle control.

[0061] (2) In the map generating device 60 described in (1) above, the selection unit 172 thins out feature points from a distance region having more feature points than the average value of the number of feature points in the first distance region and the number of feature points in the second distance region so as to bring the number of feature points closer to the average value, and selects the feature points remaining after thinning out. This configuration reduces the bias between distance ranges in the number of feature points used to generate an environmental map, ensuring the accuracy of the estimated attitude and position of the camera 1a (host vehicle) when calculating the three-dimensional positions of feature points, and generating an accurate environmental map.

[0062] (3) In the map generating device 60 described in (2) above, the selection unit 172 divides the camera image IM, which is the detection information of the camera 1a, into multiple groups, for example, rectangular groups, and when thinning out feature points from distance ranges having more feature points than the average value, thins out the feature points so as not to be biased among the groups. This configuration reduces the bias between distance ranges in the number of feature points used to generate an environmental map, ensuring the accuracy of the estimated attitude and position of the camera 1a (host vehicle) when calculating the three-dimensional positions of feature points, and generating an accurate environmental map.

[0063] (4) In the map generating device 60 described above in (1), the selection unit 172 thins out feature points from the distance region having the larger number of feature points out of the first distance region and the second distance region so that the number of feature points approaches that of the distance region having the smaller number of feature points, and selects the feature points remaining after thinning. This configuration reduces the bias between distance ranges in the number of feature points used to generate an environmental map, ensuring the accuracy of the estimated attitude and position of the camera 1a (host vehicle) when calculating the three-dimensional positions of feature points, and generating an accurate environmental map.

[0064] (5) In the map generating device 60 described in (4) above, the selection unit 172 divides the camera image IM, which is the detection information of the camera 1a, into multiple groups, for example, rectangular groups, and when thinning out feature points from distance ranges with a large number of feature points, thins out the feature points so that there is no imbalance among the groups. This configuration reduces the bias between distance ranges in the number of feature points used to generate an environmental map, ensuring the accuracy of the estimated attitude and position of the camera 1a (host vehicle) when calculating the three-dimensional positions of feature points, and generating an accurate environmental map.

[0065] The above embodiment can be modified in various ways, and modifications will be described below. (Variation 1) The numbers of distance zones and feature points shown in Fig. 3C or Fig. 4A are examples and may be changed as appropriate. The number of groups shown in Fig. 4B is also an example and may be changed as appropriate.

[0066] (Variation 2) In the above description, an example has been described in which the selection unit 172 thins out a total of eight feature points indicated by the reference numerals 21, 41, 51, 52, 71, and 91 to 93 from the second, fourth, fifth, seventh, and ninth distance regions, which have more feature points than the average number of feature points, thereby bringing the number of feature points in these five regions closer to the average value. It is also possible to configure the number of feature points to be closer to the median value instead of the average value. In the second modification, the selection unit 172 thins out feature points from each distance region having more feature points than the median number of feature points, thereby bringing the number of feature points in these regions closer to the median value. According to the second modification, the deviation between distance ranges is suppressed with respect to the number of feature points used to generate an environmental map, which ensures the accuracy of the estimated attitude and estimated position of the camera 1a (host vehicle) when calculating the three-dimensional positions of the feature points, and an accurate environmental map is generated.

[0067] The above description is merely an example, and the present invention is not limited to the above-described embodiment and modifications as long as the features of the present invention are not impaired. One or more of the above-described embodiment and modifications can be arbitrarily combined, and modifications can also be combined with each other. [Explanation of symbols]

[0068] 1a camera, 1b radar, 1c lidar, 10 controller, 11 calculation unit, 12 memory unit, 13 vehicle position recognition unit, 14 external environment recognition unit, 17 map generation unit, 60 map generation device, 171 extraction unit, 172 selection unit, 173 calculation unit, 174 generation unit, BL1 to BL3 buildings, IM camera images, OL roadway outer line, SG traffic light, SL lane boundary line, TS1, TS2 traffic sign, V1, V2 other vehicles

Claims

1. an extraction unit that extracts feature points from detection information detected by an on-board detector that detects the surrounding conditions of the host vehicle; a selection unit that selects feature points to be used for calculation by a calculation unit described later from the plurality of feature points extracted by the extraction unit; a calculation unit that calculates, based on the plurality of pieces of detection information and using the positions and orientations of the on-vehicle detector, a three-dimensional position of the same feature point included in the plurality of pieces of detection information for each of the plurality of different feature points selected by the selection unit; a generation unit that generates a map including information on each of the three-dimensional positions using the three-dimensional positions of the plurality of different feature points calculated by the calculation unit, the selection unit thins out feature points from a distance range having more feature points than the average value of the number of feature points in the first distance range and the number of feature points in the second distance range, so as to reduce the imbalance in the number of feature points on a plurality of objects located in a first distance range on the vehicle-mounted detector side and the number of feature points on a plurality of objects located in a second distance range farther than the first distance range, and selects the feature points that remain after the thinning out.

2. In the map generating device according to claim 1, the selection unit divides the detection information into a plurality of groups, and when thinning out the feature points from the distance range having more feature points than the average value, thins out the feature points so as not to be biased among the groups.

3. In the map generating device according to claim 1, the selection unit thins out the feature points from the distance region with a larger number of feature points, either the first distance region or the second distance region, so as to approach the number of feature points in the distance region with a smaller number of feature points, and selects the feature points that remain after the thinning.

4. In the map generating device according to claim 3, the selection unit divides the detection information into a plurality of groups, and when thinning out the feature points from the distance ranges having a large number of feature points, thins out the feature points so as not to bias them among the groups.

Images