Image processing device

Abstract

To properly generate a map required for a safe vehicle control.SOLUTION: An image processing device 60 for detecting the circumference outside a vehicle based on image information acquired by a detector 1a mounted on the vehicle comprises an extraction unit 142 to extract a feature point of an object included in the image information, a selection unit 143 to select the feature point used for the processing from multiple feature points extracted by the extraction unit 142, and an adjustment unit 144 to divide the image information into multiple regions and to adjust the number of feature points selected in each region based on a distance to the object included in the multiple regions.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to an image processing apparatus for detecting an external situation of a host vehicle based on image information.

Background Art

[0002] As this type of technology, an imaging image is divided into a short-distance region with a large parallax value and a long-distance region with a small parallax value, and feature amounts (road surface candidate points) are detected using criteria and detection methods suitable for each divided region, respectively, to improve the road surface detection accuracy of each region. An information detection device is known (see Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Generally, since it is highly necessary to detect the external situation of a traveling vehicle in real time, it is required to reduce the processing load by suppressing the number of feature points on the image. However, in the conventional technology, the consideration for reducing the processing load was not sufficient. Detecting the external situation of a vehicle enables smooth movement of the vehicle, leading to an improvement in traffic convenience and safety. As a result, it can contribute to the development of a sustainable transportation system.

Means for Solving the Problems

[0005] An image processing apparatus according to one aspect of the present invention is an image processing apparatus for detecting an external situation of a vehicle based on image information acquired by a detector mounted on the vehicle, and includes an extraction unit that extracts feature points of an object included in the image information, a selection unit that selects feature points to be used for processing from a plurality of feature points extracted by the extraction unit, and an adjustment unit that divides the image information into a plurality of regions and adjusts the number of feature points to be selected in each region based on the distances to the objects included in the plurality of regions.

Effects of the Invention

[0006] According to the present invention, it becomes possible to appropriately detect external information necessary for safe vehicle control.

Brief Description of the Drawings

[0007]

Figure 1

Figure 2

Figure 3A

Figure 3B

Figure 4A

Figure 4B

Figure 5A

Figure 5B

Embodiments for Carrying Out the Invention

[0008] Hereinafter, embodiments of the invention will be described with reference to the drawings. The image processing apparatus according to the embodiment of the invention can be applied to a vehicle having an automatic driving function, that is, an autonomous vehicle. Note that the vehicle to which the image processing apparatus according to the present embodiment is applied may be referred to as the host vehicle to distinguish it from other vehicles. The host vehicle may be any of an engine vehicle having an internal combustion engine (engine) as a driving source for traveling, an electric vehicle having a driving motor as a driving source for traveling, and a hybrid vehicle having an engine and a driving motor as driving sources for traveling. The host vehicle can travel not only in the automatic driving mode that does not require driving operation by the driver, but also in the manual driving mode by the driver's driving operation.

[0009] First, the schematic configuration of the host vehicle related to automatic driving will be described. FIG. 1 is a block diagram schematically showing the overall configuration of a vehicle control system 100 of a host vehicle having an image processing apparatus according to the embodiment. As shown in FIG. 1, the vehicle control system 100 mainly includes a controller 10, an external sensor group 1, an internal sensor group 2, an input / output device 3, a positioning unit 4, a map database 5, a navigation device 6, a communication unit 7, and a traveling actuator AC, which are communicably connected to the controller 10 respectively.

[0010] The external sensor group 1 is a general term for a plurality of sensors (external sensors) that detect the external situation, which is the surrounding information of the host vehicle. For example, the external sensor group 1 includes a lidar that measures the scattered light with respect to the omnidirectional irradiation light of the host vehicle to measure the distance from the host vehicle to surrounding obstacles, a radar that irradiates electromagnetic waves and detects reflected waves to detect other vehicles, obstacles, etc. around the host vehicle, and a camera mounted on the host vehicle and having an image sensor (image sensor) such as a CCD or CMOS to image the surrounding (front, rear, and side) of the host vehicle.

[0011] The internal sensor group 2 is a general term for a plurality of sensors (internal sensors) that detect the driving state of the host vehicle. For example, the internal sensor group 2 includes a vehicle speed sensor that detects the vehicle speed of the host vehicle, an acceleration sensor that detects the acceleration in the longitudinal direction and the lateral acceleration (lateral acceleration) of the host vehicle respectively, a rotation speed sensor that detects the rotation speed of the driving power source, a yaw rate sensor that detects the rotational angular velocity of the center of gravity of the host vehicle around the vertical axis, etc. A sensor that detects the driving operations of the driver in the manual driving mode, such as the operation of the accelerator pedal, the operation of the brake pedal, the operation of the steering wheel, etc. is also included in the internal sensor group 2.

[0012] The input / output device 3 is a general term for a device through which commands are input from the driver or information is output to the driver. For example, the input / output device 3 includes various switches through which the driver inputs various commands by operating operation members, a microphone through which the driver inputs commands by voice, a display that provides information to the driver via a display image, a speaker that provides information to the driver by voice, etc.

[0013] The positioning unit (GNSS unit) 4 has a positioning sensor that receives positioning signals transmitted from positioning satellites. The positioning satellites are artificial satellites such as GPS satellites and quasi-zenith satellites. The positioning unit 4 measures the current position (latitude, longitude, altitude) of the host vehicle using the positioning information received by the positioning sensor.

[0014] The map database 5 is a device that stores general map information used in the navigation device 6, and is composed of, for example, a hard disk or a semiconductor element. The map information includes the position information of roads, the information of road shapes (curvature, etc.), and the position information of intersections and branch points. Note that the map information stored in the map database 5 is different from the high-precision map information stored in the storage unit 12 of the controller 10.

[0015] The navigation device 6 is a device that searches for a target route on the road to the destination input by the driver and provides guidance along the target route. The input of the destination and the guidance along the target route are performed via the input / output device 3. The target route is calculated based on the current position of the host vehicle measured by the positioning unit 4 and the map information stored in the map database 5. It is also possible to measure the current position of the host vehicle using the detection values of the external sensor group 1, and the target route may be calculated based on this current position and the highly accurate map information stored in the storage unit 12.

[0016] The communication unit 7 communicates with various servers (not shown) via a network including a wireless communication network typified by the Internet or a mobile phone network, and periodically or at an arbitrary timing acquires map information, driving history information, traffic information, etc. from the server. Not only can it acquire driving history information, but it may also transmit the driving history information of the host vehicle to the server via the communication unit 7. The network includes not only a public wireless communication network but also a closed communication network provided for each predetermined management area, such as a wireless LAN, Wi-Fi (registered trademark), Bluetooth (registered trademark), etc. The acquired map information is output to the map database 5 or the storage unit 12, and the map information is updated.

[0017] The actuator AC is a driving actuator for controlling the driving of the host vehicle. When the driving power source is an engine, the actuator AC includes an actuator for the throttle for adjusting the opening degree (throttle opening) of the throttle valve of the engine. When the driving power source is a driving motor, the driving motor is included in the actuator AC. The actuator AC also includes a brake actuator for operating the braking device of the host vehicle and a steering actuator for driving the steering device.

[0018] The controller 10 is composed of an electronic control unit (ECU). More specifically, the controller 10 includes a computer having an arithmetic unit 11 such as a CPU (microprocessor), a storage unit 12 such as a ROM and a RAM, and other peripheral circuits (not shown) such as an I / O interface. Although a plurality of ECUs with different functions, such as an engine control ECU, a driving motor control ECU, and a braking device ECU, can be provided separately, in FIG. 1, for the sake of convenience, the controller 10 is shown as a set of these ECUs.

[0019] High-precision detailed map information (referred to as high-precision map information) is stored in the storage unit 12. The high-precision map information includes road position information, road shape (such as curvature) information, road gradient information, intersection and branch point position information, types and position information of road demarcation lines such as white lines, number of lanes information, lane width and position information for each lane (information on the center position of the lane and the boundary lines of the lane position), position information of landmarks (buildings, traffic lights, signs, etc.) as marks on the map, and road surface profile information such as road surface unevenness. In the embodiment, the center line, lane boundary line, outside lane line, etc. are collectively referred to as road demarcation lines. The high-precision map information stored in the storage unit 12 includes map information (referred to as external map information) acquired from outside the host vehicle via the communication unit 7, and a map (referred to as internal map information) created by the host vehicle itself using the detection values of the external sensor group 1 or the detection values of the external sensor group 1 and the internal sensor group 2.

[0020] The external map information is, for example, information of a map (referred to as a cloud map) obtained via a cloud server, and the internal map information is information of a map (referred to as an environmental map) composed of three-dimensional point cloud data generated by mapping using a technique such as SLAM (Simultaneous Localization and Mapping). The external map information is shared between the host vehicle and other vehicles, while the internal map information is unique map information of the host vehicle (for example, map information uniquely possessed by the host vehicle). In the case of a road not traveled by the host vehicle, a newly constructed road, etc., an environmental map is created by the host vehicle itself. Note that the internal map information may be provided to a server device or other vehicles via the communication unit 7. In addition to the above-described high-precision map information, the storage unit 12 also stores the traveling locus information of the host vehicle, various control programs, and information such as threshold values used in the programs.

[0021] Functionally, the arithmetic unit 11 includes a host vehicle position recognition unit 13, an external environment recognition unit 14, a behavior plan generation unit 15, a traveling control unit 16, and a map generation unit 17.

[0022] Based on the position information of the host vehicle obtained by the positioning unit 4 and the map information of the map database 5, the host vehicle position recognition unit 13 recognizes (which may also be referred to as estimating) the position of the host vehicle on the map (host vehicle position). The host vehicle position may be recognized (estimated) using the high-precision map information stored in the storage unit 12 and the peripheral information of the host vehicle detected by the external sensor group 1, whereby the host vehicle position can be recognized with high precision. The movement information (movement direction, movement distance) of the host vehicle can also be calculated based on the detection values of the internal sensor group 2, whereby the host vehicle position can be recognized. When the host vehicle position can be measured by a sensor installed outside the road or on the side of the road, the host vehicle position can also be recognized by communicating with the sensor via the communication unit 7.

[0023] The external situation recognition unit 14 recognizes the external situation around the host vehicle based on signals from an external sensor group 1 such as a lidar, radar, camera, etc. For example, it recognizes the positions, speeds, and accelerations of surrounding vehicles (front vehicles and rear vehicles) traveling around the host vehicle, the positions of surrounding vehicles parked or stopped around the host vehicle, and the positions and states of other objects. Other objects include signs, traffic lights, markings such as road lane lines and stop lines, buildings, guardrails, utility poles, billboards, pedestrians, bicycles, etc. The states of other objects include the colors of traffic lights (red, blue, yellow), the moving speeds and directions of pedestrians and bicycles, etc. A part of the stationary objects among other objects constitutes landmarks that are indicators of positions on the map, and the external situation recognition unit 14 also recognizes the positions and types of landmarks.

[0024] The action plan generation unit 15 generates a driving trajectory (target trajectory) of the host vehicle from the current time to a predetermined time in the future based on, for example, the target route calculated by the navigation device 6, the high-precision map information stored in the storage unit 12, the host vehicle position recognized by the host vehicle position recognition unit 13, and the external situation recognized by the external situation recognition unit 14. When there are a plurality of trajectories that are candidates for the target trajectory on the target route, the action plan generation unit 15 selects the optimal trajectory that complies with the laws and regulations and meets criteria such as efficient and safe driving from among them, and sets the selected trajectory as the target trajectory. Then, the action plan generation unit 15 generates an action plan corresponding to the generated target trajectory. The action plan generation unit 15 generates various action plans corresponding to overtaking driving for overtaking a preceding vehicle, lane change driving for changing the driving lane, following driving for following a preceding vehicle, lane keep driving for maintaining the lane so as not to deviate from the driving lane, deceleration driving, or acceleration driving. When generating the target trajectory, the action plan generation unit 15 first determines the driving mode and generates the target trajectory based on the driving mode.

[0025] In the automatic driving mode, the travel control unit 16 controls each actuator AC so that the host vehicle travels along the target trajectory generated by the action plan generation unit 15. More specifically, in the automatic driving mode, the travel control unit 16 calculates a required driving force for obtaining the target acceleration per unit time calculated by the action plan generation unit 15 in consideration of the travel resistance determined by the road gradient and the like. Then, for example, the actuator AC is feedback-controlled so that the actual acceleration detected by the internal sensor group 2 becomes the target acceleration. That is, the actuator AC is controlled so that the host vehicle travels at the target vehicle speed and the target acceleration. In the manual driving mode, the travel control unit 16 controls each actuator AC according to the travel command (such as a steering operation) from the driver acquired by the internal sensor group 2.

[0026] While traveling in the manual driving mode, the map generation unit 17 generates an environmental map around the road on which the host vehicle has traveled as internal map information using the detection values detected by the external sensor group 1. For example, from a plurality of frames of camera images acquired by a camera, an edge indicating the contour of an object is extracted based on the luminance and color information of each pixel, and feature points are extracted using the edge information. The feature points are, for example, intersections of edges and correspond to corners of buildings, corners of road signs, and the like. The map generation unit 17 calculates the three-dimensional position of the feature points while estimating the position and orientation of the camera so that the same feature points converge to one point among a plurality of frames of camera images according to the algorithm of the SLAM technique. By performing this calculation process for each of a plurality of feature points, an environmental map composed of three-dimensional point cloud data is generated. Note that instead of the camera, data acquired by a radar or a lidar may be used to extract feature points of the objects around the host vehicle and generate an environmental map. Also, when generating the environmental map, if the map generation unit 17 determines by object detection such as pattern matching processing that the map-important ground features (for example, road markings, traffic signals, signs, etc.) are included in the camera image, the position information of the points corresponding to the feature points of the ground features based on the camera image is added to the environmental map and recorded in the storage unit 12.

[0027] The own-vehicle position recognition unit 13 performs the position recognition process of the own vehicle in parallel with the map creation process by the map generation unit 17. That is, the own-vehicle position is estimated based on the change in the position of the feature points over time. The map creation process and the position recognition (estimation) process are performed simultaneously according to the algorithm of the SLAM technology. The map generation unit 17 can generate an environmental map not only when driving in the manual driving mode but also when driving in the automatic driving mode. When an environmental map has already been generated and stored in the storage unit 12, the map generation unit 17 may update the environmental map based on the feature points newly extracted from the newly acquired camera image (which may be referred to as new feature points).

[0028] By the way, the more feature points are used in the generation process of the environmental map using the SLAM technology, the higher the accuracy of the matching between the environmental map and the camera image, and the more accurately the own-vehicle position can be estimated. However, if a large number of feature points are extracted from the vicinity to the far distance of the own vehicle in the camera image, the processing load will become too large. Therefore, it is desirable to preferentially select the feature points of the features of the map-important ground objects over the feature points of other ground objects in the camera image and suppress the total number of feature points used in the processing. In the embodiment, the same feature points are tracked among a plurality of frames of camera images according to the algorithm of the SLAM technology, and the three-dimensional positions of the feature points are calculated. At this time, the distance from the own vehicle extracted from the camera image to the feature points (objects) is acquired, and the number of feature points on the objects close to the own vehicle and the number of feature points far from the own vehicle are adjusted so that there is no excessive bias.

[0029] The image processing apparatus that performs the above processing will be described in more detail. FIG. 2 is a block diagram showing the main configuration of the image processing apparatus 60 according to the embodiment. The image processing apparatus 60 is used for controlling the driving operation of the own vehicle and constitutes a part of the vehicle control system 100 in FIG. 1. As shown in FIG. 2, the image processing apparatus 60 includes a controller 10, a camera 1a, a radar 1b, and a lidar 1c.

[0030] Camera 1a forms part of the external sensor group 1 in FIG. 1. Camera 1a may be a monocular camera or a stereo camera, and images the surroundings of the host vehicle. Camera 1a is attached, for example, at a predetermined position in the front part of the host vehicle, continuously images the front space of the host vehicle at a predetermined frame rate, and sequentially outputs frame image data (simply referred to as a camera image) as detection information to the controller 10. FIG. 3A is a diagram showing an example of a camera image of a certain frame acquired by camera 1a. The camera image IM includes other vehicles V1 traveling in front of the host vehicle, other vehicles V2 traveling in the right lane of the host vehicle, traffic signals SG around the host vehicle, pedestrians PE, traffic signs TS1, TS2, buildings BL1, BL2, BL3 around the host vehicle, outer lane lines OL, lane boundary lines SL, and the like.

[0031] The radar 1b in FIG. 2 is mounted on the host vehicle and detects other vehicles, obstacles, etc. around the host vehicle by irradiating electromagnetic waves and detecting reflected waves. The radar 1b outputs a detection value (detection data) as detection information to the controller 10. The lidar 1c is mounted on the host vehicle and measures scattered light with respect to the omnidirectional irradiation light of the host vehicle to detect the distance from the host vehicle to surrounding obstacles. The lidar 1c outputs a detection value (detection data) as detection information to the controller 10.

[0032] The controller 10 includes an arithmetic unit 11 and a storage unit 12. The arithmetic unit 11 functionally includes an information acquisition unit 141, an extraction unit 142, a selection unit 143, an adjustment unit 144, a calculation unit 171, a generation unit 172, and a host vehicle position recognition unit 13. The information acquisition unit 141, the extraction unit 142, the selection unit 143, and the adjustment unit 144 are included, for example, in the external environment recognition unit 14 in FIG. 1. The calculation unit 171 and the generation unit 172 are included, for example, in the map generation unit 17 in FIG. 1.

[0033] The information acquisition unit 141 acquires information used for controlling the driving operation of the host vehicle from the storage unit 12. More specifically, the information acquisition unit 141 reads out landmark information included in the environmental map from the storage unit 12, and further acquires information indicating the positions of the lane lines of the road on which the host vehicle travels and the extending directions of those lane lines (hereinafter referred to as lane line information) from the landmark information. When the lane line information does not include information indicating the extending direction of the lane lines, the information acquisition unit 141 may calculate the extending direction of those lane lines based on the positions of the lane lines. Also, information indicating the positions and extending directions of the lane lines of the road on which the host vehicle travels may be acquired from the road map information, white line map (information indicating the positions of lane lines such as white and yellow) stored in the storage unit 12, and the like.

[0034] The extraction unit 142 extracts edges indicating the outlines of objects from the camera image IM (exemplified in FIG. 3A) acquired by the camera 1a, and extracts feature points using the edge information. As described above, the feature points are, for example, intersections of edges. FIG. 3B is a diagram exemplifying the feature points based on the camera image IM of FIG. 3A. The black circles in the figure indicate the feature points. Note that the extraction unit 142 according to the embodiment targets static objects included in the camera image IM for extraction of feature points, and excludes moving objects from the targets for extraction of feature points. As described above, this is because geographically important features, in other words, features useful for estimating the self-position and map generation (trees, traffic lights, signs located above the road, etc.) are static objects and not moving objects. The identification of static objects and moving objects may be performed, for example, as follows. The image data of static objects (in other words, objects fixed to the ground) is targeted for extraction of feature points because the positions on the environmental map match between frames of the camera image IM. On the other hand, the image data of moving objects is excluded from the targets for extraction of feature points because the positions on the environmental map do not match between frames.

[0035] The selection unit 143 selects feature points for which three-dimensional positions are to be calculated from among the feature points extracted by the extraction unit 142. For example, unique feature points that are easy to distinguish from other feature points and that remain after the adjustment process by the following adjustment unit 144 are selected.

[0036] The adjustment unit 144 performs an adjustment process for adjusting the number of feature points as follows. First, the adjustment unit 144 divides the camera image IM into a plurality of regions. FIG. 4A is a diagram illustrating, as an example, a camera image IM divided into a total of 30 regions of 6 horizontally and 5 vertically in a rectangular shape. The numbers described in each region indicate the region IDs.

[0037] Next, the adjustment unit 144 calculates the average value of the distances to the feature points extracted in each region (referred to as the average distance). FIG. 4B is a schematic diagram showing the average distance of each region in the camera image IM of a certain frame. The horizontal axis indicates the average distance, and the vertical axis indicates the region ID. The average distance and the region ID shown in FIG. 4B are those of a camera image different from the camera image IM illustrated in FIGS. 3A and 3B. Note that the region ID may also be referred to as a grid ID.

[0038] The adjustment unit 144 reduces the number of feature points in the regions where the average distance to the feature points (the surface of the object) is farther among the plurality of regions so that it is less than the number of feature points in the regions where the average distance is closer. In other words, more feature points in the regions where the average distance to the feature points is shorter are left as selection targets than the feature points in the regions where the average distance is longer. In the example shown in FIG. 4B, the average distances of region IDs 1 to 6, 10 to 12, and 16 to 18 are shorter than the average distances of the other region IDs. Therefore, by reducing the number of feature points in region IDs 7 to 9, 13 to 15, and 19 to 30, more feature points in region IDs 1 to 6, 10 to 12, and 16 to 18 are left as selection targets.

[0039] The adjustment unit 144 further leaves more feature points on the object where more feature points are extracted in each region as selection targets than the other feature points on the object in that region. In other words, in each region, the number of feature points on the object where the number of extracted feature points is small is reduced. Note that when the number of feature points extracted from the camera image IM by the adjustment unit 144 is equal to or less than a predetermined number, the adjustment of the number of feature points for the camera image IM of the frame may be omitted.

[0040] The external recognition unit 14 performs, on the camera image IM of each frame acquired by the camera 1a, the acquisition of information by the above-described information acquisition unit 141, the extraction of feature points by the extraction unit 142, the adjustment of the number of feature points by the adjustment unit 144, and the selection of feature points by the selection unit 143.

[0041] The distance information from the camera 1a to the feature points (the surface of the object) is estimated by, for example, the external recognition unit 14 using machine learning (such as DNN (Deep Neural Network)) technology based on the position on the image of the object captured in the camera image IM, to estimate the distance in the depth direction from the camera 1a to the object including the feature points. Note that the distance from the host vehicle to the object may be calculated based on the detection values of the radar 1b and the lidar 1c. Also, as will be described later, the host vehicle position recognition unit 13 may estimate the distance from the host vehicle to the object on the environmental map based on the positions of the feature points of the landmarks captured in the camera image IM.

[0042] The calculation unit 171 in FIG. 2 calculates the three-dimensional position of the feature points while estimating the position and orientation of the camera 1a so that the same feature points converge to one point among the camera images IM of a plurality of frames. The calculation unit 171 calculates the three-dimensional positions of a plurality of different feature points selected by the selection unit 143, respectively.

[0043] The generation unit 172 generates an environmental map composed of three-dimensional point cloud data including information on each three-dimensional position, using the three-dimensional positions of a plurality of different feature points calculated by the calculation unit 171.

[0044] The host vehicle position recognition unit 13 estimates the position of the host vehicle on the environmental map based on the environmental map stored in the storage unit 12. First, the host vehicle position recognition unit 13 estimates the position of the host vehicle in the vehicle width direction. Specifically, the host vehicle position recognition unit 13 uses machine learning technology to recognize the lane lines of the road included in the camera image IM newly acquired by the camera 1a. The host vehicle position recognition unit 13 recognizes the position and extension direction of the lane lines included in the camera image IM on the environmental map based on the lane line information acquired from the landmark information included in the environmental map stored in the storage unit 12. Then, the host vehicle position recognition unit 13 estimates the relative positional relationship (positional relationship on the environmental map) in the vehicle width direction between the host vehicle and the lane lines based on the position and extension direction of the lane lines on the environmental map. In this way, the position of the host vehicle in the vehicle width direction on the environmental map is estimated.

[0045] Next, the host vehicle position recognition unit 13 estimates the position of the host vehicle in the traveling direction. Specifically, the host vehicle position recognition unit 13 recognizes a landmark (for example, building BL1) from the camera image IM (FIG. 3A) newly acquired by the camera 1a by processing such as pattern matching, and recognizes the feature points on the landmark from among the feature points extracted by the extraction unit 142. Further, the host vehicle position recognition unit 13 estimates the distance in the traveling direction from the host vehicle to the landmark based on the positions of the feature points of the landmark shown in the camera image IM. Note that the distance from the host vehicle to the landmark may be calculated based on the detection values of the radar 1b and the lidar 1c.

[0046] The host vehicle position recognition unit 13 searches for the feature points corresponding to the above-described landmark in the environmental map stored in the storage unit 12. In other words, the host vehicle position recognition unit 13 recognizes the feature points that match the feature points of the landmark recognized from the newly acquired camera image IM from among the plurality of feature points (point group data) constituting the environmental map. Next, the host vehicle position recognition unit 13 estimates the position of the host vehicle in the traveling direction on the environmental map based on the position of the feature points on the environmental map corresponding to the feature points of the landmark and the distance in the traveling direction from the host vehicle to the landmark. As described above, the host vehicle position recognition unit 13 recognizes the position of the host vehicle on the environmental map based on the estimated positions of the host vehicle in the vehicle width direction and the traveling direction on the environmental map.

[0047] The storage unit 12 stores the information of the environmental map generated by the generation unit 172. Further, the storage unit 12 stores the information indicating the traveling locus of the host vehicle. The traveling locus is represented, for example, as the position of the host vehicle on the environmental map recognized during traveling by the host vehicle position recognition unit 13.

[0048] <Explanation of the flowchart> An example of the process executed by the controller 10 in FIG. 2 according to a predetermined program will be described with reference to the flowcharts of FIGS. 5A and 5B. FIG. 5A shows the process of creating an environmental map, which is started, for example, in the manual driving mode and repeated at a predetermined cycle. FIG. 5B shows the details of step S30 in FIG. 5A.

[0049] In step S10 of FIG. 5A, the controller 10 acquires the camera image IM as detection information from the camera 1a and proceeds to step S20.

[0050] In step S20, the controller 10 extracts feature points from the camera image IM by the extraction unit 142 and proceeds to step S30.

[0051] In step S30, the controller 10 selects feature points by the selection unit 143 and proceeds to step S40.

[0052] In step S40, the controller 10 calculates the three-dimensional positions of a plurality of different feature points by the calculation unit 171 and proceeds to step S50.

[0053] In step S50, the controller 10 generates an environmental map composed of three-dimensional point cloud data including the information of the three-dimensional positions of a plurality of different feature points by the generation unit 172 and proceeds to step S60.

[0054] In step S60, when the controller 10 recognizes that the position where the host vehicle is traveling is on the past travel trajectory, the controller 10 corrects the information of the three-dimensional position included in the environmental map by loop closing processing and proceeds to step S70. The loop closing process will be briefly described. Generally, in SLAM technology, errors accumulate because the position of the host vehicle is recognized while the vehicle is moving. For example, when the host vehicle travels around a road that forms a closed loop like the shape of the Japanese character "ロ" (ro), the accumulated error causes the positions of the starting point and the ending point not to match. Therefore, when the controller 10 recognizes that the position where the host vehicle is traveling is on the past travel trajectory, a loop closing process is performed to make the position of the host vehicle recognized using the feature points (referred to as new feature points) extracted from the newly acquired camera image at the same travel point as in the past and the position of the host vehicle recognized in the past using the feature points extracted from the camera image acquired during past travel have the same coordinates.

[0055] In step S70, the controller 10 records the information of the environmental map in the storage unit 12 and ends the process according to FIG. 5A.

[0056] In step S31 of FIG. 5B, the adjustment unit 144 divides the camera image IM into a plurality of regions and proceeds to step S32.

[0057] In step S32, the adjustment unit 144 calculates the average distance, which is the average value of the distances to the feature points extracted in each region, and proceeds to step S33. As described above, the distance to the feature point can be estimated as the depth direction distance from the camera 1a to the object surface corresponding to the feature point. Also, the distance calculated based on the detection values of the radar 1b and the lidar 1c may be acquired. Furthermore, the distance estimated on the environmental map based on the positions of the feature points of the landmarks shown in the camera image IM may be acquired from the host vehicle position recognition unit 13.

[0058] In step S33, the adjustment unit 144 reduces the feature points in the region where the average distance is far and proceeds to step S34. As described above, more feature points in the region where the average distance to the feature points is shorter are left as selection targets than the feature points in the region where the average distance is longer.

[0059] In step S34, the adjustment unit 144 leaves more feature points on the object from which more feature points have been extracted and proceeds to step S35. As described above, in each region, the number of feature points on the object from which the number of extracted feature points is small is reduced.

[0060] In step S35, the selection unit 143 selects the feature points remaining in each outlier region after the adjustment by the adjustment unit 144, and ends the process according to FIG. 5B.

[0061] According to the embodiment described above, the following operational effects can be obtained. (1) The image processing device 60 is an image processing device for detecting the external situation of the host vehicle based on the camera image IM as the image information acquired by the camera 1a as a detector mounted on the host vehicle. The extraction unit 142 extracts the feature points of the object included in the camera image IM, the selection unit 143 selects the feature points to be used in the process from the plurality of feature points extracted by the extraction unit 142, and the camera image IM is divided into a plurality of grid-like regions as an example, and based on the distances to the objects included in the plurality of regions, an adjustment unit 144 that adjusts the number of feature points to be selected in each region is provided. With this configuration, since the number of feature points for each region of the camera image IM is optimized based on the distance to the object, the total number of feature points used in the process can be suppressed. Therefore, for example, the processing load when calculating the three-dimensional position of the feature points is reduced, and it becomes possible to quickly generate an environmental map necessary for safe vehicle control.

[0062] (2) In the image processing device 60 of (1) above, the adjustment unit 144 performs adjustment so as to leave the feature points on the object from which more feature points have been extracted by the extraction unit 142. With such a configuration, since more characteristic points on the object are adopted for processing, it becomes possible to perform processing based on more reliable information in the camera image IM.

[0063] (3) In the image processing apparatus 60 according to (1) or (2) above, the adjustment unit 144 further has a function as a distance measuring unit that calculates the distance to the object based on the camera image IM, and calculates the average distance from the host vehicle position to a plurality of feature points for each region based on the calculation result of the distance, and adjusts the number of feature points so that the number of feature points in a region with a longer average distance is reduced. Generally, feature points far from the host vehicle have less influence on driving compared to feature points near the host vehicle. Therefore, by reducing the number of feature points in a region with a longer average distance compared to a region with a shorter average distance, it becomes possible to appropriately suppress the total number of feature points in the camera image IM used for processing. As a result, for example, the processing load when calculating the three-dimensional position of the feature points is reduced, and it becomes possible to quickly generate an environmental map necessary for safe vehicle control.

[0064] (4) In the image processing apparatus 60 according to (1) to (3) above, the extraction unit 142 targets stationary objects included in the camera image IM as extraction targets for feature points, and excludes moving objects from the extraction targets for feature points. With such a configuration, for example, even an object shown at the upper part of the screen of the camera image IM, features useful for estimating the self-position and map generation (such as tall trees, traffic lights, signs located above the road, etc.) become extraction targets for feature points. As a result, it becomes possible to appropriately generate an environmental map necessary for safe vehicle control.

[0065] (5) The image processing apparatus 60 according to (1) to (4) above further includes a storage unit 12 that stores a plurality of frames of the camera image IM in frame units, a map generation unit 17 as a search unit that searches for a set of the same feature points commonly included in the plurality of frames of the camera image IM stored in the storage unit 12, and a host vehicle position recognition unit 13 as an estimation unit that estimates the host vehicle position based on the set of feature points searched by the map generation unit 17. With such a configuration, it becomes possible to search for a set of identical feature points commonly included in the camera images IM between different frames and accurately estimate the position of the host vehicle.

[0066] (6) In the image processing apparatus 60 of (5) above, when the decrease in the reliability of the estimated host vehicle position occurs a predetermined number or more times, the host vehicle position estimation is interrupted. With such a configuration, when the reliability of the host vehicle position estimation decreases, it becomes possible to reduce unnecessary processing compared to the case where it is not interrupted by interrupting the host vehicle position estimation itself.

[0067] (7) In the image processing apparatus 60 of (5) or (6) above, the host vehicle position recognition unit 13 interrupts the host vehicle position estimation according to the situation of the host vehicle. With such a configuration, for example, when a situation where it is difficult to continue running the host vehicle is predicted due to an accident such as a flat tire or deterioration of the road surface condition, etc., by interrupting the host vehicle position estimation itself, it becomes possible to reduce unnecessary processing compared to the case where it is not interrupted.

[0068] (8) In the image processing apparatus 60 of (1) or (2) above, a calculation unit 171 that calculates the three-dimensional positions of the same feature points commonly included in the image information of a plurality of frames for each of the plurality of different feature points selected by the selection unit 143, and a generation unit 172 and a host vehicle position recognition unit 13 as calculation units that perform map generation and host vehicle position estimation based on the three-dimensional positions of the plurality of different feature points calculated by the calculation unit 171 are further provided. The adjustment unit 144 calculates the average distance from the host vehicle position to the plurality of feature points for each region based on the calculation result by the host vehicle position recognition unit 13, and adjusts the number of feature points so that the number of feature points is reduced in regions where the average distance is longer. Generally, since feature points far from the host vehicle have less influence on driving compared to feature points near the host vehicle, by reducing the number of feature points in a region with a longer average distance compared to a region with a shorter average distance, it becomes possible to appropriately suppress the total number of feature points in the camera image IM used for processing. As a result, for example, the processing load when calculating the three-dimensional positions of the feature points is reduced, and it becomes possible to quickly generate an environmental map necessary for safe vehicle control.

[0069] The above-described embodiment can be modified into various forms. Hereinafter, modification examples will be described. (Modification Example 1) The number of regions illustrated in FIG. 4A is an example and may be changed as appropriate. Also, although an example of dividing into a plurality of rectangular regions (which may be called a grid pattern) has been described, it may be divided into a plurality of hexagonal regions (which may be called a honeycomb pattern).

[0070] (Modification Example 2) In the above description, an example has been described in which the average distance from the host vehicle position to a plurality of feature points in the region is calculated for each region, and the number of feature points is adjusted so that the number of feature points decreases as the average distance becomes longer. Instead of the average distance from the host vehicle position to a plurality of feature points in the region, based on the median of the distances from the host vehicle position to a plurality of feature points in the region, the number of feature points may be adjusted so that the number of feature points decreases as the median of the distances becomes longer.

[0071] (Modification Example 3) Also, in the above description, an example has been described in which the number of feature points is adjusted so that the number of feature points decreases as the region is farther from the host vehicle position. In addition to this, in a region near the host vehicle position, the number of feature points on an object closer to the host vehicle position may be increased to such an extent that there is no excessive bias in the number of feature points on an object farther from the host vehicle position. An object for which the number of feature points should be increased near the host vehicle position is, for example, a feature of a map-important ground object. That is, the adjustment unit 144 adjusts the number of feature points so as to increase the number of feature points on an object defined in advance as a map-important ground object regardless of the average distance for each region from the host vehicle position to a plurality of feature points. By configuring in this way, the detection accuracy of the feature points in the vicinity of the host vehicle position is improved, and the host vehicle position recognition unit 13 can accurately recognize the host vehicle position.

[0072] The above description is merely an example, and the present invention is not limited to the above-described embodiments and modifications as long as the features of the present invention are not impaired. It is also possible to arbitrarily combine one or more of the above embodiments and modifications, and it is also possible to combine the modifications with each other.

Explanation of Reference Numerals

[0073] 1a Camera, 1b Radar, 1c Lidar, 10 Controller, 11 Arithmetic Unit, 12 Storage Unit, 13 Host Vehicle Position Recognition Unit, 14 External World Recognition Unit, 17 Map Generation Unit, 60 Image Processing Device, 141 Information Acquisition Unit, 142 Extraction Unit, 143 Selection Unit, 144 Adjustment Unit, 171 Calculation Unit, 172 Generation Unit, BL1 to BL3 Buildings, IM Camera Image

Claims

1. An image processing apparatus for detecting an external situation of a vehicle based on image information acquired by a detector mounted on the vehicle, comprising: an extraction unit that extracts feature points of an object included in the image information; a selection unit that selects feature points to be used for processing from the plurality of feature points extracted by the extraction unit; a adjustment unit that divides the image information into a plurality of regions and adjusts the number of feature points to be selected in each region based on the distances to the objects included in the plurality of regions; An image processing apparatus characterized by comprising the above.

2. In the image processing apparatus according to Claim 1, the adjustment unit performs the adjustment so as to leave feature points on an object from which more feature points are extracted by the extraction unit. An image processing apparatus characterized by the above.

3. In the image processing apparatus according to Claim 1 or 2, further comprising a distance measuring unit that calculates the distance to the object based on the image information, the adjustment unit calculates, for each region, an average distance from the host vehicle position to the plurality of feature points based on the calculation result by the distance measuring unit, and adjusts the number of feature points so as to reduce the number of feature points in a region where the average distance is longer. An image processing apparatus characterized by the above.

4. In the image processing apparatus according to Claim 3, the extraction unit targets stationary objects included in the image information for extraction of the feature points and excludes moving objects from the targets for extraction of the feature points. An image processing apparatus characterized by the above.

5. In the image processing apparatus according to Claim 4, a storage unit that stores a plurality of frames of the image information in frame units; a search unit that searches for a set of the same feature points commonly included in the plurality of frames of the image information stored in the storage unit; an estimation unit that estimates the host vehicle position based on the set of feature points searched by the search unit. An image processing apparatus further characterized by comprising the above.

6. In the image processing apparatus according to Claim 5, the estimation unit interrupts the host vehicle position estimation when a decrease in the reliability of the estimated host vehicle position occurs by a predetermined number or more. An image processing apparatus characterized by the above.

7. In the image processing apparatus according to Claim 6, the estimation unit interrupts the host vehicle position estimation according to the situation of the vehicle. An image processing apparatus characterized by the above.

8. In the image processing apparatus according to Claim 1 or 2, A calculation unit that calculates the three-dimensional positions of the same feature points commonly included in the image information of a plurality of frames for each of the plurality of different feature points selected by the selection unit; An arithmetic unit that performs map generation and own-vehicle position estimation based on the three-dimensional positions of the plurality of different feature points calculated by the calculation unit; and The adjustment unit calculates, for each of the regions, an average distance from the own-vehicle position to the plurality of feature points based on the calculation result by the arithmetic unit, and adjusts the number of the feature points so that the number of the feature points is reduced in a region where the average distance is long. An image processing apparatus characterized by the above.

9. In the image processing apparatus according to claim 3, The adjustment unit adjusts the number of the feature points so as to increase the number of the feature points on a predetermined object regardless of the average distance for each of the regions from the own-vehicle position to the plurality of feature points. An image processing apparatus characterized by the above.

Images