Image processing apparatus

Abstract

Provided is an image processing device (60) for detecting a condition outside a vehicle on the basis of image information acquired by a detector (1a) mounted on the vehicle, the image processing device (60) being provided with: an extraction unit (142) for extracting feature points of an object included in the image information; a selection unit (143) that selects a feature point to be used for processing from among the plurality of feature points extracted by the extraction unit (142); and an adjustment unit (144) that divides the image information into a plurality of regions and adjusts the number of feature points selected in each region on the basis of the distance to the object included in the plurality of regions.

Description

Technical Field

[0001] The present invention relates to an image processing device for detecting the external conditions of a vehicle based on image information. Background Art

[0002] As such a technique, there is known an information detection device that divides a captured image into a close-range area with a large parallax value and a long-range area with a small parallax value, and uses criteria and detection methods respectively suitable for the divided areas to detect feature amounts (road surface candidate points), thereby improving the road surface detection accuracy of each area (see Patent Document 1).

[0003] Generally, detection of the external conditions of a traveling vehicle needs to be performed in real time, and thus it is required to suppress the number of feature points in an image to reduce the processing load. However, the study on reducing the processing load in the prior art is not sufficient.

[0004] Detecting the external conditions of a vehicle enables the vehicle to move smoothly, and thus can improve the convenience and safety of traffic. This contributes to the sustainable development of the transportation system.

[0005] Prior Art Documents

[0006] Patent Documents

[0007] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2015-011619 (JP2015-011619A). Summary of the Invention

[0008] An image processing device according to one aspect of the present invention is an image processing device for detecting the external conditions of a vehicle based on image information obtained by a detector mounted on the vehicle, and includes: an extraction unit that extracts feature points of an object included in the image information; a selection unit that selects feature points for processing from among the plurality of feature points extracted by the extraction unit; and an adjustment unit that divides the image information into a plurality of regions and adjusts the number of feature points selected in each region according to the distance to the object included in the plurality of regions. Brief Description of the Drawings

[0009] The object, features, and advantages of the present invention are further clarified by the following description of embodiments related to the drawings.

[0010] Figure 1 is a block diagram schematically showing the overall configuration of a vehicle control system according to an embodiment of the present invention;

[0011] Figure 2 is a block diagram showing the main part configuration of the image processing device according to the embodiment;

[0012] Figure 3A is a diagram showing an example of a camera image;

[0013] Figure 3B A diagram illustrating the extracted feature points;

[0014] Figure 4A A diagram illustrating the camera image segmented into regions;

[0015] Figure 4B A schematic diagram showing the average distance of each region in the camera image of a certain frame;

[0016] Figure 5A This is an example of a flowchart for explaining the processing of a program executed by the Figure 2 controller;

[0017] Figure 5B This is an example of a flowchart for explaining the processing of a program executed by the Figure 2 controller. Detailed implementation mode

[0018] Next, the implementation mode of the invention will be described with reference to the drawings.

[0019] The image processing apparatus according to the implementation mode of the present invention can be applied to a vehicle having an autonomous driving function, that is, an autonomous driving vehicle. It should be noted that the vehicle to which the image processing apparatus of the present implementation mode is applied is sometimes distinguished from other vehicles and referred to as the present vehicle. The present vehicle can be any one of an engine vehicle having an internal combustion engine (engine) as a driving source for traveling, an electric vehicle having a driving motor as a driving source for traveling, and a hybrid vehicle having an engine and a driving motor as driving sources for traveling. The present vehicle can travel not only in an autonomous driving mode without the driver performing driving operations but also in a manual driving mode based on the driver's driving operations.

[0020] First, the schematic structure of the present vehicle related to autonomous driving will be described. Figure 1 This is a block diagram schematically showing the overall configuration of the vehicle control system 100 of the present vehicle having the image processing apparatus according to the implementation mode. As Figure 1 shown, the vehicle control system 100 mainly includes a controller 10, an external sensor group 1, an internal sensor group 2, an input / output device 3, a positioning unit 4, a map database 5, a navigation device 6, a communication unit 7, and a driving actuator AC, which are respectively communicably connected to the controller 10.

[0021] The external sensor group 1 is a general term for a plurality of sensors (external sensors) that detect the external conditions as the surrounding information of the present vehicle. For example, the external sensor group 1 includes a lidar that measures the distance from the present vehicle to surrounding obstacles by irradiating omnidirectional light of the present vehicle and measuring scattered light, a radar that detects other vehicles and obstacles around the present vehicle by irradiating electromagnetic waves and detecting reflected waves, a camera mounted on the present vehicle and having a imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) (image sensor) and photographing the surrounding (front, rear, and side) of the present vehicle, and the like.

[0022] The internal sensor group 2 is a general term for a plurality of sensors (internal sensors) that detect the driving state of the present vehicle. For example, the internal sensor group 2 includes a vehicle speed sensor that detects the vehicle speed of the present vehicle, an acceleration sensor that respectively detects the acceleration in the front-rear direction and the acceleration in the left-right direction (lateral acceleration) of the present vehicle, a rotation speed sensor that detects the rotation speed of the driving power source, a yaw rate sensor that detects the rotational angular velocity of the center of gravity of the present vehicle about the vertical axis, and the like. A sensor that detects the driving operation of the driver in the manual driving mode, such as an operation of the accelerator pedal, an operation of the brake pedal, an operation of the steering wheel, and the like, is also included in the internal sensor group 2.

[0023] The input / output device 3 is a general term for devices that allow a driver to input commands or output information to the driver. For example, the input / output device 3 includes various switches that allow a driver to input various commands by operating an operation member, a microphone that allows a driver to input commands by voice, a display that provides information to the driver by displaying an image, a speaker that provides information to the driver by sound, and the like.

[0024] The positioning unit (GNSS unit) 4 has a positioning sensor that receives positioning signals transmitted from positioning satellites. The positioning satellites are artificial satellites such as GPS satellites and quasi-zenith satellites. The positioning unit 4 measures the current position (latitude, longitude, altitude) of the present vehicle using the positioning information received by the positioning sensor.

[0025] The map database 5 is a device that stores general map information used by the navigation device 6, and is composed of, for example, a hard disk and a semiconductor element. The map information includes the position information of roads, the information of road shapes (curvature, etc.), and the position information of intersections or branch points. It should be noted that the map information stored in the map database 5 is different from the high-precision map information stored in the storage unit 12 of the controller 10.

[0026] The navigation device 6 is a device that searches for a target route on the road to the destination input by the driver and guides along the target route. The input of the destination and the guidance along the target route are performed via the input / output device 3. The target route is calculated based on the current position of the own vehicle measured by the positioning unit 4 and the map information stored in the map database 5. It is also possible to use the detection values of the external sensor group 1 to measure the current position of the own vehicle, and the target route can also be calculated based on this current position and the highly accurate map information stored in the storage unit 12.

[0027] The communication unit 7 communicates with various servers (not shown) via a network including a wireless communication network represented by the Internet, mobile phone networks, etc., and acquires map information, driving history information, traffic information, etc. from the servers at regular intervals or at any time. It is not only possible to acquire driving history information, but also to send the driving history information of the own vehicle to the server via the communication unit 7. The network includes not only public wireless communication networks, but also closed communication networks set for each specified management area, such as wireless LAN, Wi-Fi (registered trademark), Bluetooth (registered trademark), etc. The acquired map information is output to the map database 5 and the storage unit 12, and the map information is updated.

[0028] The actuator AC is a driving actuator for controlling the running of the own vehicle. When the driving power source is an engine, the actuator AC includes a throttle actuator for adjusting the opening degree of the throttle valve (throttle opening) of the engine. When the driving power source is a driving motor, the driving motor is included in the actuator AC. The braking actuator for driving the braking device of the own vehicle and the steering actuator for driving the steering device are also included in the actuator AC.

[0029] The controller 10 is composed of an electronic control unit (ECU). More specifically, the controller 10 includes a computer having an arithmetic unit 11 such as a CPU (microprocessor), a storage unit 12 such as a ROM (read-only memory) and a RAM (random access memory), and other peripheral circuits (not shown) such as an I / O interface. It should be noted that although it is possible to separately provide a plurality of ECUs with different functions such as an engine control ECU, a driving motor control ECU, and a braking device ECU, Figure 1 here, for convenience, the controller 10 is shown as a collection of these ECUs.

[0030] High-precision detailed map information (referred to as high-precision map information) is stored in the storage unit 12. The high-precision map information includes road position information, road shape (such as curvature) information, road slope information, intersection or branch point position information, category or position information of road markings such as white lines, lane number information, lane width and position information of each lane (information on the center position of the lane or the boundary line of the lane position), position information of landmarks (buildings, traffic lights, signs, etc.) as marks on the map, and road surface profile information such as road surface unevenness. In the embodiment, center lines, lane boundary lines, lane outer lines, etc. are collectively referred to as road markings.

[0031] The high-precision map information stored in the storage unit 12 includes map information (referred to as external map information) obtained from outside the vehicle via the communication unit 7, and a map (referred to as internal map information) generated by the vehicle itself using the detection values of the external sensor group 1 or the detection values of the external sensor group 1 and the internal sensor group 2.

[0032] The external map information is, for example, information on a map (referred to as cloud map) obtained through a cloud server, and the internal map information is information on a map (referred to as environmental map) composed of three-dimensional point cloud data generated by mapping using technologies such as SLAM (Simultaneous Localization and Mapping). The external map information is shared between this vehicle and other vehicles, while the internal map information is unique map information of this vehicle (for example, map information uniquely owned by this vehicle). In roads where this vehicle has not traveled, newly established roads, etc., the vehicle itself generates an environmental map. It should be noted that the internal map information can also be provided to the server device and other vehicles via the communication unit 7.

[0033] In addition to the above high-precision map information, the storage unit 12 also stores the driving trajectory information of this vehicle, programs for various controls, and information such as thresholds used in the programs.

[0034] The operation unit 11 has a vehicle position recognition unit 13, an external environment recognition unit 14, an action plan generation unit 15, a driving control unit 16, and a map generation unit 17 as functional structures.

[0035] The vehicle position recognition unit 13 recognizes (which can also be referred to as estimating) the position of this vehicle (vehicle position) on the map based on the position information of this vehicle obtained by the positioning unit 4 and the map information of the map database 5.

[0036] It is also possible to use the high-precision map information stored in the storage unit 12 and the surrounding information of this vehicle detected by the external sensor group 1 to recognize (estimate) the vehicle position, thereby enabling high-precision recognition of the vehicle position.

[0037] The movement information (movement direction, movement distance) of the host vehicle is calculated based on the detection values of the internal sensor group 2, and thus the position of the host vehicle can also be identified. It should be noted that when the position of the host vehicle can be measured by sensors externally installed on or beside the road, the position of the host vehicle can also be identified by communicating with the sensors via the communication unit 7.

[0038] The external recognition unit 14 recognizes the external conditions around the host vehicle based on signals from the external sensor group 1 such as lidar, radar, cameras, etc. For example, it recognizes the positions, speeds or accelerations of surrounding vehicles (front vehicle, rear vehicle) traveling around the host vehicle, the positions of surrounding vehicles parked or standing still around the host vehicle, and the positions or states of other objects. Other objects include signs, traffic lights, road markings or stop lines and other markings, buildings, guardrails, utility poles, signs, pedestrians, bicycles, etc. The states of other objects include the colors (red, green, yellow) of traffic lights, the moving speeds or directions of pedestrians or bicycles, etc. A part of the stationary objects among other objects constitutes a landmark that serves as an indicator of the position on the map, and the external recognition unit 14 also recognizes the positions and categories of the landmarks.

[0039] The action plan generation unit 15 generates, for example, the travel track (target track) of the host vehicle after a specified time from the current time point based on the target path calculated by the navigation device 6, the high-precision map information stored in the storage unit 12, the position of the host vehicle identified by the host vehicle position recognition unit 13, and the external conditions recognized by the external recognition unit 14. When there are multiple tracks that are candidates for the target track on the target path, the action plan generation unit 15 selects the best track that meets criteria such as complying with laws and regulations and driving efficiently and safely from among them, and uses the selected track as the target track. Then, the action plan generation unit 15 generates an action plan corresponding to the generated target track. The action plan generation unit 15 generates various action plans corresponding to overtaking driving for overtaking a preceding vehicle, lane change driving for changing the driving lane, following driving for following a preceding vehicle, lane keeping driving for maintaining the lane without deviating from the driving lane, decelerating driving or accelerating driving, etc. When generating the target track, the action plan generation unit 15 first determines the driving mode and generates the target track according to the driving mode.

[0040] The driving control unit 16 controls each actuator AC in the autonomous driving mode so that the host vehicle travels along the target trajectory generated by the action plan generation unit 15. More specifically, the driving control unit 16 considers the driving resistance determined by the road gradient or the like in the autonomous driving mode, and calculates the required driving force for obtaining the target acceleration per unit time calculated by the action plan generation unit 15. And, for example, feedback control is performed on the actuator AC so that the actual acceleration detected by the internal sensor group 2 becomes the target acceleration. That is, the actuator AC is controlled so that the host vehicle travels at the target vehicle speed and the target acceleration. It should be noted that in the manual driving mode, the driving control unit 16 controls each actuator AC according to the driving instruction (steering operation or the like) from the driver obtained by the internal sensor group 2.

[0041] While traveling in the manual driving mode, the map generation unit 17 generates an environmental map of the periphery of the road traveled by the host vehicle as internal map information using the detection values detected by the external sensor group 1. For example, from a plurality of frames of camera images obtained by the camera, edges showing the outlines of objects are extracted based on the luminance and color information of each pixel, and feature points are extracted using the information of the edges. The feature points are, for example, intersections of the edges, corresponding to the corners of buildings, the corners of road traffic signs, and the like. The map generation unit 17 calculates the three-dimensional positions of the feature points while estimating the position and orientation of the camera according to the algorithm of the SLAM technique so that the same feature point converges to one point among the plurality of frames of camera images. By performing this calculation process for each of the plurality of feature points, an environmental map composed of three-dimensional point cloud data is generated.

[0042] It should be noted that instead of the camera, data obtained by a radar or a lidar may be used to extract the feature points of the objects around the host vehicle to generate an environmental map.

[0043] In addition, when the map generation unit 17 generates an environmental map, if it is determined by object detection such as pattern matching processing that important ground objects (such as road markings, signal lights, signs, etc.) on the map are included in the camera image, the position information of the points corresponding to the feature points of the ground objects based on the camera image is added to the environmental map and recorded in the storage unit 12.

[0044] The vehicle position recognition unit 13 performs the vehicle position recognition process in parallel with the map generation process of the map generation unit 17. That is, based on the position change of the feature points over time, the vehicle position is estimated. The map generation process and the position recognition (estimation) process are carried out simultaneously according to the algorithm of the SLAM technology. The map generation unit 17 can generate an environmental map not only when driving in the manual driving mode but also when driving in the autonomous driving mode. In the case where an environmental map has been generated and stored in the storage unit 12, the map generation unit 17 can also update the environmental map based on the feature points newly extracted from the newly acquired camera image (which can also be referred to as new feature points).

[0045] However, in the generation process of the environmental map using the SLAM technology, the more the number of feature points used, the higher the matching accuracy between the environmental map and the camera image, and the vehicle position can be estimated with higher accuracy. However, if a large number of feature points are extracted from near to far from the vehicle in the camera image, the processing load will become too large. Therefore, it is preferable to preferentially select the feature points of the ground objects important on the map in the camera image over the feature points of other ground objects, thereby suppressing the total number of feature points used in the processing.

[0046] In the embodiment, according to the algorithm of the SLAM technology, the same feature point is tracked among multiple frames of camera images, and the three-dimensional position is calculated for this feature point. At this time, the distance between the vehicle extracted from the camera image and the feature point (object) is obtained, and the number of feature points is adjusted to such an extent that the number of feature points on the object close to the vehicle and the number of feature points on the object far from the vehicle do not deviate more than a necessary degree.

[0047] A more detailed description of the image processing device that performs the above processing will be given.

[0048] Figure 2 It is a block diagram showing the main part configuration of the image processing device 60 of the embodiment. The image processing device 60 is used for controlling the driving operation of the vehicle and forms Figure 1 a part of the vehicle control system 100. As Figure 2 shown, the image processing device 60 has a controller 10, a camera 1a, a radar 1b, and a lidar 1c.

[0049] The camera 1a forms Figure 1 a part of the external sensor group 1. The camera 1a can be either a monocular camera or a stereo camera, and shoots the surroundings of the vehicle. The camera 1a is installed, for example, at a specified position in the front of the vehicle, continuously shoots the front space of the vehicle at a specified frame rate, and sequentially outputs the frame image data (simply referred to as camera image) as detection information to the controller 10.

[0050] Figure 3AThis is a diagram showing an example of a camera image of a certain frame acquired by camera 1a. The camera image IM includes other vehicles V1 traveling in front of the vehicle, other vehicles V2 traveling in the right lane of the vehicle, traffic signal SG around the vehicle, pedestrians PE, traffic signs TS1, TS2, buildings BL1, BL2, BL3 around the vehicle, outer lane lines OL, lane boundary lines SL, etc.

[0051] Figure 2 The radar 1b is mounted on the vehicle and detects other vehicles, obstacles, etc. around the vehicle by irradiating electromagnetic waves and detecting reflected waves. The radar 1b outputs the detected values (detection data) as detection information to the controller 10. The lidar 1c is mounted on the vehicle and measures scattered light for omnidirectional irradiation light of the vehicle to detect the distance from the vehicle to surrounding obstacles. The lidar 1c outputs the detected values (detection data) as detection information to the controller 10.

[0052] The controller 10 includes an arithmetic unit 11 and a storage unit 12. The arithmetic unit 11 has an information acquisition unit 141, an extraction unit 142, a selection unit 143, an adjustment unit 144, a calculation unit 171, a generation unit 172, and a vehicle position recognition unit 13 as functional structures.

[0053] The information acquisition unit 141, extraction unit 142, selection unit 143, and adjustment unit 144 are included, for example, in Figure 1 the external recognition unit 14.

[0054] The calculation unit 171 and generation unit 172 are included, for example, in Figure 1 the map generation unit 17.

[0055] The information acquisition unit 141 acquires information for controlling the driving operation of the vehicle from the storage unit 12. More specifically, the information acquisition unit 141 reads out landmark information included in the environmental map from the storage unit 12, and further acquires information indicating the position of the road markings of the road on which the vehicle is traveling and the extending direction of these road markings (hereinafter referred to as road marking information) from the landmark information.

[0056] It should be noted that when the road marking information does not include information indicating the extending direction of the road markings, the information acquisition unit 141 can also calculate the extending direction of these road markings based on the position of the road markings. In addition, information indicating the position and extending direction of the road markings of the road on which the vehicle is traveling can also be acquired from the road map information, white line map (information indicating the position of road markings such as white and yellow) stored in the storage unit 12, etc.

[0057] The extraction unit 142 extracts from the camera image IM acquired by the camera 1a (in Figure 3AAs exemplified above, edges showing the outline of an object are extracted, and feature points are extracted using the edge information. As described above, the feature points are, for example, intersections of edges. Figure 3B is a diagram exemplifying feature points of a camera image IM based on Figure 3A The black circles in the diagram represent the feature points.

[0058] It should be noted that the extraction unit 142 of the embodiment extracts stationary objects included in the camera image IM as objects for extracting feature points, and excludes moving objects from the objects for extracting feature points. This is because, as described above, important ground objects on the map, in other words, ground objects useful for estimating one's own position and map generation (tall trees, traffic lights, signs located above roads, etc.) are stationary objects, not moving objects.

[0059] The identification of stationary objects and moving objects can be performed as follows, for example. The image data of stationary objects (in other words, objects fixed to the ground) is consistent in position on the environmental map between frames of the camera image IM, and thus is an object for extracting feature points. On the other hand, since the image data of moving objects is not consistent in position on the environmental map between frames, it is excluded from the objects for extracting feature points.

[0060] The selection unit 143 selects feature points for calculating three-dimensional positions from the feature points extracted by the extraction unit 142. For example, feature points that are unique and easy to distinguish from other feature points and remain in the adjustment process of the following adjustment unit 144 are selected.

[0061] The adjustment unit 144 performs an adjustment process for adjusting the number of feature points as described below.

[0062] First, the adjustment unit 144 divides the camera image IM into a plurality of regions. Figure 4A is a diagram exemplifying a camera image IM divided into 30 regions in a 6×5 horizontal and vertical quadrilateral shape as an example. The numbers recorded in each region represent the region ID.

[0063] Next, the adjustment unit 144 calculates the average value of the distances to the feature points extracted in each region (referred to as the average distance). Figure 4B is a schematic diagram showing the average distance of each region in a certain frame of the camera image IM. The horizontal axis represents the average distance, and the vertical axis represents the region ID. Figure 4B The average distance and region ID shown are Figure 3A and 3B The average distance and region ID of a camera image different from the camera image IM shown. In addition, the region ID may also be referred to as the grid ID.

[0064] The adjustment unit 144 reduces the number of feature points in regions with a relatively large average distance to the feature points (the surface of the object) to be less than the number of feature points in regions with a relatively small average distance in multiple regions. In other words, the feature points in the region with a relatively short average distance to the feature points are selected as the target to be retained more than the feature points in the region with a relatively long average distance.

[0065] In Figure 4B In the example shown, the average distances of region IDs 1 to 6, region IDs 10 to 12, and region IDs 16 to 18 are shorter than the average distances of other region IDs. Therefore, by reducing the number of feature points in region IDs 7 to 9, region IDs 13 to 15, and region IDs 19 to 30, the feature points in region IDs 1 to 6, region IDs 10 to 12, and region IDs 16 to 18 are retained as the selection target more.

[0066] The adjustment unit 144 also retains the feature points on the object that extracts more feature points in each region as the selection target more than the feature points on other objects in the region. In other words, in each region, the number of feature points on the object with a smaller number of extracted feature points is reduced.

[0067] It should be noted that when the number of feature points extracted from the camera image IM is less than or equal to a predetermined specified number, the adjustment of the number of feature points for the camera image IM of this frame can also be omitted by the adjustment unit 144.

[0068] The external recognition unit 14 performs the acquisition of information by the above information acquisition unit 141, the extraction of feature points by the extraction unit 142, the adjustment of the number of feature points by the adjustment unit 144, and the selection of feature points by the selection unit 143 on each frame of the camera image IM obtained by the camera 1a.

[0069] The distance information from the camera 1a to the feature points (the surface of the object) is, for example, estimated by the external recognition unit 14 using machine learning (such as DNN (Deep Neural Network)) technology based on the position of the object imaged on the camera image IM in the image, to estimate the distance in the depth direction from the camera 1a to the object including the feature points. It should be noted that the distance from the vehicle to the object can also be calculated based on the detection values of the radar 1b and the lidar 1c.

[0070] In addition, as described later, the vehicle position recognition unit 13 can also estimate the distance from the vehicle to the object on the environmental map based on the positions of the feature points of the landmarks imaged on the camera image IM.

[0071] Figure 2The calculation unit 171 calculates the three-dimensional position of the feature points while estimating the position and orientation of the camera 1a, so that the same feature point converges to one point among the camera images IM of multiple frames. The calculation unit 171 calculates the three-dimensional positions of a plurality of different feature points selected by the selection unit 143 respectively.

[0072] The generation unit 172 uses the three-dimensional positions of a plurality of different feature points calculated by the calculation unit 171 to generate an environmental map composed of three-dimensional point cloud data including information on each three-dimensional position.

[0073] The own vehicle position recognition unit 13 estimates the position of the own vehicle on the environmental map based on the environmental map stored in the storage unit 12.

[0074] First, the own vehicle position recognition unit 13 estimates the position of the own vehicle in the vehicle width direction. Specifically, the own vehicle position recognition unit 13 uses machine learning techniques to recognize the road markings included in the newly acquired camera image IM by the camera 1a. The own vehicle position recognition unit 13 recognizes the position and extension direction of the road markings included in the camera image IM on the environmental map based on the road marking information obtained from the landmark information included in the environmental map stored in the storage unit 12. Then, the own vehicle position recognition unit 13 estimates the relative position relationship (position relationship on the environmental map) between the own vehicle and the road markings in the vehicle width direction based on the position and extension direction of the road markings on the environmental map. In this way, the position of the own vehicle in the vehicle width direction on the environmental map is estimated.

[0075] Next, the own vehicle position recognition unit 13 estimates the position of the own vehicle in the traveling direction. Specifically, the own vehicle position recognition unit 13 recognizes a landmark (such as a building BL1) from the newly acquired camera image IM ( Figure 3A ) by processing such as pattern matching, and recognizes the feature points on the landmark from the feature points extracted by the extraction unit 142. Further, the own vehicle position recognition unit 13 estimates the distance in the traveling direction from the own vehicle to the landmark based on the position of the feature points of the landmark imaged in the camera image IM. It should be noted that the distance from the own vehicle to the landmark can also be calculated based on the detection values of the radar 1b and the lidar 1c.

[0076] The own vehicle position recognition unit 13 searches for the feature points corresponding to the above-mentioned landmark in the environmental map stored in the storage unit 12. In other words, it recognizes the feature points that match the feature points of the landmark recognized from the newly acquired camera image IM from among the multiple feature points (point cloud data) constituting the environmental map.

[0077] Next, the own vehicle position recognition unit 13 estimates the position of the own vehicle in the traveling direction on the environmental map based on the position of the feature points on the environmental map corresponding to the feature points of the landmark and the distance in the traveling direction from the own vehicle to the landmark.

[0078] As described above, the vehicle position recognition unit 13 recognizes the position of the vehicle on the environmental map based on the estimated position of the vehicle on the environmental map in the vehicle width direction and the traveling direction.

[0079] The storage unit 12 stores the information of the environmental map generated by the generation unit 172. In addition, the storage unit 12 stores the information indicating the traveling locus of the vehicle. The traveling locus is represented, for example, by the position of the vehicle on the environmental map recognized by the vehicle position recognition unit 13 during traveling.

[0080] <Description of the flowchart>

[0081] Refer to Figure 5A and Figure 5B The flowchart description of Figure 2 is an example of the process executed by the controller 10 of Figure 5A showing the generation process of the environmental map, which starts, for example, in the manual driving mode and is repeatedly executed at a specified cycle. Figure 5B Showing Figure 5A the details of step S30 of

[0082] In Figure 5A step S10, the controller 10 acquires the camera image IM as detection information from the camera 1a and proceeds to step S20.

[0083] In step S20, the controller 10 extracts feature points from the camera image IM by the extraction unit 142 and proceeds to step S30.

[0084] In step S30, the controller 10 selects feature points by the selection unit 143 and proceeds to step S40.

[0085] In step S40, the controller 10 calculates the three-dimensional positions of a plurality of different feature points respectively by the calculation unit 171 and proceeds to step S50.

[0086] In step S50, the controller 10 generates an environmental map composed of three-dimensional point cloud data including the information of the three-dimensional positions of a plurality of different feature points by the generation unit 172 and proceeds to step S60.

[0087] In step S60, when the controller 10 recognizes that the position where the vehicle travels is on the past traveling locus, it corrects the information of the three-dimensional positions included in the environmental map through loop closing processing and proceeds to step S70.

[0088] Briefly describe the closed-loop process. Generally speaking, in SLAM technology, since the position of the vehicle is recognized while the vehicle is moving, errors will accumulate. For example, when the vehicle drives along a road in the shape of a "square", due to the accumulated errors, the positions of the starting point and the ending point will become inconsistent. Therefore, when it is recognized that the position where the vehicle is driving is on the past driving trajectory, a closed-loop process is performed, that is, the position of the vehicle recognized using the feature points (referred to as new feature points) extracted from the camera image newly obtained at the same driving location as in the past and the position of the vehicle recognized in the past using the feature points extracted from the camera image obtained during past driving are set to the same coordinates.

[0089] In step S70, the controller 10 records the information of the environmental map in the storage unit 12 and ends the Figure 5A processing.

[0090] In Figure 5B step S31, the adjustment unit 144 divides the camera image IM into multiple regions and proceeds to step S32.

[0091] In step S32, the adjustment unit 144 calculates the average distance, which is the average of the distances to the feature points extracted in each region, and proceeds to step S33.

[0092] As described above, the distance to the feature point can be presumed to be the distance in the depth direction from the camera 1a to the surface of the object corresponding to the feature point.

[0093] In addition, the distance calculated based on the detection values of the radar 1b and the lidar 1c can also be obtained.

[0094] Furthermore, the vehicle position recognition unit 13 can also obtain the distance presumed on the environmental map based on the positions of the feature points of the landmarks imaged on the camera image IM.

[0095] In step S33, the adjustment unit 144 reduces the feature points in the regions with a far average distance and proceeds to step S34. As described above, the feature points in the regions with a shorter average distance to the feature points are selected as the objects to be retained, and more are retained than the feature points in the regions with a longer average distance.

[0096] In step S34, the adjustment unit 144 retains more feature points on the object with more feature points extracted and proceeds to step S35. As described above, in each region, the number of feature points on the object with a smaller number of extracted feature points is reduced.

[0097] In step S35, the selection unit 143 selects the feature points remaining in each outlier after being adjusted by the adjustment unit 144 and ends the Figure 5B processing.

[0098] By adopting the embodiments described above, the following operational effects can be obtained.

[0099] (1) The image processing device 60 is an image processing device for detecting the external situation of the vehicle based on the camera image IM as image information obtained by the camera 1a as a detector mounted on the vehicle, and includes: an extraction unit 142 that extracts feature points of an object included in the camera image IM; a selection unit 143 that selects feature points to be processed from among the plurality of feature points extracted by the extraction unit 142; and an adjustment unit 144 that divides the camera image IM into a plurality of grid-like regions as an example, and adjusts the number of feature points selected in each region according to the distance to the object included in the plurality of regions.

[0100] With such a configuration, by making the number of feature points in each region of the camera image IM appropriate according to the distance to the object, the total number of feature points used in the processing can be suppressed. Therefore, for example, the processing load when calculating the three-dimensional position of the feature points is reduced, and the generation of an environmental map required for safe vehicle control can be performed quickly.

[0101] (2) Based on the image processing device 60 in (1) above, the adjustment unit 144 makes an adjustment to retain the feature points on the object from which the extraction unit 142 extracts more feature points.

[0102] With such a configuration, by using the points on the more characteristic object in the processing, the processing can be performed based on more reliable information in the camera image IM.

[0103] (3) Based on the image processing device 60 in (1) or (2) above, the adjustment unit 144 further has a function as a distance measurement unit that calculates the distance to the object based on the camera image IM, calculates the average distance from the vehicle position to the plurality of feature points for each region according to the calculation result of the distance, and adjusts the number of feature points in such a way that the number of feature points in the region with a longer average distance is reduced.

[0104] Generally, compared with the feature points near the vehicle, the feature points far from the vehicle have less influence on driving. Therefore, by reducing the number of feature points in the region with a longer average distance compared with the region with a shorter average distance, the total number of feature points in the camera image IM used in the processing can be appropriately suppressed. Thereby, for example, the processing load when calculating the three-dimensional position of the feature points can be reduced, and the generation of an environmental map required for safe vehicle control can be performed quickly.

[0105] (4) Based on the image processing device 60 in (1) to (3) above, the extraction unit 142 sets the stationary object included in the camera image IM as the extraction target of the feature points, and excludes the moving object from the extraction target of the feature points.

[0106] With such a configuration, for example, even an object photographed at the upper part of the screen of the camera image IM becomes an object for extracting feature points, such as ground objects (e.g., tall trees, signal lights, signs located above the road, etc.) that are useful for estimating the vehicle's own position and map generation. As a result, the generation of an environmental map required for safe vehicle control can be appropriately performed.

[0107] (5) Based on the image processing device 60 in the above (1) to (4), it further includes: a storage unit 12 that stores multiple frames of camera images IM in units of frames; a map generation unit 17 that, as a search unit, searches for a group of the same feature points commonly included in the multiple frames of camera images IM stored in the storage unit 12; and a vehicle position recognition unit 13 that, as an estimation unit, estimates the vehicle's own position based on the group of feature points searched by the map generation unit 17.

[0108] With such a configuration, it is possible to search for a group of the same feature points commonly included in the camera image IM between different frames, and accurately estimate the vehicle's own position.

[0109] (6) Based on the image processing device 60 in the above (5), when the reliability of the estimated vehicle's own position decreases by a specified number or more, the vehicle position recognition unit 13 interrupts the estimation of the vehicle's own position. More specifically, when the reliability of the vehicle's own position estimation decreases by a specified number or more between frames, or when the reliability of the vehicle's own position estimation decreases between each of a specified number of consecutive frames, the estimation of the vehicle's own position is interrupted.

[0110] With such a configuration, in the case where the reliability of the vehicle's own position estimation decreases, by interrupting the vehicle's own position estimation itself, it is possible to reduce unnecessary processing compared to the case where it is not interrupted.

[0111] (7) Based on the image processing device 60 in the above (5) or (6), the vehicle position recognition unit 13 interrupts the estimation of the vehicle's own position according to the vehicle's condition.

[0112] With such a configuration, in the case where it is foreseen that the vehicle is difficult to continue driving, for example, due to an accident such as a flat tire or deterioration of the road surface condition, by interrupting the vehicle's own position estimation itself, it is possible to reduce unnecessary processing compared to the case where it is not interrupted.

[0113] (8) Based on the image processing apparatus 60 in the above (1) or (2), it further includes: a calculation unit 171 that calculates the three-dimensional positions of the same feature points commonly included in the image information of multiple frames for multiple different feature points selected by the selection unit 143; a generation unit 172 and a vehicle position recognition unit 13, which, as arithmetic units, perform map generation and vehicle position estimation based on the three-dimensional positions of multiple different feature points calculated by the calculation unit 171. The adjustment unit 144 calculates the average distance from the vehicle position to multiple feature points for each area according to the calculation result of the vehicle position recognition unit 13, and adjusts the number of feature points in such a way that the number of feature points decreases in areas with longer average distances.

[0114] Generally speaking, compared with the feature points near the vehicle, the feature points far from the vehicle have less influence on driving. Therefore, by reducing the number of feature points in areas with longer average distances compared to areas with shorter average distances, the total number of feature points in the camera image IM used in the processing can be appropriately suppressed. Thus, for example, the processing load when calculating the three-dimensional positions of feature points can be reduced, and the generation of the environmental map required for safe vehicle control can be performed quickly.

[0115] The above-described embodiment can be deformed into various forms. Hereinafter, deformation examples will be described.

[0116] (Modification Example 1)

[0117] Figure 4A The number of areas exemplified is an example and can be appropriately changed. In addition, an example of dividing into multiple quadrilateral areas (which can also be called grid-like) is described, but it can also be divided into multiple hexagonal areas (which can also be called honeycomb-like).

[0118] (Modification Example 2)

[0119] In the above description, an example of calculating the average distance from the vehicle position to multiple feature points in each area and adjusting the number of feature points in such a way that the number of feature points decreases in areas with longer average distances is described.

[0120] Instead of the average distance from the vehicle position to multiple feature points in the area, the number of feature points can also be adjusted according to the median value of the distances from the vehicle position to multiple feature points in the area, in such a way that the number of feature points decreases in areas with larger median values of the distances.

[0121] (Modification Example 3)

[0122] In addition, in the above description, an example of adjusting the number of feature points is described in such a way that the number of feature points decreases in a region that is farther from the vehicle position. In addition to this, in a region near the vehicle position, the number of feature points on an object close to the vehicle position can be increased to such an extent that the deviation from the number of feature points on an object far from the vehicle position does not exceed a necessary level. An object for which the number of feature points should be increased near the vehicle position is, for example, an important ground object on the map.

[0123] That is, regardless of the average distance from the vehicle position to each region of the plurality of feature points, the adjustment unit 144 adjusts the number of feature points in such a way as to increase the number of feature points on an object that is predetermined as an important ground object on the map.

[0124] With this configuration, the detection accuracy of the feature points near the vehicle position is improved, and the vehicle position recognition unit 13 can accurately recognize the vehicle position.

[0125] The above description is ultimately just an example, and the present invention is not limited by the above-described embodiments and modified examples as long as the features of the present invention are not impaired. One or more of the above-described embodiments and modified examples can be arbitrarily combined, and the modified examples can also be combined with each other.

[0126] By adopting the present invention, it is possible to appropriately detect external information required for safe vehicle control.

[0127] The present invention has been described above in conjunction with preferred embodiments, but those skilled in the art should understand that various modifications and changes can be made without departing from the scope of disclosure of the claims.

Claims

1. An image processing device, which is an image processing device (60) for detecting the external condition of a vehicle based on image information obtained by a detector (1a) mounted on the vehicle, characterized in that, Comprising: An extraction unit (142) that extracts feature points of an object included in the image information; A selection unit (143) that selects feature points to be processed from among the plurality of feature points extracted by the extraction unit (142); An adjustment unit (144) that divides the image information into a plurality of regions and adjusts the number of feature points selected in each region according to the distance to the object included in the plurality of regions.

2. The image processing apparatus according to claim 1, It is characterized in that. The adjustment unit (144) performs the adjustment in such a way as to retain the feature points on the object on which more feature points are extracted by the extraction unit (142).

3. The image processing apparatus according to claim 1 or 2, characterized in that, It further comprises a distance measurement unit that calculates the distance to the object according to the image information, The adjustment unit (144) calculates the average distance from the vehicle position to the plurality of feature points for each of the regions according to the calculation result of the distance measurement unit, and adjusts the number of feature points in such a way that the number of feature points is reduced more in the region where the average distance is longer.

4. The image processing apparatus according to claim 1 or 2, characterized in that, It further comprises a distance measurement unit that calculates the distance to the object according to the image information, The adjustment unit (144) calculates the median of the distances from the vehicle position to the plurality of feature points for each of the regions according to the calculation result of the distance measurement unit, and adjusts the number of feature points in such a way that the number of feature points is reduced more in the region where the median is larger.

5. The image processing apparatus according to claim 1, characterized in that, The extraction unit (142) uses the stationary object included in the image information as the extraction target of the feature points and excludes the moving object from the extraction target of the feature points.

6. The image processing apparatus according to claim 1, wherein Comprising: A storage unit (12) that stores multiple frames of the image information in units of frames; A search unit that searches for a group of the same feature points commonly included in the multiple frames of the image information stored in the storage unit (12); And An estimation unit that estimates the vehicle position according to the group of feature points searched by the search unit.

7. The image processing apparatus according to claim 6, characterized in that, When the reliability of the estimated vehicle position decreases by a specified number or more, the estimation unit interrupts the vehicle position estimation.

8. The image processing apparatus according to claim 7, characterized in that, The estimation unit interrupts the vehicle position estimation according to the vehicle condition.

9. The image processing apparatus according to claim 1 or 2, wherein It further comprises: A calculation unit (171) that calculates the three-dimensional positions of the same feature points commonly included in the multiple frames of the image information for each of the multiple different feature points selected by the selection unit; An arithmetic unit that generates a map and estimates the vehicle position according to the three-dimensional positions of the multiple different feature points calculated by the calculation unit (171), The adjustment unit (144) calculates, for each of the regions, an average distance from the vehicle position to a plurality of the feature points according to the calculation result of the arithmetic unit, and adjusts the number of the feature points such that the number of the feature points is reduced for a region having a longer average distance.

10. The image processing apparatus according to claim 3, wherein the adjustment unit (144) adjusts the number of the feature points such that the number of the feature points on a predetermined object is increased regardless of the average distance of each of the regions from the vehicle position to the plurality of the feature points.

Images