Entrance detection device, vehicle control device, entrance detection method, and entrance detection program

Abstract

The system easily detects entrances facing the road the vehicle is traveling on. [Solution] The entrance detection device 60 includes a camera 1a that captures an area in the direction of travel of the vehicle as it travels on the road, a calculation unit 111 that acquires a distance map in which distance information indicating the distance to an object included in the imaging area of ​​the camera 1a is recorded for each pixel of the image captured by the camera 1a, an extraction unit 112 that extracts pixels from the image captured where the difference in distance information value with a comparison pixel that is a predetermined number of pixels away in the horizontal axis direction of the image captured is greater than or equal to a predetermined value, and a detection unit 113 that detects an entrance facing the road based on the pixels extracted by the extraction unit 112.

Description

Technical Field

[0001] The present invention relates to an entrance detection device that detects an entrance facing a traveling road on which a host vehicle travels, a vehicle control device, an entrance detection method, and an entrance detection program.

Background Art

[0002] Conventionally, as this type of device, there is known a device that performs sensor fusion processing on detection results from a plurality of sensors such as a camera and a lidar to recognize the positions of objects around a moving body (see, for example, Patent Document 1). The device described in Patent Document 1 detects, as a high-risk area, the vicinity of the end of an obstacle existing along the extending direction of a road.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, if a high-risk area is detected using sensor fusion processing as in the device described in Patent Document 1, there is a risk of increasing the processing load of the device.

Means for Solving the Problems

[0005] An entrance detection device according to one aspect of the present invention comprises: a camera that captures an image of the area in the direction of travel of the vehicle traveling on a road; an acquisition unit that acquires a distance map in which distance information indicating the distance to an object included in the camera's imaging area is recorded for each pixel of the camera's captured image; an extraction unit that extracts pixels from the captured image in which the difference in distance information values ​​with a comparison pixel located a predetermined number of pixels away in the horizontal axis direction of the captured image is greater than or equal to a predetermined value; and an entrance detection unit that detects an entrance facing the road based on the pixels extracted by the extraction unit.

[0006] Another aspect of the present invention provides a vehicle control device comprising the above-mentioned entrance detection device, a driving actuator, and a driving control unit. The entrance detection unit further outputs entrance information indicating the detection result of the entrance, and the driving control unit controls the actuator based on the entrance information.

[0007] A further different aspect of the present invention is an entrance detection method which includes the steps of: obtaining a distance map in which distance information indicating the distance to an object included in the imaging area of ​​a camera that images the area in the direction of travel of the vehicle traveling on a road is recorded for each pixel of the camera's image; extracting pixels from the image in which the difference in distance information values ​​with comparison pixels located a predetermined number of pixels away in the horizontal axis direction of the image is greater than or equal to a predetermined value; and detecting an entrance facing the road based on the extracted pixels.

[0008] An entrance detection program in another aspect of the present invention causes a computer to perform the following steps: acquire a distance map, which records distance information indicating the distance to an object included in the imaging area of ​​a camera that images the area in the direction of travel of the vehicle traveling on a road, for each pixel of the image captured by the camera; extract pixels from the image captured where the difference in distance information value between the image and a comparison pixel located a predetermined number of pixels away in the horizontal axis direction of the image is greater than or equal to a predetermined value; and detect an entrance facing the road based on the extracted pixels. [Effects of the Invention]

[0009] According to the present invention, the entrance facing the road on which the vehicle is traveling can be easily detected. [Brief explanation of the drawing]

[0010] [Figure 1] A block diagram schematically showing the overall configuration of a vehicle control system according to an embodiment of the present invention. [Figure 2A] A diagram illustrating the visibility of the entrance facing the road on which the vehicle is traveling. [Figure 2B] A diagram illustrating the visibility of the entrance facing the road on which the vehicle is traveling. [Figure 3] A block diagram showing the main components of a vehicle control device according to an embodiment of the present invention. [Figure 4A] A diagram illustrating the extraction of candidate pixels for occlusion. [Figure 4B] A diagram illustrating the extraction of candidate pixels for occlusion. [Figure 5A] A diagram illustrating an object that does not affect the visibility of the entrance. [Figure 5B] A diagram illustrating an object that does not affect the visibility of the entrance. [Figure 6] A diagram illustrating the relationship between a predetermined threshold and a judgment value. [Figure 7] A flowchart showing an example of the processing performed by the controller's CPU in Figure 3. [Modes for carrying out the invention]

[0011] Embodiments of the invention will be described below with reference to the drawings. The entrance detection device according to the embodiment of the present invention can be applied to a vehicle having an automatic driving function, i.e., an autonomous vehicle. The vehicle to which the entrance detection device according to this embodiment is applied may be referred to as "the vehicle" to distinguish it from other vehicles. The vehicle may be an engine vehicle having an internal combustion engine as a driving source, an electric vehicle having a drive motor as a driving source, or a hybrid vehicle having both an engine and a drive motor as driving sources. The vehicle can be driven not only in an autonomous driving mode that does not require driver operation, but also in a manual driving mode with driver operation.

[0012] First, the general configuration of the vehicle involved in autonomous driving will be described. Figure 1 is a block diagram that schematically shows the overall configuration of the vehicle control system 100 of the vehicle having an entrance detection device according to this embodiment. As shown in Figure 1, the vehicle control system 100 mainly consists of a controller 10, a group of external sensors 1 and 2, an input / output device 3, a positioning unit 4, a map database 5, a navigation device 6, a communication unit 7, and an actuator AC for driving.

[0013] External sensor group 1 is a general term for multiple sensors (external sensors) that detect external conditions, which are information about the surroundings of the vehicle. For example, external sensor group 1 includes a lidar that measures the distance from the vehicle to surrounding obstacles by measuring the reflected light from the light illuminating the vehicle in all directions, a radar that detects other vehicles and obstacles around the vehicle by irradiating electromagnetic waves and detecting the reflected waves, and a camera mounted on the vehicle that has an image sensor such as a CCD or CMOS and captures images of the area around the vehicle (front, rear, and sides).

[0014] Internal sensor group 2 is a collective term for multiple sensors (internal sensors) that detect the vehicle's driving state. For example, internal sensor group 2 includes a vehicle speed sensor that detects the vehicle's speed (driving speed), acceleration sensors that detect the vehicle's acceleration in the longitudinal direction and acceleration in the lateral direction (lateral acceleration), a rotation speed sensor that detects the rotation speed of the driving power source, and a yaw rate sensor that detects the rotational angular velocity of the vehicle's center of gravity around the vertical axis. Sensors that detect the driver's driving operations in manual driving mode, such as operation of the accelerator pedal, brake pedal, and steering wheel, are also included in internal sensor group 2.

[0015] The input / output device 3 is a general term for devices that receive commands from the driver or output information to the driver. For example, the input / output device 3 includes various switches for the driver to input various commands by operating an operation member, a microphone for the driver to input commands by voice, a display for providing information to the driver via a display image, a speaker for providing information to the driver by voice, and the like.

[0016] The positioning unit (GNSS unit) 4 has a positioning sensor that receives positioning signals transmitted from positioning satellites. The positioning satellites are artificial satellites such as GPS satellites and quasi-zenith satellites. The positioning unit 4 measures the current position (latitude, longitude, altitude) of the host vehicle using the positioning information received by the positioning sensor.

[0017] The map database 5 is a device that stores general map information used in the navigation device 6, and is constituted by, for example, a hard disk or a semiconductor element. The map information includes road position information, road shape (curvature, etc.) information, intersection and branch point position information. Note that the map information stored in the map database 5 is different from the high-precision map information stored in the storage unit 12 of the controller 10.

[0018] The navigation device 6 is a device that searches for a target route on the road to a destination input by the driver and provides guidance along the target route. The input of the destination and the guidance along the target route are performed via the input / output device 3. The target route is calculated based on the current position of the host vehicle measured by the positioning unit 4 and the map information stored in the map database 5. It is also possible to measure the current position of the host vehicle using the detection values of the external sensor group 1 and calculate the target route based on this current position and the high-precision map information stored in the storage unit 12.

[0019] The communication unit 7 communicates with various servers (not shown) via a network including wireless communication networks such as the Internet and mobile phone networks, and obtains map information, driving history information, and traffic information from the servers periodically or at arbitrary times. In addition to obtaining driving history information, the communication unit 7 may also transmit its own vehicle's driving history information to the server. The network includes not only public wireless communication networks but also closed communication networks established for each designated management area, such as wireless LAN, Wi-Fi (registered trademark), Bluetooth (registered trademark), etc. The acquired map information is output to the map database 5 and the storage unit 12, and the map information is updated.

[0020] Actuator AC is a drive actuator used to control the movement of the vehicle. When the drive source is an engine, actuator AC includes a throttle actuator that adjusts the opening degree (throttle opening) of the engine's throttle valve. When the drive source is a drive motor, the drive motor is included in actuator AC. Brake actuators that operate the vehicle's braking system and steering actuators that drive the steering system are also included in actuator AC.

[0021] The controller 10 is comprised of an electronic control unit (ECU). More specifically, the controller 10 includes a computer comprising an arithmetic unit 11 such as a CPU (microprocessor), a storage unit 12 such as ROM or RAM, and other peripheral circuits (not shown) such as an I / O interface. While multiple ECUs with different functions, such as an engine control ECU, a drive motor control ECU, and a braking system ECU, can be provided separately, in Figure 1, for convenience, the controller 10 is shown as a collection of these ECUs.

[0022] The memory unit 12 stores highly accurate and detailed map information (referred to as high-precision map information). This high-precision map information includes road location information, road shape information (curvature, etc.), road gradient information, intersection and branching point location information, type and location information of road markings such as white lines, number of lanes, lane width and location information for each lane (information on the center position of the lane and the boundary lines of the lane positions), location information of landmarks (buildings, traffic lights, signs, etc.) as markers on the map, and road surface profile information such as road surface irregularities. In this embodiment, the center line, lane boundary lines, road edge lines, etc. are collectively referred to as road markings. The high-precision map information stored in the memory unit 12 includes map information acquired from outside the vehicle via the communication unit 7 (referred to as external map information) and maps created by the vehicle itself using detection values ​​from the external sensor group 1 or detection values ​​from the external sensor group 1 and the internal sensor group 2 (referred to as internal map information).

[0023] External map information is, for example, map information obtained via a cloud server (referred to as a cloud map), while internal map information is, for example, map information consisting of 3D point cloud data generated by mapping using technologies such as SLAM (Simultaneous Localization and Mapping) (referred to as an environmental map). External map information is shared between the vehicle and other vehicles, whereas internal map information is the vehicle's own map information (for example, map information owned solely by the vehicle). For roads not yet traveled by the vehicle, newly constructed roads, etc., the vehicle itself creates the environmental map. Internal map information may also be provided to the server device or other vehicles via the communication unit 7. In addition to the high-precision map information described above, the storage unit 12 also stores information such as the vehicle's travel trajectory, various control programs, and thresholds used in the programs.

[0024] The calculation unit 11 has the following functional configuration: a vehicle position recognition unit 13, an external environment recognition unit 14, an action plan generation unit 15, a driving control unit 16, and a map generation unit 17.

[0025] The vehicle position recognition unit 13 recognizes (or estimates) the vehicle's position on the map (vehicle position) based on the vehicle's position information obtained by the positioning unit 4 and the map information in the map database 5. The vehicle position may also be recognized (estimated) using high-precision map information stored in the memory unit 12 and surrounding information of the vehicle detected by the external sensor group 1, thereby enabling high-precision recognition of the vehicle position. The vehicle's movement information (direction of movement, distance traveled) can also be calculated based on the detection values ​​of the internal sensor group 2, and the vehicle position can be recognized accordingly. Furthermore, when the vehicle's position can be measured by sensors installed on or beside the road, the vehicle position can also be recognized by communicating with those sensors via the communication unit 7.

[0026] The external environment recognition unit 14 recognizes the external conditions around the vehicle based on signals from the external sensor group 1, including LiDAR, radar, and cameras. For example, it recognizes the position, speed, and acceleration of surrounding vehicles (vehicles in front and behind) traveling around the vehicle, the position of surrounding vehicles that are stopped or parked around the vehicle, and the position and state of other objects. Other objects include signs, traffic lights, road markings such as lane markings and stop lines, buildings, guardrails, utility poles, billboards, pedestrians, and bicycles. The state of other objects includes the color of traffic lights (red, blue, yellow), the speed and direction of pedestrians and cyclists, etc. Some of the stationary objects among the other objects constitute landmarks that serve as indicators of location on a map, and the external environment recognition unit 14 also recognizes the position and type of these landmarks.

[0027] The action plan generation unit 15 generates a driving trajectory (target trajectory) for the vehicle from the present time to a predetermined time in advance, based on, for example, the target route calculated by the navigation device 6, the high-precision map information stored in the memory unit 12, the vehicle's position recognized by the vehicle position recognition unit 13, and the external conditions recognized by the external environment recognition unit 14. If there are multiple candidate trajectories for the target trajectory on the target route, the action plan generation unit 15 selects the optimal trajectory from among them that meets criteria such as complying with laws and regulations and driving efficiently and safely, and sets the selected trajectory as the target trajectory. The action plan generation unit 15 then generates an action plan corresponding to the generated target trajectory. The action plan generation unit 15 generates various action plans corresponding to overtaking driving to overtake a preceding vehicle, lane change driving to change driving lanes, following driving to follow a preceding vehicle, lane keeping driving to maintain the driving lane without deviating from the driving lane, deceleration driving, or acceleration driving. When generating a target trajectory, the action plan generation unit 15 first determines the driving mode and generates the target trajectory based on the driving mode.

[0028] In automatic driving mode, the driving control unit 16 controls each actuator AC so that the vehicle travels along the target trajectory generated by the action plan generation unit 15. More specifically, in automatic driving mode, the driving control unit 16 calculates the required driving force to obtain the target acceleration per unit time calculated by the action plan generation unit 15, taking into account the driving resistance determined by the road gradient, etc. Then, it provides feedback control to the actuator AC so that the actual acceleration detected by, for example, the internal sensor group 2 becomes the target acceleration. In other words, it controls the actuator AC so that the vehicle travels at the target vehicle speed and target acceleration. In manual driving mode, the driving control unit 16 controls each actuator AC in accordance with driving commands (such as steering operations) from the driver acquired by the internal sensor group 2.

[0029] The map generation unit 17 generates an environmental map of the roads the vehicle has traveled on, as internal map information, using detection values ​​detected by the external sensor group 1 while the vehicle is driving in manual driving mode. For example, it extracts edges and characteristic regions (blobs) that indicate the outlines of objects based on the brightness and color information of each pixel from multiple frames of camera images acquired by the camera, and extracts feature points using the information of those edges and blobs. Feature points are, for example, the intersections of edges and correspond to the corners of buildings or road signs. The map generation unit 17 calculates the 3D position of a feature point while estimating the camera's position and orientation so that identical feature points converge to a single point across multiple frames of camera images, according to the SLAM technology algorithm. By performing this calculation process for each of the multiple feature points, it generates an environmental map consisting of 3D point cloud data. Alternatively, instead of a camera, data acquired by radar or LiDAR may be used to extract feature points of objects around the vehicle and generate an environmental map.

[0030] The vehicle position recognition unit 13 may perform vehicle position recognition processing based on the environmental map generated by the map generation unit 17 and feature points extracted from the camera image. The vehicle position recognition unit 13 may also perform vehicle position recognition processing in parallel with the map creation processing by the map generation unit 17. The map creation processing and position recognition (estimation) processing are performed simultaneously according to the SLAM technology algorithm. The map generation unit 17 can generate an environmental map not only when driving in manual driving mode but also when driving in automatic driving mode. If an environmental map has already been generated and stored in the storage unit 12, the map generation unit 17 may update the environmental map based on newly extracted feature points from newly acquired camera images.

[0031] Incidentally, there are cases where an entrance facing the road on which the vehicle is traveling cannot be seen from the vehicle due to an obstruction installed adjacent to the entrance. Figures 2A and 2B are diagrams to explain the visibility of an entrance facing the road on which the vehicle is traveling. Figure 2A schematically shows a view from above of the road on which the vehicle 101 is traveling. Figure 2B schematically shows the scenery included in the imaging area of ​​the on-board camera when the vehicle 101 is traveling at position P1 (Figure 2A).

[0032] As shown in Figures 2A and 2B, when a wall WL1 is provided adjacent to an access road AR1 connected to road RD1, the entrance to access road AR1 (the connection point between road RD1 and access road AR1) GW1 is obscured by the wall WL1. As a result, it becomes difficult for the vehicle 101 to recognize the entrance GW1 or pedestrians PT attempting to enter road RD1 from the entrance GW1 based on the camera image.

[0033] When the vehicle 101 is at position P0, that is, when there is a sufficient distance between the vehicle 101 and the entrance GW1, the vehicle 101 can perform avoidance control (brake control and steering control) with ample margin to avoid a pedestrian PT entering road RD1 from entrance GW1, even if it cannot recognize the presence of entrance GW1. Also, when the vehicle 101 is at position P2, that is, when the vehicle 101 is approaching entrance GW1, the vehicle 101 can more easily recognize entrance GW1 and pedestrian PT, so it can perform avoidance control, such as slowing down, at an appropriate timing in preparation for the pedestrian PT entering road RD1.

[0034] On the other hand, when the vehicle 101 is traveling at position P1, it is difficult to recognize the entrance GW1 based on the camera image, and furthermore, because there is not enough distance between the vehicle 101 and the entrance GW1, there is a risk that the start of avoidance control for pedestrian PT entering road RD1 may be delayed, or that avoidance control may not be executed properly. Therefore, in order to address these problems, the vehicle control device is configured as follows in this embodiment.

[0035] Figure 3 is a block diagram showing the main components of the vehicle control device 50 according to this embodiment. This vehicle control device 50 constitutes a part of the vehicle control system 100 shown in Figure 1. As shown in Figure 3, the vehicle control device 50 includes a controller 10, a camera 1a, a vehicle speed sensor 2a, a display 3a, and an actuator AC. The vehicle control device 50 also includes an entrance detection device 60, which constitutes a part of the vehicle control device 50. The entrance detection device 60 detects entrances provided facing the road on which the vehicle is traveling, based on detection data (camera images) from the camera 1a.

[0036] Camera 1a is a monocular camera having an image sensor such as a CCD or CMOS, and constitutes part of the external sensor group 1 in Figure 1. Camera 1a detects the external conditions around the vehicle. Camera 1a is mounted, for example, at a predetermined position on the front of the vehicle, and continuously captures images of the area in the direction of travel of the vehicle (forward space) at a predetermined frame rate, and sequentially outputs frame image data (camera images) as detection information to the controller 10.

[0037] The vehicle speed sensor 2a detects the vehicle speed of the vehicle 101. The vehicle speed sensor 2a constitutes part of the internal sensor group 2 in Figure 1. The display 3a is provided, for example, on the instrument panel inside the vehicle. The display 3a constitutes part of the input / output device 3 in Figure 1. The display 3a displays information such as prompting the driver to operate the steering wheel (hands-on) and information that warns and notifies the driver of switching to manual driving mode.

[0038] The controller 10 includes a calculation unit 11 and a storage unit 12. The calculation unit 11 has a functional configuration of a calculation unit 111, an extraction unit 112, a detection unit 113, an action plan generation unit 15, and a driving control unit 16. The calculation unit 111, the extraction unit 112, and the detection unit 113 are, for example, composed of the external environment recognition unit 14 shown in Figure 1. The calculation unit 111, the extraction unit 112, the detection unit 113, the camera 1a, the vehicle speed sensor 2a, the display 3a, and the storage unit 12 are included in the entrance detection device 60.

[0039] The calculation unit 111 acquires detection data (camera images) from the camera 1a at predetermined intervals. More specifically, the calculation unit 111 acquires camera images at a period T (=1 / F) (seconds) determined by the frame rate F (fps) of the camera 1a.

[0040] The calculation unit 111 generates a distance map in which distance information indicating the distance to each object included in the imaging area of ​​the camera 1a is recorded for each pixel. Specifically, the calculation unit 111 takes the acquired camera image as input and calculates the relative depth from the camera 1a to each object in the imaging area for each pixel by monocular depth estimation. Based on the correspondence between relative depth and actual distance, the calculation unit 111 converts the estimated relative depth to actual distance and generates a distance map based on the distance information obtained by this conversion. Note that the method of generating the distance map is not limited to this, and the distance map may also be generated using distance information obtained based on detection values ​​of a LiDAR or radar, either together with or in place of the camera image. The calculation unit 111 stores the generated distance map in the storage unit 12.

[0041] The extraction unit 112 extracts pixels from the camera image based on the distance map, where the difference in distance information values ​​between a comparison pixel located a predetermined number of pixels away in the horizontal axis direction of the camera image is greater than or equal to a predetermined value (hereinafter referred to as occluding candidate pixels). Figures 4A and 4B are diagrams illustrating the extraction of occluding candidate pixels. Hereinafter, occluding candidate pixels may simply be referred to as candidate pixels or candidate points.

[0042] Figure 4A shows the target pixel TP and the comparison pixel CP that is compared with the target pixel TP. The comparison pixel CP is a pixel located a predetermined number of pixels (N pixels) away from the target pixel TP in the horizontal axis direction (left-right direction in the figure) of the camera image. The predetermined number of pixels N may be adjusted based on the distance information value corresponding to the target pixel TP. Specifically, the larger the distance information value corresponding to the target pixel TP, the smaller the predetermined number of pixels may be. Note that the predetermined number of pixels is set to a value that is at least greater than the judgment value E described later.

[0043] The extraction unit 112 uses each pixel of the camera image as a target pixel TP and searches for target pixel TPs whose distance information value difference from that of a comparison pixel CP is greater than or equal to a predetermined value. More specifically, the extraction unit 112 searches for target pixel TPs whose distance information value is less than or equal to a predetermined value from that of a comparison pixel CP. In this way, candidate pixels are extracted from the camera image.

[0044] Furthermore, to reduce the processing load, the extraction unit 112 may exclude target pixels TP that contain comparison pixels CP in the area corresponding to the sky from the search target. Also, the extraction unit 112 may exclude areas above a predetermined height from the road surface (for example, the height of the vehicle 101) from the search target. In addition, when the vehicle 101 is traveling on a left-hand traffic road RD1, entrances provided on the right side of road RD1 are easily recognized by the vehicle 101, and delays in avoidance control as described above are less likely to occur. Therefore, to further reduce the processing load, the extraction unit 112 may exclude from the search target the area to the right of the road on which the vehicle 101 is traveling, specifically the area to the right of the center position of the road.

[0045] Figure 4B shows a magnified view of a portion of the camera image (near the entrance GW1) acquired when the vehicle 101 was at position P1 (Figure 2A), corresponding to Figure 2B. Figure 4B also schematically superimposes target pixels TP, where the difference in distance information values ​​between the target pixel TP and the comparison pixel CP is greater than or equal to a predetermined value, onto the camera image. As shown in Figure 4B, when the target pixel TP corresponds to the wall WL1 and the comparison pixel CP corresponds to the road surface of the entrance road AR1 or the wall WL2 located further back from the entrance road AR1, a difference of greater than a predetermined value can occur between the distance information values ​​of the two pixels. As a result, the target pixel TP shown in Figure 4B is extracted as a candidate pixel.

[0046] The detection unit 113 detects entrances facing the road on which the vehicle 101 is traveling, or more specifically, entrances that are difficult or impossible to see from the vehicle 101 due to obstructions, based on candidate pixels extracted by the extraction unit 112. The detection of entrances by the detection unit 113 will now be described in detail.

[0047] First, the detection unit 113 detects an obstruction based on candidate pixels extracted by the extraction unit 112. The obstruction targeted for detection by the detection unit 113 is an object that obstructs the view of the camera 1a and reduces the visibility of an entrance provided facing the road on which the vehicle 101 travels, or makes the entrance invisible.

[0048] The detection unit 113 detects a group of candidate pixels that are close to each other in the horizontal or vertical axis direction of the camera image from the multiple candidate pixels extracted by the extraction unit 112, as a group of candidate occluding pixels. Hereinafter, the group of candidate occluding pixels may simply be referred to as a group of candidate pixels or a group of candidate points. In the example in Figure 4B, the pixel group PG1 is detected as a group of candidate pixels. Based on the group of candidate pixels, the detection unit 113 detects an object that is a candidate occluding object (hereinafter referred to as a candidate object).

[0049] Figures 5A and 5B illustrate objects that do not affect the visibility of the entrance. Figure 5A schematically shows the scenery included in the imaging area of ​​camera 1a when the vehicle 101 is traveling on road RD2, which is different from road RD1. On road RD2, an entrance GW2 is provided between a utility pole EP and a wall WL3. Figure 5B shows a magnified view of a portion of the camera image corresponding to Figure 5A (near the entrance GW2).

[0050] In the example shown in Figure 5B, the pixel group PG2 corresponding to the utility pole EP located on the near side of the access road AR2 in the direction of travel is detected as a candidate pixel group. However, narrow objects like utility poles EP do not reduce the visibility of the access entrance connected to them. Similarly, objects with low height, such as hedges, do not reduce the visibility of the access entrance connected to them.

[0051] Therefore, the detection unit 113 determines whether the detected candidate object satisfies predetermined conditions so that objects that do not affect the visibility of the entrance (such as utility poles or bushes) are not detected as obstructions. Specifically, the detection unit 113 determines whether the width and height of the candidate object (the object corresponding to the candidate pixel group PG1) are both greater than a predetermined threshold Th, based on the size of the candidate pixel group PG1 in the horizontal axis direction (left-right direction in the figure) and the vertical axis direction (up-down direction in the figure).

[0052] At this time, the detection unit 113 determines that the width of the candidate object is greater than a predetermined threshold Th when the maximum number of pixels in the horizontal axis direction of the candidate pixel group PG1 is greater than the determination value E. Also, the detection unit 113 determines that the height of the candidate object is greater than a predetermined threshold Th when the maximum number of pixels in the vertical axis direction of the candidate pixel group PG1 is greater than the determination value E.

[0053] Figure 6 is a diagram illustrating the relationship between a predetermined threshold Th and a judgment value E. Characteristic f1 in Figure 6 shows the relationship between the distance from the vehicle 101 to the candidate object and the number of pixels corresponding to the predetermined threshold Th. As shown in characteristic f1, the number of pixels corresponding to the predetermined threshold Th changes depending on the distance from the vehicle 101 to the candidate object. Specifically, as the distance from the vehicle 101 to the candidate object increases, the number of pixels corresponding to the predetermined threshold Th decreases exponentially. Characteristic f2 shows the judgment value E. As shown in characteristic f2, the judgment value E is a value obtained by multiplying the number of pixels corresponding to the predetermined threshold Th by a predetermined reduction ratio (for example, 80%).

[0054] The number of pixels in the candidate pixel group in the horizontal axis direction and vertical direction decreases as the distance from the vehicle 101 to the candidate object increases. Therefore, if the judgment value E is a fixed value, it may not be possible to accurately detect candidate objects that are far away from the vehicle 101. However, as shown in characteristic f2, by adjusting the judgment value E according to the distance from the vehicle 101 to the candidate object, the candidate object can be accurately detected regardless of the distance from the vehicle 101 to the candidate object. Also, as shown in characteristic f2, by setting the judgment value E to a value smaller than the number of pixels corresponding to a predetermined threshold Th, it is possible to suppress the failure to detect candidate objects.

[0055] The detection unit 113 does not detect candidate objects that do not meet the predetermined conditions from among the candidate objects detected from the camera image as occluding objects. In other words, the detection unit 113 detects candidate objects that meet the predetermined conditions from among the candidate objects detected from the camera image as occluding objects.

[0056] Furthermore, in order to suppress false detection of obstructions, the detection unit 113 may detect a candidate object as an obstruction when a predetermined number of candidate objects that satisfy the above predetermined conditions are detected consecutively over a predetermined number of frames (for example, 3).

[0057] Specifically, the detection unit 113 determines whether a candidate object detected from the camera image of the current frame is the same as a candidate object detected from the camera image of the previous frame immediately preceding the current frame. The detection unit 113 then detects candidate objects that have been determined to be the same object across a predetermined number of consecutive frames as occluding objects.

[0058] Here, we will explain how to identify candidate objects between frames. The detection unit 113 estimates (calculates) the distance from the vehicle 101 to the candidate object detected in the camera image of the current frame (hereinafter referred to as the current distance) based on a distance map generated by the calculation unit 111 from the camera image of the current frame. The detection unit 113 also estimates (calculates) the distance from the vehicle 101 to the candidate object detected in the camera image of the previous frame (hereinafter referred to as the previous distance) based on a distance map generated by the calculation unit 111 from the camera image of the previous frame immediately preceding the current frame. Furthermore, the detection unit 113 estimates (calculates) the distance traveled by the vehicle 101 between frames based on the detection value of the vehicle speed sensor 2a. Note that the detection value of a sensor other than the vehicle speed sensor may be used to estimate the distance traveled by the vehicle 101.

[0059] The detection unit 113 determines whether a candidate object detected from the camera image of the current frame is the same as a candidate object detected from the camera image of a past frame, based on the estimated previous distance, current distance, and distance traveled. In this case, the detection unit 113 determines that both candidate objects are the same if the difference between the previous distance and the current distance corresponds to (approximately equal to) the distance traveled. When identifying candidate objects between frames, the direction of movement of the vehicle 101 may also be taken into consideration.

[0060] As described above, the detection unit 113 detects an obstruction. In the example of Figure 4B, a part of the wall WL1 (the part corresponding to the candidate pixel group PG1) corresponding to the pixel group PG1 is detected as an obstruction. When an obstruction is detected, the detection unit 113 detects an entrance that is connected to the obstruction along the direction of travel, based on the camera image. Specifically, the detection unit 113 detects the end of the pixel group corresponding to the detected obstruction in the horizontal axis direction, which is the end closer to the vehicle 101 in the lane width direction, as the end of the entrance on the near side in the direction of travel. In the example of Figure 4B, the right end of the pixel group PG1 is detected as the end of the entrance GW1 on the near side in the direction of travel.

[0061] The detection unit 113 outputs information indicating the detection result of the entrance (hereinafter referred to as entrance information). More specifically, the detection unit 113 outputs information indicating the position of the end of the entrance on the near side in the direction of travel to the action plan generation unit 15. In this case, the entrance information output from the detection unit 113 is used by the action plan generation unit 15 to generate a target trajectory. The driving control unit 16 controls each actuator AC based on the target trajectory generated using the entrance information.

[0062] The detection unit 113 may also output to the display 3a display information as entrance information, which is a display information superimposed on the camera image, indicating the position of the front end of the detected entrance in the direction of travel. Alternatively, the detection unit 113 may output to the display 3a text information indicating the position of the detected entrance as entrance information. Furthermore, the detection unit 113 may output to a speaker (not shown) an audio message notifying the position of the detected entrance as entrance information.

[0063] Furthermore, the detection unit 113 may output (store) the entry point information to the storage unit 12. More specifically, the detection unit 113 may update the internal map information stored in the storage unit 12 based on the entry point information. In addition, the detection unit 113 may output the entry point information to an external device (such as a server device) via a communication unit (not shown).

[0064] Figure 7 is a flowchart showing an example of a process executed by the CPU of the controller 10 in Figure 3 according to a predetermined program. The process shown in this flowchart is executed repeatedly, for example, when the vehicle 101 is in motion.

[0065] First, in step S1, the controller 10 determines whether or not a camera image has been acquired. If the result in step S1 is negative, the process ends. If the result in step S1 is positive, the process proceeds to steps S2 to S9. In this way, steps S2 to S9 are repeated each time a camera image is acquired, that is, at time intervals determined by the frame rate of camera 1a.

[0066] In step S2, the controller 10 generates a distance map based on the camera image acquired in step S1. In step S3, the controller 10 extracts candidate pixels from the camera image based on the distance map generated in step S2.

[0067] In step S4, the controller 10 detects a group of candidate pixels from the multiple candidate pixels extracted in step S3 that are adjacent to each other in the horizontal or vertical axis direction. If multiple candidate pixel groups are detected in step S4, the processing in steps S5 to S7 is performed for each of the multiple candidate pixels.

[0068] In step S5, the controller 10 determines whether the object (candidate object) corresponding to the candidate pixel group satisfies predetermined conditions based on the size (number of pixels) of the candidate pixel group in the horizontal axis direction and the vertical axis direction. More specifically, the controller 10 determines whether the width and height of the candidate object are greater than predetermined thresholds.

[0069] If affirmed in step S5, in step S6, the controller 10 determines whether the candidate object detected from the camera image of the current frame is the same as the candidate object detected from the camera image of the previous frame immediately preceding the current frame. More specifically, the controller 10 determines whether the object corresponding to the group of candidate pixels detected in step S4 is the same as the object corresponding to the group of candidate pixels detected in step S4 of the previous cycle.

[0070] If affirmed in step S6, in step S7 the controller 10 detects the candidate object affirmed in step S6 as an occluding object. At this time, the controller 10 may also detect as an occluding object a candidate object that has been determined to be the same object over a predetermined number of consecutive frames (for example, 3). If denied in step S5 or step S6, the controller 10 proceeds to the processing in step S8.

[0071] In step S8, the controller 10 determines whether there are any unprocessed candidate pixels, i.e., candidate pixels for which the processing in steps S5 to S7 has not been performed. If the determination in step S8 is positive, the controller 10 performs the processing in steps S5 to S7 on the unprocessed candidate pixels. If the determination in step S8 is negative, in step S9, the controller 10 detects entry points that are aligned with the direction of travel to the occluding object detected in step S8, i.e., entry points whose visibility is reduced or impossible due to the occluding object. The controller 10 outputs entry point information indicating the result of the entry point detection.

[0072] According to the embodiments described above, the following effects and advantages can be obtained. (1) The entrance detection device 60 includes a camera that captures an area in the direction of travel of the vehicle as it travels on the road, a calculation unit 111 which is an acquisition unit that acquires a distance map in which distance information indicating the distance to an object included in the imaging area of ​​the camera 1a is recorded for each pixel of the image captured by the camera 1a, an extraction unit 112 which extracts from the image captured image pixels in which the difference in distance information values ​​with comparison pixels that are a predetermined number of pixels away in the horizontal axis direction of the image captured image is greater than or equal to a predetermined value, and a detection unit 113 which is an entrance detection unit that detects entrances facing the road based on the pixels extracted by the extraction unit 112. This makes it possible to easily detect entrances facing the road on which the vehicle is traveling, in particular entrances that are difficult or impossible to see due to obstructions. As a result, the processing load on the device associated with entrance detection can be reduced.

[0073] (2) The detection unit 113 further acts as an obstruction detection unit, detecting obstructions based on the pixels extracted by the extraction unit 112. The detection unit 113 detects entry points that are connected to the detected obstruction along the direction of travel. This makes it possible to detect entry points that are difficult to see due to occlusion caused by the obstruction.

[0074] (3) The detection unit 113 further acts as a candidate detection unit, acquiring a group of pixels that are close to each other in the horizontal axis direction or vertical axis direction of the captured image from a group of pixels extracted by the extraction unit 112 as a group of candidate occluding pixels, and detecting an object that is a candidate occluding object based on the group of candidate occluding pixels. More specifically, the detection unit 113 determines whether the width and height of the object corresponding to the group of candidate occluding pixels are both greater than a predetermined threshold based on the size of the group of candidate occluding pixels in the horizontal axis direction and the vertical axis direction, and if both the width and height are greater than the predetermined threshold, it detects the object corresponding to the group of candidate occluding pixels as a candidate occluding object. The detection unit 113 then detects an occluding object from among the detected objects (candidate occluding objects). This prevents objects that do not affect the visibility of the entrance (such as utility poles and bushes) from being mistakenly detected as occluding objects, and reduces the processing load on the device.

[0075] (4) The detection unit 113 determines that the height of the object (candidate occluder) corresponding to the candidate occluder pixel group is greater than a predetermined threshold when the maximum number of pixels in the vertical axis direction of the candidate occluder pixel group is greater than a determination value. The detection unit 113 also determines that the width of the object corresponding to the candidate occluder pixel group is greater than a predetermined threshold when the maximum number of pixels in the horizontal axis direction of the captured image is greater than a determination value. The determination value is set to the number of images obtained by multiplying the number of pixels corresponding to the predetermined threshold by a predetermined reduction ratio. The number of pixels corresponding to the predetermined threshold is determined based on the predetermined threshold and the distance to the object (candidate occluder) corresponding to the candidate occluder pixel group, as indicated by distance information. This makes it possible to suppress the failure to detect occluders.

[0076] (5) Camera 1a repeatedly captures images of the area in the direction of travel of the vehicle according to a predetermined frame rate. The detection unit 113 further acts as a determination unit and determines whether the first object and the second object are the same object based on the distance traveled by the vehicle between the first frame (current frame) and the second frame immediately preceding the first frame, the distance from the vehicle 101 to the first object (candidate for an obstruction) detected from the image captured in the first frame, and the distance from the vehicle 101 to the second object (candidate for an obstruction) detected from the image captured in the second frame. The detection unit 113 detects an object (candidate for an obstruction) that has been determined to be the same object over a predetermined number of consecutive frames (3 as an example) as an obstruction. This further suppresses false detection of obstructions and improves the accuracy of obstruction detection.

[0077] (6) The vehicle control device 50 comprises an entrance detection device 60, an actuator AC, and a driving control unit 16. The detection unit 113 further outputs entrance information indicating the detection result of the entrance. The driving control unit 16 controls the actuator AC based on the entrance information output by the detection unit 113. This enables appropriate preventive control (such as brake control) in preparation for pedestrians suddenly appearing on the road from the entrance. It also enables appropriate avoidance control (such as brake control) in response to pedestrians suddenly appearing from the entrance.

[0078] The above embodiment can be modified into various forms. Modifications will be described below. In the above embodiment, the entrance detection device 60 detects entrances facing the road on which the vehicle 101 is traveling based on camera images. However, the entrance detection device 60 may detect entrances facing the road on which the vehicle 101 is traveling based on, for example, lidar detection values ​​(lidar images) or radar detection values ​​(radar images) instead of camera images.

[0079] Furthermore, although the above embodiment applied the vehicle control device 50 and the entrance detection device 60 to an autonomous vehicle, the vehicle control device 50 and the entrance detection device 60 can also be applied to vehicles other than autonomous vehicles. For example, the vehicle control device 50 and the entrance detection device 60 can also be applied to a manually driven vehicle equipped with ADAS (Advanced driver-assistance systems). In that case, the detection unit 113 may output information to an output device such as a display or speaker, based on the entrance information, prompting the driver to take hands-on driving, or providing advance notice and notification of switching to manual driving mode.

[0080] From another perspective, the entrance detection device of the above embodiment can also be configured as an entrance detection method for detecting entrances facing the road on which the vehicle is traveling. That is, it can also be configured as an entrance detection method that includes the steps of: acquiring a distance map in which distance information indicating the distance to an object included in the imaging area of ​​a camera that images the area in the direction of travel of the vehicle traveling on the road is recorded for each pixel of the camera's captured image; extracting pixels from the captured image in which the difference in distance information value between a comparison pixel a predetermined number of pixels away in the horizontal axis direction of the captured image is greater than or equal to a predetermined value; and detecting an entrance facing the road based on the extracted pixels.

[0081] Furthermore, the present invention can be constructed by replacing the above-described entrance detection method with a program that causes a computer to execute a process to detect entrances facing the road on which the vehicle is traveling. Moreover, the present invention can also be constructed by replacing this with a computer-readable storage medium on which this program is recorded.

[0082] The above description is merely an example, and the present invention is not limited by the embodiments and modifications described above, as long as the features of the present invention are not impaired. It is also possible to arbitrarily combine one or more of the above embodiments and modifications, and to combine modifications with each other. [Explanation of Symbols]

[0083] 1a Camera, 2a Vehicle speed sensor, 3a Display, 10 Controller, 11 Calculation unit, 12 Storage unit, 15 Action plan generation unit, 16 Driving control unit, 50 Vehicle control device, 60 Entrance detection device, 111 Calculation unit, 112 Extraction unit, 113 Detection unit, AC Actuator

Claims

1. A camera that captures images of the area in the direction of travel of the vehicle while it is moving on the road, An acquisition unit acquires a distance map in which distance information indicating the distance to an object included in the imaging area of ​​the camera is recorded for each pixel of the image captured by the camera. An extraction unit extracts from the captured image pixels in which the difference in distance information values ​​between the captured image and comparison pixels located a predetermined number of pixels apart in the horizontal axis direction is greater than or equal to a predetermined value. An entrance detection device characterized by comprising: an entrance detection unit that detects an entrance facing the road based on the pixels extracted by the extraction unit.

2. In the entrance detection device according to claim 1, The system further includes an obstacle detection unit that detects an obstacle based on the pixels extracted by the extraction unit, The entrance detection device is characterized in that the entrance detection unit detects the entrance that is connected to the obstruction detected by the obstruction detection unit along the direction of travel.

3. In the entrance detection device according to claim 2, The extraction unit further comprises a candidate detection unit that obtains a group of pixels that are close to each other in the horizontal axis direction or vertical axis direction of the captured image from a plurality of pixels extracted by the extraction unit as a group of candidate occluding pixels, and detects the object that is a candidate for the occluding object based on the group of candidate occluding pixels. The entrance detection device is characterized in that the obstruction detection unit detects the obstruction from among the objects detected by the candidate detection unit.

4. In the entrance detection device according to claim 3, The entrance detection device is characterized in that the candidate detection unit determines whether the width and height of the object corresponding to the candidate pixel group of occluding objects are both greater than a predetermined threshold based on the size of the candidate pixel group of occluding objects in the horizontal axis direction and the vertical axis direction, and when both the width and height are greater than the predetermined threshold, it detects the object corresponding to the candidate pixel group of occluding objects as a candidate occluding object.

5. In the entrance detection device according to claim 4, The candidate detection unit determines that the height of the object corresponding to the candidate pixel group of occluding objects is greater than the predetermined threshold when the maximum number of pixels in the vertical axis direction of the candidate pixel group of occluding objects is greater than the determination value. The determination value is set to the number of images obtained by multiplying the number of pixels corresponding to the predetermined threshold by a predetermined reduction ratio. The entrance detection device is characterized in that the number of pixels corresponding to the predetermined threshold is determined based on the predetermined threshold and the distance to the object corresponding to the group of candidate pixels for the obstruction, as indicated by the distance information.

6. In the entrance detection device according to claim 4, The candidate detection unit determines that when the maximum number of pixels in the group of candidate occluding pixels in the horizontal axis direction is greater than the determination value, the width of the object corresponding to the group of candidate occluding pixels is greater than the predetermined threshold. The determination value is set to the number of images obtained by multiplying the number of pixels corresponding to the predetermined threshold by a predetermined reduction ratio. The entrance detection device is characterized in that the number of pixels corresponding to the predetermined threshold is determined based on the predetermined threshold and the distance to the object corresponding to the group of candidate pixels for the obstruction, as indicated by the distance information.

7. In the entrance detection device according to any one of claims 3 to 6, It is further equipped with a judgment unit, The camera repeatedly captures images of the region according to a predetermined frame rate. The determination unit determines whether the first object and the second object are the same object based on the distance traveled by the vehicle between the first frame and the second frame immediately preceding the first frame, the distance from the vehicle to the first object detected by the candidate detection unit from the captured image of the first frame, and the distance from the vehicle to the second object detected by the candidate detection unit from the captured image of the second frame. The entrance detection device is characterized in that the obstruction detection unit detects as the obstruction an object that has been determined to be the same object by the determination unit over a predetermined number of consecutive frames.

8. In the entrance detection device according to claim 7, An entry point detection device characterized in that the predetermined number is 3.

9. The entrance detection device according to claim 1, Actuator for driving, It comprises a driving control unit, The entry point detection unit further outputs entry point information indicating the detection result of the entry point, The vehicle control device is characterized in that the driving control unit controls the actuator based on the entry information.

10. The steps include: obtaining a distance map, which records distance information indicating the distance to an object included in the imaging area of ​​a camera that images the area in the direction of travel of the vehicle as it travels on the road, for each pixel of the image captured by the camera; The steps include extracting pixels from the captured image in which the difference in distance information value between a comparison pixel located a predetermined number of pixels away in the horizontal axis direction of the captured image is greater than or equal to a predetermined value, An entrance detection method characterized by comprising the step of detecting an entrance facing the road based on the extracted pixels.

11. The steps include: obtaining a distance map, which records distance information indicating the distance to an object included in the imaging area of ​​a camera that images the area in the direction of travel of the vehicle as it travels on the road, for each pixel of the image captured by the camera; The steps include extracting pixels from the captured image in which the difference in distance information value between a comparison pixel located a predetermined number of pixels away in the horizontal axis direction of the captured image is greater than or equal to a predetermined value, An entrance detection program characterized by causing a computer to perform the steps of detecting an entrance facing the road based on the extracted pixels.

Images