Manual driving support device of vehicle

Abstract

To improve a support on manual driving of a vehicle.SOLUTION: A manual driving support device of a vehicle comprises: a vehicle sensor group including a camera; a display device, and a control section connected to the vehicle sensor group and the display device. The control section performs: generating a travel direction grid indicating a passing range in traveling of the vehicle so as to be superimposed on a photographed image of a travel direction visual field by the camera; and extracting an interference candidate object having a possibility to interfere with the vehicle based on detection information of the vehicle sensor group. The control section performs: changing a part overlapped with the interference candidate object on the travel direction grid; converting the processed travel direction grid into a plane image so as to output the image; and allowing the display device to display the image by overlapping with the photographed image of the travel direction visual field.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a manual driving assistance device for a vehicle. [Background technology]

[0002] In the field of vehicles such as automobiles, technological innovations are advancing to enable autonomous driving and assist manual driving. Patent Document 1 discloses a method of displaying perspective images of a road surface superimposed on an image captured from a vehicle. Patent Document 2 discloses a method of displaying a planar grid of the road surface superimposed on a bird's-eye view image captured from a vehicle. In this way, it is believed that by superimposing perspective images of the road surface and a planar grid of the road surface on an image captured from the vehicle, it becomes easier to understand the situation around the vehicle. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2011 / 114597 [Patent Document 2] Japanese Patent Application Laid-Open No. 2012-051678 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the perspective images and planar grids of the road surface in Patent Documents 1 and 2 are merely two-dimensional representations of the road surface on which a vehicle travels, and are intended to make the road surface conditions easier to understand. Even if such perspective images and planar grids of the road surface are superimposed on an image, it does not necessarily make it easy to understand whether or not there is interference with surrounding obstructions in the direction of travel when the vehicle is being manually driven. For example, tree branches or the like may be protruding in the direction of travel of the vehicle. Even if perspective images and planar grids of the road surface are superimposed on an image containing tree branch components, it is not easy to understand that there will be interference with the tree branches or the like. Furthermore, superimposing the perspective images and planar grids of the road surface may draw the driver's attention to the perspective images and planar grids, making it more likely that the driver will not pay attention to tree branches or the like included in the image.

[0005] Some vehicles are designed to automatically drive into a parking space and park the vehicle there. In this automated driving system, it is desirable to determine whether a parking space is available for parking and, if available, to execute automated driving control to stop the vehicle in the parking space. For example, if the parking space is small, if a person is present in the parking space, or if the road surface of the parking space is significantly uneven, it is not desirable to automatically drive the vehicle into the parking space. Furthermore, automated driving is also desirable to check for interference with not only protrusions and people in the road surface of the parking space, but also with tree branches and other obstacles in the parking space, and to drive the vehicle into a parking space where interference will not occur. In most cases, parking spaces are designed to accommodate a standard-sized vehicle, but depending on the installer's convenience, they may be designed to be too small for a standard-sized vehicle. If the parking space is small or if there is an obstruction in the parking space, the vehicle may not be able to park in the parking space using automated driving. In this case, the driver of the vehicle must manually drive the vehicle and park it in the parking space.

[0006] Thus, there is a demand for improved assistance for manual driving of vehicles. [Means for solving the problem]

[0007] According to one embodiment of the present invention, there is provided a manual driving assistance device for a vehicle, the device comprising: a group of vehicle sensors including a camera that captures an image in the direction of travel of the vehicle; a display device capable of displaying an image captured by the camera in a field of view in the direction of travel; and a control unit connected to the group of vehicle sensors and the display device, wherein the control unit executes the following processes: a grid generation process that generates a traveling direction grid that indicates a passing range of the vehicle as it travels, to be overlaid on the captured image; an interference candidate extraction process that extracts, based on detection information from the group of vehicle sensors, an interference candidate that may interfere with the vehicle when it travels in the direction of travel of the vehicle; a grid processing process that changes a portion of the generated traveling direction grid that overlaps with the interference candidate to a color or shape that is different from that of other portions of the traveling direction grid; and a display process that converts the processed traveling direction grid into a planar image in the field of view in the direction of travel and outputs the planar image in the field of view in the direction of travel, and displays, on the display device, an image in which the processed planar image in the field of view in the direction of travel of the traveling direction grid is overlaid on the captured image in the field of view in the direction of travel. [Effects of the Invention]

[0008] In the present invention, a control unit connected to a group of vehicle sensors including a camera that captures an image in the traveling direction of the vehicle and to a display device causes the display device to display an image in which a traveling direction grid in the traveling direction field of view is superimposed on an image in the traveling direction field of view. The image in which the traveling direction grid in the traveling direction field of view is superimposed on an image in the traveling direction field of view allows the driver of the vehicle to easily understand the state of the traveling direction when manually driving the vehicle in the traveling direction.

[0009] Moreover, in the present invention, the control unit generates a traveling direction grid indicating the range through which the vehicle passes when traveling, to be overlaid on an image captured in the traveling direction field of view by a camera that captures the traveling direction of the vehicle. The control unit also extracts interference candidates that may interfere with the vehicle when traveling in the traveling direction of the vehicle, based on detection information from the vehicle sensors. The control unit changes the portion of the generated traveling direction grid that overlaps with the interference candidate to a color or shape different from that of the other portions of the grid. The control unit converts the processed traveling direction grid obtained by this processing into a planar image in the traveling direction field of view, overlays it on the captured image in the traveling direction field of view, and displays it on a display device. As a result, the driver of the vehicle can easily understand, based on the highlighted display of the changed color or shape of the grid, the interference candidate object that is likely to interfere with the vehicle as it travels in the direction of travel when the vehicle is manually driven in the direction of travel. In particular, in the present invention, the superimposed image and the traveling direction grid are both in the field of view of the traveling direction. Therefore, the driver of the vehicle can easily understand, based on the position and range of the changed portion of the traveling direction grid, how far the interference candidate object will need to travel before it will interfere. As a result, the driver of the vehicle can drive and stop the vehicle in a narrow parking space where parking is difficult using automated driving, while easily understanding the situation on the screen and suppressing interference with protrusions on the road surface, tree branches, and the like.

[0010] In this way, the present invention allows the vehicle occupants to easily understand whether or not there is interference with surrounding obstacles in the direction of travel when the vehicle is being driven manually. As a result, the present invention can provide excellent support for manual driving of a vehicle. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a top view of a vehicle according to a first embodiment of the present invention. [Figure 2]FIG. 2 is a view of the car of FIG. 1 parked in a parking space, seen from the front side of the car. [Figure 3] FIG. 3 is an explanatory diagram of the manual driving assistance device for the automobile of FIG. [Figure 4] FIG. 4 is a flowchart of assistance activation control during manual driving, which is executed by the CPU of the manual driving assistance device of FIG. [Figure 5] FIG. 5 is a block diagram showing a manual driving assistance function implemented by the CPU of the assistance image generating device of FIG. 3 when assistance control during manual driving is activated in step ST5 of FIG. [Figure 6] FIG. 6 is an explanatory diagram of an example of an image captured by an external camera in the field of view in the traveling direction. [Figure 7] FIG. 7 is an explanatory diagram of an interference virtual space including the traveling direction of the automobile. [Figure 8] FIG. 8 is an explanatory diagram of a first frame corresponding to a width equal to or greater than the width and height of an automobile. [Figure 9] FIG. 9 is an explanatory diagram of a traveling direction grid in the traveling direction field of view, which corresponds to the trajectory of the first frame body traveling. [Figure 10] FIG. 10 is an explanatory diagram of a heading grid in the heading field of view when the vehicle is being steered. [Figure 11] FIG. 11 is an explanatory diagram of a state in which the traveling direction grid in the traveling direction field of view of FIG. 9 is superimposed on the traveling direction interference virtual space of FIG. [Figure 12] FIG. 12 is an explanatory diagram showing an example of the processed travel direction grid. [Figure 13] FIG. 13 is an explanatory diagram of an example of a display image displayed on a display device. [Figure 14] FIG. 14 is an explanatory diagram showing an example of processing a progression direction grid in the second embodiment of the present invention. [Figure 15] FIG. 15 is an explanatory diagram of an example of a display image displayed on a display device. [Figure 16]FIG. 16 is an explanatory diagram of a second frame that is larger in width and height than the first frame and that can be used by the cruise control device for automatic driving in the third embodiment of the present invention. [Figure 17] FIG. 17 is a flowchart of driving control for the driving control device to park the automobile in a parking space by automatic driving. [Figure 18] FIG. 18 is a flowchart of the support activation control during manual driving, which is executed by the CPU of the support image generating device of FIG. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0013] [First embodiment] FIG. 1 is a top view of an automobile 1 according to a first embodiment of the present invention. 1 shows a parking lot consisting of multiple parking spaces. The multiple parking spaces are separated by multiple boundary lines 3 on the road surface. Also, standing trees 4 are shown behind the parking spaces. In the parking lot in FIG. 1, other cars 2 are parked in the parking spaces on the right and left. Therefore, the automobile 1 must proceed to and park in a central parking space near the standing tree 4, as shown by the dashed line in Figure 1. Here, the automobile 1 is an example of a vehicle.

[0014] FIG. 2 is a view of the car 1 of FIG. 1 parked in the central parking space, as seen from the front side of the car 1. In FIG. In this way, by driving backward, the car 1 can drive into the central parking space near the tree 4 and park there. The branches of tree 4 are closely overlapping on top of car 1. If another car 2 on the right tries to park in the center parking space, car 2 will get in the way of the branches of tree 4. Furthermore, the branches of tree 4 extend lower over the other car 2 on the left. Therefore, when car 1 tries to park in the parking space on the left, the branches of tree 4 interfere with the car.

[0015] When parking automobile 1 in this way so as not to interfere with the branches of standing trees 4, it is necessary to consider not only obstacles on the road surface such as stones or people in the parking space in which automobile 1 is driving, but also hollow obstacles such as the branches of standing trees 4 that may interfere with automobile 1 as it moves forward, as they may cause interference with the automobile.

[0016] Meanwhile, development of driving control units for automatic driving is underway for vehicles such as the automobile 1. For example, it is conceivable that driving for parking as described above may be realized by automatic driving. However, current autonomous driving is not a panacea. For example, it is conceivable that the self-driving automobile 1 detects potential interfering objects on the road surface, such as stones or people, using the exterior camera 12 of the automobile 1, and controls its driving based on this. In this case, the self-driving automobile 1 will not be able to control its driving so as to suppress interference with hollow interfering objects, such as the branches of standing trees 4. In addition, for example, an autonomously driven automobile 1 is generally expected to proceed and stop in a parking space with sufficient space for the vehicle. For this reason, an autonomously driven automobile 1 may not be able to proceed and stop in a narrow parking space that would be possible if the driver of the automobile 1 were to manually drive the automobile 1. In particular, when trying to secure space for passengers to get in and out of the automobile 1 after parking, the parking space required for autonomous driving becomes larger. If there is no parking space with sufficient space remaining, the driver of the automobile 1 must manually drive the automobile 1 based on his or her own operation to proceed and stop the automobile 1 in a parking space that is not sufficient. In this case, the driver of the automobile 1 may proceed and stop the automobile 1 to the left of the central parking space, as shown by the dashed line in Figure 2, and cause the automobile 1 to interfere with the branches of the standing tree 4. Thus, in the automobile 1, there is a need to improve assistance for manual driving of the automobile 1.

[0017] FIG. 3 is an explanatory diagram of the manual driving assistance device 10 of the automobile 1 of FIG. The manual driving assistance device 10 of the automobile 1 in FIG. 3 is incorporated into the control system of the automobile 1, for example, to park the automobile 1 by automatic driving or manual driving. 3 includes a vehicle sensor group 11, a cruise control device 17, a steering sensor 18, a speed sensor 19, an assist button 20, a display device 21, a speaker 22, and an assist image generation device 23 to which these are connected. Note that each component connected to the assist image generation device 23 may be connected to the assist image generation device 23 via a network for a control system for the automobile 1, such as a controller area network (CAN) or a local interconnect network (LIN).

[0018] The vehicle sensor group 11 is made up of a plurality of sensors for observing the outside of the automobile 1. Examples of such sensors include an exterior camera 12, a millimeter-wave radar 13, an infrared sensor 14, a Lidar (Light Detection And Ranging) 15, and a GNSS (Global Navigation Satellite System) receiver 16.

[0019] The exterior camera 12 captures images of the surroundings outside the vehicle 1, including the direction of travel of the vehicle 1. The exterior camera 12 may be a monocular camera, a compound camera such as a stereo camera, or a 360-degree camera. Furthermore, the vehicle 1 may be equipped with multiple cameras positioned to capture, for example, a 360-degree view of the vehicle 1. The captured images may include images of potential interference objects, such as stones, people, or tree branches, that are in the direction of travel of the vehicle 1. The exterior camera 12 is installed at a fixed position on the vehicle 1. In this case, the relative direction and distance of the interference candidate from the vehicle 1 can be obtained based on the image pixels captured by the exterior camera 12. In particular, the relative direction and distance of the interference candidate from the vehicle 1 can be obtained with high accuracy by performing trigonometric calculations based on the difference in position within the image. When calculating the relative direction and distance of the interference candidate from the vehicle 1 based on the position of the image pixels, it is recommended to correct for distortions of the lens used in the exterior camera 12.

[0020] The millimeter-wave radar 13 outputs millimeter waves while scanning in the traveling direction of the automobile 1. The output millimeter waves are reflected by potential interference objects such as stones, people, and branches of standing trees 4 in the traveling direction of the automobile 1, and the millimeter-wave radar 13 receives the reflected waves. The millimeter-wave radar 13 may calculate the relative direction and distance from the automobile 1 of the potential interference objects based on, for example, the TOF (Time Of Flight) from the time the millimeter waves are output to the time they are received.

[0021] The infrared sensor 14 detects infrared rays outside the vehicle 1 around the vehicle 1. Tangible objects such as stones, people, and branches of standing trees 4 generally emit infrared rays. For example, the infrared sensor 14, which is a dot projector or the like, can capture a detected image including components of potential interference objects such as stones, people, and branches of standing trees 4 that are in the traveling direction of the vehicle 1. The infrared sensor 14 is also installed at a fixed position on the vehicle 1. In this case, the relative direction and distance of the potential interference object from the vehicle 1 can be obtained based on the detected pixels in the detected image by the infrared sensor 14.

[0022] The lidar 15 emits a laser beam while scanning it in the traveling direction of the automobile 1. The emitted laser beam is reflected by potential interference objects such as stones, people, and branches of standing trees 4 in the traveling direction of the automobile 1, and the lidar 15 receives the reflected wave. The lidar 15 may calculate the relative direction and distance from the automobile 1 of the potential interference object based on the TOF from the time of emitting the laser beam to the time of receiving it.

[0023] The GNSS receiver 16 receives radio waves from GNSS satellites and generates information on the position and time of the automobile 1. Using the position of the automobile 1 detected by the GNSS receiver 16 and high-precision map data (not shown), it is possible to detect structures and the like that exist around the current position of the automobile 1.

[0024] In this way, the vehicle sensor group 11 includes a camera that captures images in the direction in which the automobile 1 is traveling. The vehicle sensor group 11 then outputs the detection information from these various sensors outside the vehicle to the support image generation device 23.

[0025] The cruise control device 17 controls the driving of the automobile 1 based on manual driving by the driver, or controls the driving of the automobile 1 by automatic driving. The cruise control device 17 generates control values according to the driving state of the automobile 1 and the state of operation of the steering wheel (not shown) by the driver, and outputs the control values to a drive device, a braking device, and a steering device (not shown) of the automobile 1. When the driver is driving manually, the cruise control device 17 may generate control values corrected according to the driving state of the automobile 1, etc., rather than generating control values corresponding to the driver's operation itself.

[0026] The steering sensor 18 detects the steering direction and steering amount of a steering wheel (not shown) by the driver of the automobile 1. The steering direction and steering amount may be based on those when the automobile 1 travels straight.

[0027] The speed sensor 19 detects the speed at which the automobile 1 is traveling. The speed sensor 19 may detect the acceleration of the automobile 1 while it is traveling, and may integrate this to detect the speed.

[0028] The assistance button 20 is provided so as to be operable by the driver in the cabin of the automobile 1. The assistance button 20 here is operated when the driver permits or requests assistance with manual driving of the automobile 1.

[0029] The display device 21 is provided in the passenger compartment of the automobile 1 so as to be visible to the driver. The display device 21 may be, for example, a meter panel, a center information display, or a HUD (Head-Up Display).

[0030] The speaker 22 is provided in the passenger compartment of the automobile 1 so as to be audible to the driver. Alternatively, or in addition to the speaker 22, a buzzer device may be provided in the automobile 1. The display device 21 and the speaker 22 constitute an in-vehicle infotainment device.

[0031] The assistance image generating device 23 generates an image for assisting the driver of the automobile 1 in manual driving, and outputs the image to the display device 21. As a result, the image for assisting the driver of the automobile 1 in manual driving can be displayed on the display device 21. The support image generating device 23 includes an interface unit 25, a timer 24, a memory 26, a CPU (Central Processing Unit) 27, and an internal bus 28 to which these are connected.

[0032] The interface unit 25 connects the support image generation device 23 to the control system of the automobile 1. The vehicle sensor group 11, the cruise control device 17, the steering sensor 18, the speed sensor 19, the support button 20, the display device 21, and the speaker 22 are connected to the interface unit 25. The interface unit 25 outputs information from outside the support image generation device 23 to the internal bus 28 through relay processing, and outputs information from the internal bus 28 to outside the support image generation device 23.

[0033] The timer 24 measures time and hour, and may measure the cycle at which the CPU 27 repeatedly executes a program.

[0034] The memory 26 records data and programs used by the CPU 27. The data and programs recorded in the memory 26 include setting information for assistance with manual driving of the automobile 1, assistance control programs, etc. The memory 26 may be a non-volatile memory such as an HDD (Hard Disk Drive) or an SSD (Solid State Device), a volatile memory such as a RAM (Random Access Memory), or a combination of these.

[0035] The CPU 27 reads and executes the program recorded in the memory 26. As a result, a control unit that controls the operation of the support image generating device 23 is realized in the support image generating device 23. The CPU 27, which serves as a control unit of the support image generation device 23, is connected to the vehicle sensor group 11 and the display device 21 through the interface unit 25. The CPU 27 can, for example, display on the display device 21, an image of the field of view in the traveling direction of the automobile 1 captured by the exterior camera 12 that captures an image in the traveling direction of the automobile 1.

[0036] FIG. 4 is a flowchart of the assistance activation control during manual driving, which is executed by the CPU 27 of the manual driving assistance device 10 of FIG. The CPU 27, as a control unit, repeatedly executes the assistance activation control of FIG. 4 in order to assist the driver of the automobile 1 in manual driving.

[0037] In step ST1, the CPU 27 determines whether or not the assist button 20 has been turned on. The driver operates the assist button 20 when permitting or requesting assistance with manual driving of the automobile 1. If the support button 20 has been operated, the CPU 27 advances the process to step ST5, whereas if the support button 20 has not been operated, the CPU 27 advances the process to step ST2.

[0038] In step ST2, the CPU 27 acquires setting information for assistance with manual driving of the automobile 1 from the memory 26. The setting information for assistance with manual driving acquired here may have a value that permits or prohibits assistance with manual driving. The memory 26 may record the setting information for assistance with a default value or a value set by the user.

[0039] In step ST3, if the setting information for assistance regarding manual driving of the automobile 1 acquired from the memory 26 is a value that allows assistance, the CPU 27 proceeds to step ST4. If the setting information for assistance is not a value that allows assistance, the CPU 27 proceeds to step ST6.

[0040] In step ST4, the CPU 27 acquires the current vehicle speed of the automobile 1 from the speed sensor 19 and compares it with the vehicle speed threshold. Here, the vehicle speed threshold may be, for example, an upper limit value for low-speed driving. The upper limit value for low-speed driving may be, for example, 10 km / h. If the current vehicle speed of the automobile 1 is equal to or lower than the vehicle speed threshold, the CPU 27 proceeds to step ST5. As a result, the CPU 27 can proceed to step ST5 when the setting information for assistance regarding manual driving of the automobile 1 permits assistance and the current vehicle speed is equal to or less than the threshold vehicle speed. If the current vehicle speed is not equal to or less than the threshold vehicle speed, the CPU 27 proceeds to step ST6. On the other hand, even if the setting information for assistance regarding manual driving of the automobile 1 permits assistance, if the current vehicle speed is not equal to or lower than the vehicle speed threshold, the CPU 27 will proceed to step ST6.

[0041] In step ST5, the CPU 27 activates assistance control during manual driving to assist the manual driving of the driver of the automobile 1. After that, the CPU 27 ends this control.

[0042] In step ST6, the CPU 27 stops the assistance control during manual driving in order to stop assisting the manual driving of the driver of the automobile 1. Thereafter, the CPU 27 ends this control.

[0043] FIG. 5 is a block diagram showing a manual driving assistance function implemented by the CPU 27 of the assistance image generating device 23 in FIG. 3 when assistance control during manual driving is activated in step ST5 in FIG. FIG. 5 shows an interference candidate extraction unit 60, an interference virtual space generation unit 61, a grid generation unit 62, a grid interference determination unit 63, a grid processing unit 64, a grid image generation and output unit 65, a remaining distance determination unit 66, and an alarm output unit 67. When the CPU 27 of the support image generating device 23 starts the support control in step ST5 of FIG. 4, the CPU 27 realizes the above-described units of FIG.

[0044] The interference candidate extraction unit 60 receives various types of detection information outside the vehicle detected by the vehicle sensor group 11 and the like, and extracts interference candidates in the traveling direction that may interfere with the traveling automobile 1. The detection information outside the vehicle may include, for example, images of the outside of the vehicle including the traveling direction of the automobile 1 taken by the exterior camera 12, and information extracted by the exterior camera 12, millimeter-wave radar 13, infrared sensor 14, or lidar 15, including the relative direction and distance from the automobile 1 of the interference candidate. The interference candidate extraction unit 60 may analyze captured images of the exterior of the vehicle and extract interference candidates in the traveling direction that may interfere with the traveling automobile 1. Furthermore, the interference candidate extraction unit 60 may extract interference candidates in the traveling direction that may interfere with the traveling direction of the automobile 1 from among interference candidates acquired from the vehicle sensor group 11 or the like, based on the traveling direction of the automobile 1. Furthermore, the interference candidate extraction unit 60 may extract interference candidates in the traveling direction on map data based on the current position of the automobile 1 determined by the GNSS receiver 16 and high-precision map data (not shown) recorded in the memory 26.

[0045] Here, the traveling direction of the automobile 1 varies depending on the manual operation of the driver. For this reason, it is preferable that the interference candidate extraction unit 60 extracts interference candidates not only for one traveling direction of the automobile 1, but also for all traveling directions in which the automobile 1 may travel when manually driven by the driver. For example, when the automobile 1 is manually parked and stopped as shown in FIG. 1 , the interference candidate extraction unit 60 may extract interference candidates that exist in the traveling direction of the automobile 1 within a range of, for example, about three times the width of the automobile 1 and within a range of the height of the automobile 1, based on detection information outside the vehicle obtained by the vehicle sensor group 11, etc.

[0046] Furthermore, interfering candidates in the traveling direction that may interfere with the traveling automobile 1 include not only stones and people on the road surface on which the automobile 1 is traveling, but also hollow branches of standing trees 4. Therefore, the interference candidate extraction unit 60 extracts, as interfering candidates that may interfere with the automobile 1, interfering candidates that are within the passing range of the road surface on which the automobile 1 is traveling, and hollow interfering candidates that may interfere within the passing range of the automobile 1 as it travels. Hollow interfering candidates may be captured by imaging outside the vehicle or detected by millimeter-wave radar 13, infrared sensor 14, or lidar 15. The interference candidate extraction unit 60 can extract hollow interfering candidates based on this detection information from outside the vehicle. In addition, road surface bumps and the like may also be extracted as interfering candidates on the road surface.

[0047] The interference virtual space generation unit 61 generates a virtual space corresponding to the real space in which the automobile 1 travels. The interference virtual space generation unit 61 also generates the interference virtual space by arranging one or more interference candidates extracted by the interference candidate extraction unit 60 in the generated virtual space according to their respective positions and sizes.

[0048] The grid generation unit 62 generates a traveling direction grid 50 that indicates the range through which the automobile 1 passes when traveling, to be superimposed on an image of the traveling direction field of view captured by the outside camera 12 that captures the traveling direction of the automobile 1. When the automobile 1 travels, it passes through an area defined by the width and height of the automobile 1 . The grid generation unit 62 first uses the vehicle data and user setting data recorded in the memory 26 to generate a first frame 52 that indicates the area through which the automobile 1 will pass when traveling. Here, the first frame 52 may basically be sized to have a width equal to or greater than the width and height of the automobile 1. However, if the first frame 52 is made too large compared to the actual width and height of the automobile 1, the number of potential interfering objects that will be determined to interfere with the automobile 1 later will increase. When the automobile 1 is being manually driven, it is basically sufficient that the automobile 1 can travel without causing interference, so the first frame 52 may be made approximately the same as the actual width and height of the automobile 1. Next, the grid generation unit 62 moves the generated first frame body 52 in the traveling direction of the automobile 1 according to the speed of the automobile 1 while traveling. The traveling direction of the automobile 1 may basically be a linear traveling direction. However, if the driver of the automobile 1 is operating a steering wheel (not shown), the traveling direction of the automobile 1 is not a straight line. In this case, the grid generation unit 62 may acquire information on the steering direction and steering amount detected by the steering sensor 18 and set the traveling direction to a curved direction according to the information. The steering direction and steering amount detected by the steering sensor 18 are based on the case where the automobile 1 travels straight. Furthermore, the distance by which the first frame body 52 is moved may be a predetermined fixed distance or may be variable according to the speed of the speed sensor 19 of the automobile 1. Next, the grid generating unit 62 generates a traveling direction grid 50 that indicates the range through which the automobile 1 passes when traveling, based on the trajectory of the moved first frame 52.

[0049] The traveling direction grid 50 generated by such a procedure can have a plurality of first frame bodies 52 arranged at intervals in the traveling direction of the automobile 1. The grid generating unit 62 may also generate a traveling direction grid 50 in which adjacent frame bodies are connected by line segments along the traveling direction of the automobile 1. The traveling direction grid 50 generated by these processes can accurately indicate the range of space through which the automobile 1 passes when traveling in a direction according to the steering operation by the driver. The grid interference determination unit 63 acquires information on the interference virtual space in which the interference candidate objects are located, generated by the interference virtual space generation unit 61, and information on the travel direction grid 50, generated by the grid generation unit 62, which indicates the range through which the automobile 1 passes when traveling. The grid interference determination unit 63 first places a traveling direction grid 50 in the interference virtual space. Next, the grid interference determination unit 63 determines interference between the interference candidate object and the travel direction grid 50 in the interference virtual space in which the travel direction grid 50 is arranged. Here, in the interference virtual space, if an interference candidate object is located inside the travel direction grid 50 , the grid interference determination unit 63 determines that the interference candidate object will interfere with the travel direction grid 50 . Furthermore, in the interference virtual space, if a part of the interference candidate object is inside the traveling direction grid 50 , the grid interference determination unit 63 determines that the part of the interference candidate object interferes with the traveling direction grid 50 . On the other hand, if the interference candidate is outside the traveling direction grid 50 , the grid interference determination unit 63 determines that the interference candidate does not interfere with the traveling direction grid 50 . As a result, the grid interference determination unit 63 determines whether or not an interference candidate object that may interfere with the automobile 1 extracted in the interference candidate extraction process will interfere in virtual space with the grid 50 in the direction of travel of the automobile 1 generated by the grid generation process. The grid interference determination unit 63 can extract interference candidates that interfere with the traveling direction grid 50 and a part of the interference candidates that interfere with the traveling direction grid 50 as interference candidates that should desirably be notified to the driver who is manually driving.

[0050] The grid processing unit 64 acquires information on the traveling direction grid 50, which is generated by the grid generation unit 62 and indicates the range through which the automobile 1 passes when traveling, and information on the interfering object candidate that the grid interference determination unit 63 determines to interfere with the traveling direction grid 50. In the following description, the interfering object candidate that interferes with the interfering traveling direction grid 50 includes cases where the interfering object is a part of the interfering object candidate. When the grid processing unit 64 acquires information about an interference candidate object that is determined to interfere with the travel direction grid 50, it changes the portion of the generated travel direction grid 50 that overlaps with the interference candidate object to a color or shape different from that of the other portions of the travel direction grid 50. If the grid processing unit 64 has not acquired information about an interfering object candidate that is determined to interfere with the traveling direction grid 50, it does not perform processing on the generated traveling direction grid 50. In this case, the grid processing unit 64 uses the acquired traveling direction grid 50 as the processed traveling direction grid 50 without modification.

[0051] The grid image generating and outputting unit 65 acquires the processed travel direction grid 50 from the grid processing unit 64. Then, the grid image generating and outputting unit 65 converts the processed traveling direction grid 50 into a planar image in the traveling direction field of view. Furthermore, grid image generation and output unit 65 outputs a planar image of the processed traveling direction grid 50 in the traveling direction field of view to exterior camera 12. Exterior camera 12 generates an image in which the planar image of the processed traveling direction grid 50 in the traveling direction field of view is superimposed on the captured image, and outputs the image to display device 21. Display device 21 displays the image in which the processed traveling direction grid 50 is superimposed on the image in the traveling direction field of view.

[0052] The remaining distance determination unit 66 acquires information on the interfering object candidate that the grid interference determination unit 63 has determined to interfere with the traveling direction grid 50 . The remaining distance determination unit 66 first determines the remaining distance between the automobile 1 and an interfering object that interferes with the traveling direction grid 50 based on a threshold value. Here, the threshold value of the remaining distance may be a predetermined fixed value such as a few meters, or may be variable depending on the speed of the speed sensor 19 of the automobile 1. If the remaining distance between the automobile 1 and the interfering candidate object interfering with the travel direction grid 50 is equal to or less than the threshold value, the remaining distance determination unit 66 outputs a warning instruction to the warning output unit 67. On the other hand, if the remaining distance between the automobile 1 and the interfering object that interferes with the traveling direction grid 50 is not equal to or less than the threshold value, the remaining distance determination unit 66 does not output a warning instruction to the warning output unit 67.

[0053] The warning output unit 67 is connected to the speaker 22. When a warning instruction is input from the remaining distance determination unit 66, the warning output unit 67 drives the speaker 22 to output a warning by sound.

[0054] Through the above processing, the support image generating device 23 in Figure 3 can provide the driver manually operating the automobile 1 with an image of the field of view in the direction of travel with the direction of travel grid 50 superimposed on the display device 21, and a warning via the speaker 22. The driver can easily judge the presence or absence and degree of interference if the vehicle continues to travel in the forward direction view image on which the forward direction grid 50 is superimposed, and can manually operate the vehicle 1 to travel forward based on that judgment. Furthermore, when the driver operates the steering wheel, the arrangement of the first frame bodies 52 in the traveling direction grid 50 that is superimposed on the image in the traveling direction field of view changes, and this may also change the interfering object candidates that may interfere with the automobile 1. For this reason, the driver can select a good traveling direction that minimizes interference with interfering object candidates by checking the screen of the display device 21 while operating the steering wheel. Furthermore, when the driver manually drives the automobile 1, a warning is output while the automobile 1 is moving. Even if the driver does not check the screen thoroughly, the driver can recognize the possibility of interference before the automobile 1 and the potential interference object actually interfere with each other. Furthermore, the driver can manually drive the automobile 1 to suppress the interference.

[0055] Next, the processing of the support image generating device 23 will be explained in detail using a specific example.

[0056] FIG. 6 is an explanatory diagram of an example of an image 30 captured by the outside-vehicle camera 12 in the field of view in the direction of travel. The captured image in the field of view in the direction of travel in Fig. 6 is an image captured outside the vehicle 1 of the vacant parking space in Fig. 1 by the exterior camera 12 of the vehicle 1. A pair of boundary lines 3 exist on the left and right of the parking space.

[0057] Unlike in FIG. 1, the parking space in FIG. 6 contains a stone 32 and a person 31. Also, a branch 34 of a standing tree 4 protrudes above the parking space. Here, the stone 32 and the person 31 are interfering candidate objects on the road surface of the parking space. The branch 34 of the standing tree 4 is a hollow interfering candidate object. These interfering candidates on the road surface and hollow interfering candidates are detected by a vehicle sensor group 11 provided on the automobile 1. An interfering candidate extraction unit 60 in Fig. 5 extracts interfering candidates on the road surface and hollow interfering candidates based on detection information from the vehicle sensor group 11 about the outside of the vehicle.

[0058] FIG. 7 is an explanatory diagram of an interference virtual space 40 including the traveling direction of the automobile 1. As shown in FIG. The interference virtual space generation unit 61 uses information about the interference candidates extracted by the interference candidate extraction unit 60 to position each interference candidate so that it corresponds to the relative direction and relative distance from the automobile 1, thereby generating the interference virtual space 40 of Figure 7. As a result, on the road surface of the interference virtual space 40 in FIG. 7, a stone object 42 and a person object 41 are placed between a pair of boundary line objects 43, corresponding to FIG. 7, a branch object 44 of the tree 4 is placed in the hollow of the interference virtual space 40, corresponding to FIG. 6. The branch object 44 of the tree 4 is placed so as to grow from the left side of the interference virtual space 40. As indicated by the dashed line in the figure, part of the branch 34 is located outside the left side of the interference virtual space 40. There is no need to place an interference candidate object outside the interference virtual space 40 in the interference virtual space 40. Here, the surface of each object may be modeled after the surface of each interference candidate detected by the vehicle sensor group 11. Alternatively, the surface of each object may be a cylinder or sphere of a size corresponding to each interference candidate detected by the vehicle sensor group 11. In particular, the branches 34 of the hollow tree 4 are thin. For this reason, it may be impossible for the vehicle sensor group 11 to detect each branch 34 with high accuracy. In this case, the branch object 44 may be a substantially cylindrical object with multiple bent portions corresponding to the multiple branches 34.

[0059] FIG. 8 is an explanatory diagram of a first frame 52 corresponding to a width equal to or greater than the width and height of the automobile 1. As shown in FIG. FIG. 9 is an explanatory diagram of a traveling direction grid 50 in the traveling direction field of view, which corresponds to the trajectory of the first frame 52 traveling. FIG. 10 is an explanatory diagram of a traveling direction grid 51 in the traveling direction field of view when the vehicle is being steered.

[0060] As shown in Fig. 8, the grid generation unit 62 generates a first frame 52 that indicates the area through which the automobile 1 will pass when traveling, using vehicle data and user-set data stored in the memory 26. The first frame 52 in Fig. 8 corresponds to the actual width and height of the automobile 1. By setting the size of the first frame 52 to correspond to the actual width and height of the automobile 1 in this way, it becomes possible to manually drive the automobile 1 and park it in the smallest parking space possible for the automobile 1. When the driver operates the steering wheel to a neutral position for going straight, the traveling direction of the automobile 1 is a linear direction facing forward or backward of the automobile 1. In this case, the grid generating unit 62 moves the first frame 52 along the linear traveling direction of the automobile 1, and generates the traveling direction grid 50 in the traveling direction field of view of FIG. In contrast, when the driver operates the steering wheel from the neutral position, the traveling direction of the automobile 1 curves diagonally toward the automobile 1. In this case, the grid generation unit 62 moves the first frame 52 along the curved traveling direction of the automobile 1, and generates a traveling direction grid 51 in the traveling direction field of view as shown in the example of FIG. 10 based on the trajectory of the first frame 52. The traveling direction grid 51 in the traveling direction field of view in FIG. 10 is for the case where the traveling direction is rearward of the automobile 1 and the driver operates the steering wheel to the left. Under similar conditions, when the driver operates the steering wheel to the right, the grid generation unit 62 generates a traveling direction grid in the traveling direction field of view in which the left and right directions of FIG. 10 are reversed. Furthermore, the degree of curvature of the traveling direction grid 51 may increase or decrease depending on the amount of steering operation. This allows the grid generation unit 62 to generate a traveling direction grid 50 or the like in the traveling direction field of view, which is defined by four planes, left, right, top, bottom, and right, of the automobile 1. The automobile 1 will basically travel inside the traveling direction grid 50 or the like.

[0061] The size of the first frame 52 may be greater than or equal to the actual width or height of the automobile 1. For example, if the memory 26 stores setting information regarding the width and height of the first frame 52, the grid generation unit 62 may determine the width and height of the first frame 52 to correspond to the setting in the memory 26. For example, the memory 26 may store, as user setting information, the height of the automobile 1, an additional value to the actual height of the automobile 1. For example, the memory 26 may store the opening and closing width of the left and right doors of the automobile 1, as an additional value to the actual width of the automobile 1 for opening and closing the doors. Other information stored in the memory 26 may include, for example, the minimum ground clearance value when the vehicle height of the automobile 1 is changed, the rear width and height value for opening the rear gate of the automobile 1, and a value indicating whether or not optional components protrude from the bumper of the automobile 1. The minimum ground clearance of the automobile 1 can be used to determine whether the automobile 1 can traverse road bumps, etc. The rear width and height for opening the rear gate of the automobile 1, or an optional member protruding from the bumper, can be used to determine the stopping position of the automobile 1, etc.

[0062] FIG. 11 is an explanatory diagram of a state in which a traveling direction grid 50 in the traveling direction field of view of FIG. 9 is superimposed on the traveling direction interference virtual space 40 of FIG. The grid interference determination unit 63 places a traveling direction grid 50 generated by the grid generation unit 62 in the interference virtual space 40 generated by the interference virtual space generation unit 61 and in which the interference candidate objects of FIG. 7 are placed. In FIG. 11, the grid interference determination unit 63 places the traveling direction grid 50 for the case of traveling straight ahead as shown in FIG. 9. Note that in FIG. 11, to avoid complicating the drawing, only the portion of the traveling direction grid 50 of FIG. 9 defined by the first frame 52 is shown. Because the automobile 1 is located in front of the parking space as shown in FIG. 1, the grid interference determination unit 63 places a plurality of first frame bodies 52 corresponding to the width and height of the automobile 1 between the pair of boundary lines 3 so that they are lined up along the direction of travel toward the parking space.

[0063] Next, the grid interference determination unit 63 determines interference between each interference candidate object and the traveling direction grid 50 in the interference virtual space 40 of FIG. 11, a stone object 42 and a person object 41 are located between a pair of boundary line objects 43. The grid interference determination unit 63 determines that these interfering candidate objects on the road surface are interfering with the traveling direction grid 50. 11, a plurality of branch objects 44 of the tree 4 are arranged so as to extend from the left side of the interference virtual space 40 toward the center of the interference virtual space 40. Some of the plurality of branch objects 44 are arranged so as to extend further toward the center of the interference virtual space 40 than the left side defined by the traveling direction grid 50. The grid interference determination unit 63 determines that these hollow interfering objects are interfering with the traveling direction grid 50. If the traveling direction grid 50 is that of Fig. 10, the results of the collision determination may be different from those of Fig. 11. For example, the grid collision determination unit 63 may determine that the stone object 42 located to the left between the pair of boundary lines 3 and the multiple branch objects 44 do not interfere with the traveling direction grid 50 of Fig. 10. In this way, the grid interference determination unit 63 determines whether an interfering candidate object is on the road surface or is hollow, and whether the interfering candidate object may interfere with the traveling direction grid 50 .

[0064] FIG. 12 is an explanatory diagram showing an example of the processed travel direction grid 50. When the grid processing unit 64 acquires information about an object of an interference candidate that is determined to interfere with the traveling direction grid 50, it changes the part of the traveling direction grid 50 that overlaps with each interference candidate to a color or shape different from that of the other parts of the traveling direction grid 50. The grid processing unit 64 deforms the shape of the interfering portion of the traveling direction grid 50 so that it corresponds to a grid image obtained by projecting the traveling direction grid 50 from the side of the automobile 1 or from above the automobile 1, for example. In this case, the grid processing unit 64 will change the shape of the travel direction grid 50 at the positions of the stone object 42, the person object 41, and the branch object 44 that are determined to interfere with the travel direction grid 50. The processed travel direction grid 50 will look like that shown in FIG. 12, the processed traveling direction grid 50 is deformed in a deformed portion 54 projected onto the surface of the stone object 42, a deformed portion 53 projected onto the surface of the person object 41, and a deformed portion 55 projected onto the surface of the branch object 44. As shown in FIG. 12, the traveling direction grid 50 is significantly deformed in the deformed portion 54 projected onto the surface of the stone object 42 and the deformed portion 53 projected onto the surface of the person object 41. Furthermore, the grid processing unit 64 assigns these deformed portions a color attribute that stands out more than the basic color of the travel direction grid 50. In Figure 12, color coding is not possible due to the size of the drawing, so they are made to stand out by using thick dashed lines.

[0065] FIG. 13 is an explanatory diagram of an example of a display image displayed on the display device 21. As shown in FIG. In the display image of FIG. 13, the processed traveling direction grid 50 of FIG. 12 is superimposed on the captured image of the traveling direction field of view of FIG. The grid image generating and outputting unit 65 acquires the processed travel direction grid 50 from the grid processing unit 64. Then, the grid image generating and outputting unit 65 converts the acquired processed traveling direction grid 50 into a planar image in the traveling direction field of view. Furthermore, the grid image generation and output unit 65 outputs to the outside camera 12 a planar image of the processed traveling direction grid 50 in the traveling direction field of view. The vehicle exterior camera 12 generates an image by superimposing a planar image in the field of view in the direction of travel for the processed direction of travel grid 50 on the captured image, and outputs the image to the display device 21. As a result, the display device 21 displays an image in which the processed traveling direction grid 50 is superimposed on the image in the traveling direction field of view, as exemplified in FIG.

[0066] In the display image of FIG. 13, a traveling direction grid 50 is displayed superimposed on the image captured by the outside-vehicle camera 12. In particular, deformations and discolored portions 53, 54, 55 of the traveling direction grid 50 are superimposed on potential interference objects such as a stone 32, a person 31, and a branch 34 captured in the image captured by the outside camera 12. A driver manually operating the automobile 1 can instantly grasp the presence of these potential interfering objects that may cause interference if the automobile continues traveling as is, on the display screen of the display device 21, from the deformed and discolored portions 53, 54, 55 of the traveling direction grid 50. The driver can also manually operate the automobile 1 in response to this. In particular, the traveling direction grid 50 is deformed and discolored not only for stones 32 and people 31 captured in the image captured by the outside camera 12, but also for hollow branches 34. Therefore, even a driver who normally pays attention to the road surface can be alerted to and aware of the possibility of interference between the vehicle 1 and the branch 34 in the air.

[0067] As described above, in this embodiment, the control unit connected to the vehicle sensor group 11 including the exterior camera 12 that captures images in the traveling direction of the automobile 1 and the display device 21 causes the display device 21 to display an image in which the traveling direction grid 50 is superimposed on an image captured in the traveling direction field of view. The image in which the traveling direction grid 50 is superimposed on the image captured in the traveling direction field of view in this way allows the driver of the automobile 1 to easily understand the state of the traveling direction when the automobile 1 is being manually driven in the traveling direction. Moreover, in this embodiment, the control unit generates a traveling direction grid 50 that indicates the range through which the automobile 1 passes while traveling, to be superimposed on an image captured in the traveling direction field of view by the exterior camera 12 that captures the traveling direction of the automobile 1. The control unit also extracts interfering objects that may interfere with the automobile 1 when traveling in the traveling direction of the automobile 1, based on detection information from the vehicle sensor group 11. The control unit changes the portion of the generated traveling direction grid 50 that overlaps with the interfering object candidate to a color or shape that is different from the other portions of the traveling direction grid 50. The control unit superimposes the traveling direction grid 50 obtained by this processing on the captured image in the traveling direction field of view and displays it on the display device 21. As a result, the driver of the automobile 1 can easily understand, based on the highlighted display of the portions of the traveling direction grid 50 where the color or shape has been changed, the interference candidate objects that are likely to interfere with the automobile 1 as it travels, as the state of the traveling direction when the automobile 1 travels in the traveling direction under manual driving. In particular, in this embodiment, the superimposed images and the traveling direction grid 50 are both in the traveling direction field of view. Therefore, the driver of the automobile 1 can easily understand, based on the position and range of the changed portions of the traveling direction grid 50 in the traveling direction field of view, how far the interference candidate objects will need to travel before they interfere. As a result, the driver of the automobile 1 can drive and stop the automobile 1 in a narrow parking space where parking is difficult using automated driving, while easily understanding the situation on the screen and suppressing interference with protrusions on the road surface and branches 34 of standing trees 4.

[0068] In this embodiment, in the interference candidate extraction process, the control unit extracts, as interference candidates that may interfere with the automobile 1, interference candidates that are within the passing range on the road surface on which the automobile 1 is traveling, and hollow interference candidates that may interfere within the passing range as the automobile 1 travels. As a result, on the display device 21, the traveling direction grid 50 can be displayed with the interference candidates on the road surface and the hollow interference candidates respectively discolored or deformed in their corresponding portions. The driver of the automobile 1 can easily understand from the display on the display device 21 that not only interference candidates on the road surface but also hollow interference candidates may interfere with the automobile 1. In contrast, if a perspective image or a planar grid of the road surface is superimposed on a captured image in the forward field of view, the perspective image or the planar grid of the road surface may be superimposed on interference candidates on the road surface or on hollow interference candidates. In this case, the driver of the automobile 1 may not necessarily easily understand the position and range of each interference candidate, and thus the possibility of interference with the automobile 1. For example, the driver of the automobile 1 may not pay attention to hollow interference candidates because the planar grid is superimposed on the hollow interference candidate.

[0069] In this embodiment, in the grid generation process, the control unit generates a traveling direction grid 50 that indicates the range through which the automobile 1 passes when traveling, based on the trajectory of a first frame 52, which corresponds to a width greater than or equal to the width and height greater than or equal to the height of the automobile 1, when traveling on the road surface of the interference virtual space 40, which corresponds to the captured image of the traveling direction field of view to be displayed on the display device 21. In this way, the traveling direction grid 50 can indicate that the range through which the automobile 1 may pass when traveling is inside the traveling direction grid 50. This allows the driver of the automobile 1 to easily understand that it is possible to move the automobile 1 forward by paying attention to interfering candidate objects within the passing range of the traveling direction grid 50. Furthermore, the driver of the automobile 1 can easily determine, based on the display on the display device 21, whether or not the automobile 1 can be moved smoothly in manual driving.

[0070] In this embodiment, the control unit determines in the grid processing process whether or not an interference candidate that has the potential to interfere with the automobile 1 and that is extracted in the interference candidate extraction process interferes with the traveling direction grid 50 generated in the grid generation process in the interference virtual space 40. If the interference candidate interferes in the interference virtual space 40, the control unit changes the interfering portion of the traveling direction grid 50 that overlaps with the interference candidate to a color or shape that is different from the other portions of the traveling direction grid 50. This enables the control unit to change the traveling direction grid 50 for interference candidate objects with which the automobile 1 may actually interfere, and not change the traveling direction grid 50 for interference candidate objects with which the automobile 1 is not likely to interfere. The presence or absence and degree of interference when the automobile 1 travels inside the traveling direction grid 50 can be appropriately indicated by processing and modifying the traveling direction grid 50. The driver of the automobile 1 can easily understand, under an image that is wider than the passing range through which the automobile 1 will pass as it travels, the interference candidate objects that require attention when manually driving the automobile 1 in the traveling direction. On the other hand, it is also conceivable to change the traveling direction grid 50 for all interference candidates in the image, for example. However, in this case, when the traveling direction grid 50 is changed by many of all interference candidates in the image, the driver of the automobile 1 must use only his or her own judgment to select which interference candidates he or she should pay attention to.

[0071] In this embodiment, in the grid processing, the control unit deforms the shape of the interfering portion of the traveling direction grid 50 so that it corresponds to the grid image obtained by projecting the traveling direction grid 50 from the side of the automobile 1 or from above the automobile 1, with the interfering object existing at the position of the interfering object in the interference virtual space 40 in the traveling direction field of view. This allows for significant changes to be made to the wide range of grid portions that overlap with interfering objects that occupy a certain area in the image, such as protrusions on the road surface or people 31.

[0072] In this way, in this embodiment, when the automobile 1 is driven manually, it is easy to understand whether or not there is interference with obstacles around the direction of travel, and as a result, manual driving of the automobile 1 can be well supported.

[0073] [Second embodiment] Next, a manual driving assistance device 10 for an automobile 1 according to a second embodiment of the present invention will be described. Differences from the above-described embodiment will be mainly described below.

[0074] This embodiment differs from the above-described embodiment in the processing of the travel direction grid 50. In the embodiment described above, the grid processing unit 64 deforms and discolors the interfering portions of the travel direction grid 50 so as to correspond to the grid image obtained by projecting the travel direction grid 50. This makes it possible to highlight not only interfering objects on the road surface but also hollow interfering objects. However, when processing is performed to project the traveling direction grid 50, as shown in Figure 13, for people 31 and stones 32 with a certain area, the traveling direction grid 50 is deformed over a relatively large area, making it easy to see, whereas for hollow objects such as branches 34, the traveling direction grid 50 is less likely to be deformed significantly. Even if such a deformed traveling direction grid 50 is superimposed on an image captured by the outside camera 12, the deformed parts of the traveling direction grid 50 may not be noticeable. In this embodiment, the travel direction grid 50 is also significantly deformed for hollow branches 34 and the like.

[0075] FIG. 14 is an explanatory diagram showing an example of processing the travel direction grid 50 in the second embodiment of the present invention. When the grid processing unit 64 acquires information on the objects 41, 42, 44 that are interference candidates that are determined to interfere with the traveling direction grid 50, the grid processing unit 64 changes the deformed portions 53, 54, 55 of the traveling direction grid 50 that overlap with each interference candidate object to a color or shape different from that of the other portions of the traveling direction grid 50. In the grid processing, the grid processing unit 64 places a traveling direction grid 50 in the interference virtual space 40 of the traveling direction field of view where the interference candidate object exists at the position of the interference candidate object.

[0076] Then, the grid processing unit 64 deforms the shape of the interfering portion of the travel direction grid 50 so that it envelops and covers the interference candidate from the inside, so that the travel direction grid 50 does not intersect with the interference candidate. In this case, the portion of the travel direction grid 50 that overlaps with the collision candidate will be deformed so as to extend along the surfaces of the objects 41, 42, 44 of the collision candidate. In particular, the object 44 made up of multiple branches is deformed so as to enclose the entire object from the inside. In this case, for the objects 44 of the multiple branches, instead of placing an object for each branch 34, an object that groups the multiple branches 34 together and surrounds the entire range in which the multiple branches 34 exist may be used. Furthermore, the grid processing unit 64 assigns these deformed portions a color attribute that stands out more than the basic color of the travel direction grid 50. In Fig. 14, since it is not possible to differentiate colors in the drawing, they are made to stand out by using thick dashed lines.

[0077] FIG. 15 is an explanatory diagram of an example of a display image displayed on the display device 21. As shown in FIG. In the display image of FIG. 15, the processed traveling direction grid 50 of FIG. 14 is superimposed on the captured image of the traveling direction field of view of FIG. Thus, in the displayed image of FIG. 15, the upper left deformed portion 55 of the travel direction grid 50 is curved inward, avoiding all of the multiple branches 34 . In the displayed image of FIG. 12, the upper left portion of the travel direction grid 50 is discretely deformed and discolored only in the area where it overlaps with the branch 34 . In contrast to this, the deformed portion at the top left of the travel direction grid 50 in the display image of FIG. 15 is conspicuous.

[0078] As a result, in this embodiment, not only protrusions such as stones 32 on the road surface and people 31, but also thin and inconspicuous potential interference objects in the image, such as branches 34 of standing trees 4 in winter, can be made to stand out by significantly changing the relevant parts of the traveling direction grid 50.

[0079] [Third embodiment] Next, a manual driving assistance device 10 for an automobile 1 according to a third embodiment of the present invention will be described. Differences from the above-described embodiment will be mainly described below. This embodiment is suitable for the case where the driving control device 17 executes control to drive the automobile 1 by automatic driving and stop the automobile in a parking space.

[0080] FIG. 16 is an explanatory diagram of a second frame 56 that is larger in width and height than the first frame 52 and that can be used by the cruise control device 17 for automatic driving in the third embodiment of the present invention. FIG. 16 shows the second frame 56 together with the first frame 52 and the automobile 1 of FIG.

[0081] The second frame 56 is larger in the width direction and height direction than the first frame 52 which corresponds to the actual width and height of the automobile 1 . The second frame 56 can be generated using the vehicle data and user setting data stored in the memory 26, similar to the first frame 52. The memory 26 may store information such as the vehicle class of the automobile 1, the minimum ground clearance, the opening and closing height of the rear gate, the front-to-rear length of the automobile 1 with the rear gate open, the overhang height of luggage on the roof, and the extra width for opening and closing the doors of the automobile 1. Then, in determining whether parking is possible by automatic driving in step ST15 of Figure 17 described later, the driving control device 17 may determine whether parking is possible in the parking space without interference by automatic driving for the area corresponding to the second frame body 56 which is larger than the first frame body 52.

[0082] In this way, by making the second frame body 56, which the driving control device 17 uses to determine whether or not automatic driving is possible, larger than the first frame body 52, the possibility of interference with the automobile 1 during automatic driving can be reduced. It is expected that autonomously driving vehicles 1 will always be able to drive safely with a margin of safety relative to their surroundings. In addition, a certain width of space can be left around the vehicle 1 that has stopped in a parking space by autonomous driving, allowing passengers such as the driver to get in and out of the vehicle 1 that has stopped in a parking space by autonomous driving with ample space to maneuver.

[0083] On the other hand, since the second frame body 56 used by the driving control device 17 to determine whether or not automatic driving is possible is larger than the first frame body 52, there is a higher possibility that the driving control device 17 will determine that automatic driving is not possible. Furthermore, if the driving control device 17 determines that automatic driving is not possible, the driver of the automobile 1 will be forced to manually drive the automobile 1 and stop it in a parking space.

[0084] FIG. 17 is a flowchart of the driving control for the driving control device 17 to automatically drive the automobile 1 to stop it in a parking space. The driving control device 17 in FIG. 3 repeatedly executes the driving control in FIG. 17 in order to automatically drive the automobile 1 to a parking space and stop the automobile 1 there.

[0085] In step ST11, the driving control device 17 determines whether or not to park the automobile 1 in a parking space by automatic driving. For example, when the automobile 1 is to be stopped in a parking space after traveling by automatic driving, the driving control device 17 determines that the automobile 1 is to be stopped in the parking space by automatic driving. In addition, for example, when the automobile 1 is parked in a parking space after being driven manually, the driving control device 17 determines that the automobile 1 will be parked in the parking space by automatic driving based on the operation of the assistance button 20. If the assistance button 20 is not operated when parking the automobile 1 in a parking space after driving manually, the driving control device 17 determines that the automobile 1 will not be parked in the parking space by automatic driving. Then, when determining that the automobile 1 should be parked in the parking space by automatic driving, the driving control device 17 advances the processing to step ST12. If it is not determined that the automobile 1 is to be parked in the parking space by automatic driving, the driving control device 17 ends this control.

[0086] In step ST12, the driving control device 17 determines whether or not there is a malfunction in the automatic driving function of the automobile 1. The autonomous driving function of the automobile 1 is composed of, for example, the vehicle sensor group 11, speed sensor 19, cruise control device 17, etc. shown in Fig. 3. The cruise control device 17 uses observation information from outside the vehicle by the vehicle sensor group 11 to generate a course for the automobile 1 to travel in autonomous driving, and controls the drive device, deceleration device, and steering device (not shown) of the automobile 1 so that the automobile travels along that course. These various devices for the autonomous driving function of the automobile 1 can malfunction. If there is a malfunction in any of the various devices for the automatic driving function of the automobile 1, the driving control device 17 determines that there is a malfunction in the automatic driving function of the automobile 1, and proceeds to step ST13. If there is no malfunction in any of the various devices for the automatic driving function of the automobile 1, the driving control device 17 determines that there is no malfunction in the automatic driving function of the automobile 1, and proceeds to step ST14.

[0087] In step ST13, the driving control device 17 determines whether or not degraded operation of the automatic driving function of the automobile 1 is possible. For example, when the automobile 1 is parked in a parking space by automatic driving, it is not necessary for all sensors in the vehicle sensor group 11 in FIG. 3 to be operating normally. For example, even if there is a malfunction in the GNSS receiver 16, it is possible to perform automatic driving control of the automobile 1. In this case, the driving control device 17 may determine that it is possible to perform automatic driving control to stop the automobile 1 in a parking space using some of the devices for the automatic driving function that are not malfunctioning. On the other hand, if all sensors in the vehicle sensor group 11 are malfunctioning, the driving control device 17 may determine that it is not possible to perform automatic driving to control the vehicle 1 to park in a parking space, using some of the devices for the automatic driving function that are not malfunctioning. If degraded operation of the automatic driving function of the automobile 1 is possible, the driving control device 17 proceeds to step ST14. If degraded operation of the automatic driving function of the automobile 1 is not possible, the driving control device 17 advances the process to step ST17.

[0088] In step ST14, the driving control device 17 acquires information for determining the parking space. For example, the driving control device 17 may acquire an image captured in the traveling direction by the exterior camera 12 and information about the second frame 56, which is larger than the first frame 52.

[0089] In step ST15, the driving control device 17 determines whether parking is possible by automatic driving based on the determination information about the parking space acquired in step ST14. For example, if the parking space is small, if there is a person 31 in the parking space, or if the road surface of the parking space is significantly uneven, it is not desirable to drive the car 1 into the parking space by automatic driving. Furthermore, in automated driving, it is desirable to check for interference not only with stones 32 on the road surface of the parking space or people 31, but also with branches 34 of hollow standing trees 4 in the parking space, and to move the car 1 into a parking space where interference will not occur. Furthermore, in most cases, parking spaces are set up to a size that allows a normal-sized car 1 to park there, but depending on the convenience of the person who sets up the space, they may be set up to a size that is too small for a normal-sized car 1. If the parking space is small like this or if there is a potential interfering object in the parking space, there is a possibility that the car 1 will not be able to park in the parking space by autonomous driving.

[0090] The driving control device 17 acquires information used for these determinations in step ST14, and determines in this process whether or not the parking space is suitable for parking by automatic driving. For example, the driving control device 17 may determine whether the parking space is wide enough for the autonomously driven automobile 1 to proceed through, based on an image captured in the traveling direction by the exterior camera 12 and the second frame body 56. The driving control device 17 may compare the width between the pair of boundary lines 3 that define the parking space with the width of the second frame body 56, and determine that the parking space is accessible for autonomous driving if the width of the second frame body 56 is narrower than the width between the pair of boundary lines 3. Furthermore, when driving the automobile 1 into a parking space by autonomous driving, the driving control device 17 may determine whether or not the automobile 1 will interfere with stones 32 on the road surface, people 31, and branches 34 of the hollow tree 4 in the captured image that are in the parking space. The driving control device 17 may generate an interference virtual space 40 that includes the traveling direction of the automobile 1 corresponding to the captured image in the traveling direction field of view of FIG. 7, for example, and move the second frame body 56 in the steering direction in the interference virtual space 40. In this case, the driving control device 17 can determine whether or not the automobile 1 will interfere with stones 32 on the road surface, people 31, and branches 34 of the hollow tree 4 in the parking space in the captured image, based on the trajectory of movement of the second frame body 56 and whether or not there is interference with objects that are potential interferers in the interference virtual space 40. For example, if the second frame body 56 does not interfere with stones 32 on the road surface or people 31 included in the captured image, and does not interfere with branches 34 of hollow standing trees 4 included in the captured image, the cruise control device 17 may determine that the parking space is one into which the autonomously driven automobile 1 can proceed without interference. On the other hand, if the second frame body 56 interferes with any one of a protrusion on the road surface, people 31, and branches 34 of standing trees 4, the cruise control device 17 may determine that the parking space is one into which the autonomously driven automobile 1 cannot proceed without interference. Furthermore, if there is a step in the captured image when entering a parking space, the driving control device 17 may compare it with the information on the minimum ground clearance recorded in the memory 26 of the automobile 1. Then, if the step is equal to or less than the minimum ground clearance, the driving control device 17 may determine that the parking space can be entered by automatic driving. If it is determined that the vehicle can proceed in all of these determinations, the driving control device 17 determines that the parking space is suitable for parking by automatic driving, and proceeds to step ST16. On the other hand, if it is determined that any one of the determinations does not indicate that the vehicle can proceed, the driving control device 17 determines that the parking space is not suitable for parking by automatic driving, and proceeds to step ST17.

[0091] In step ST16, the driving control device 17 starts to execute driving control of the automobile 1 for parking by automatic driving. The driving control device 17 repeatedly acquires detection information from outside the vehicle by the vehicle sensor group 11, repeatedly generates a route for the automobile 1 to proceed to the parking space by automatic driving, and controls the drive device, reduction gear, and steering device (not shown) of the automobile 1 so that the automobile 1 travels along this continually updated route. As a result, the driving control device 17 can proceed toward the parking space and stop the vehicle in the parking space, for example, as shown in Fig. 2. When parking is possible in the parking space, the driving control device 17 executes driving control by automatic driving to stop the vehicle in the parking space, and can park the vehicle 1 in the parking space by automatic driving. Thereafter, the driving control device 17 ends this control.

[0092] Step ST17 is executed when the automobile 1 cannot be parked in the parking space by autonomous driving. The driving control device 17 notifies the driver of the automobile 1 via the display device 21 or the speaker 22 that the automobile 1 cannot be parked in the parking space by autonomous driving. The driving control device 17 then ends this control. If parking in the parking space is not possible, the driving control device 17 does not perform driving control by autonomous driving to stop the automobile in the parking space.

[0093] In this way, the driving control device 17 can drive the automobile 1 automatically based on the detection information of the vehicle sensor group 11 so as not to cause interference.

[0094] FIG. 18 is a flowchart of the support activation control during manual driving, which is executed by the CPU 27 of the support image generating device 23 of FIG. The CPU 27 of the manual driving assistance device 10 of FIG. 3 repeatedly executes the assistance activation control during manual driving of FIG. 18 instead of FIG. Then, if the setting information for assistance regarding manual driving of the automobile 1 is not a value that permits assistance in step ST3, the CPU 27 of the manual driving assistance device 10 advances the process to step ST21. Furthermore, if the current vehicle speed is not equal to or less than the vehicle speed threshold in step ST4, the CPU 27 of the manual driving assistance device 10 advances the process to step ST21.

[0095] In step ST21, the CPU 27 determines whether or not it is not possible to park the automobile 1 in the parking space by automatic driving. If the driving control device 17 notifies the driver that the automobile 1 cannot be parked in the parking space by autonomous driving, the CPU 27 determines that the automobile 1 cannot be parked in the parking space by autonomous driving. In this case, the CPU 27 proceeds to step ST5. In step ST5, the CPU 27 activates assistance control during manual driving. As a result, a screen such as that shown in FIG. 13 or 15 is displayed on the display device 21. The driver can manually drive the automobile 1 to the parking space and stop it, instead of the automobile 1 that cannot be driven autonomously. At this time, the driver can check factors that hinder parking by autonomous driving on the screen of the display device 21 and drive the automobile 1 in a way that prevents interference with potential interfering objects, such as hollow branches 34. On the other hand, if the driving control device 17 has not notified the driver that the automobile 1 cannot be parked in the parking space by automatic driving, the CPU 27 determines that the automobile 1 cannot be parked in the parking space by automatic driving. In this case, the CPU 27 proceeds to step ST6. In step ST6, the CPU 27 stops the assistance control during manual driving.

[0096] In this way, for example, when the driving control unit does not determine that parking is possible in the parking space, the CPU 27 can cause the display device 21 to display an image in which the processed traveling direction grid 50 is superimposed on the captured image of the traveling direction field of view. Furthermore, when there is a problem with the vehicle sensor group 11 or the driving control unit, the CPU 27 can cause the display device 21 to display an image in which the processed traveling direction grid 50 is superimposed on a captured image of the traveling direction field of view.

[0097] As described above, in this embodiment, when the driving control unit of the automobile 1 performs driving control to drive the automobile 1 by automatic driving and stop the automobile 1 in a parking space, the driving control unit determines whether parking is possible without interfering with the parking space within an area corresponding to the second frame body 56 that is wider or taller than the first frame body 52 and has a width equal to or greater than the passing width and height equal to or greater than the passing height of the automobile 1. If it determines that parking is possible without interfering with the parking space, the driving control unit of the automobile 1 drives the automobile 1 by automatic driving and stops the automobile 1 in the parking space. As a result, the automobile 1 can be parked by automatic driving without interfering with the parking space. Furthermore, the driver and other occupants of the automobile 1 can easily get in and out of the automobile 1 by using the extra parking space provided around the automobile 1 by the area corresponding to the second frame 56, which is larger in width or height than the first frame 52 and has a width equal to or greater than the passing width and height equal to or greater than the passing height of the automobile 1. The occupants of the automobile 1 will have a good impression of the parking performed by the automatic driving of the driving control unit. Furthermore, in this embodiment, if there is a malfunction in the vehicle sensor group 11 or the driving control unit, the control unit executes a display process to display an image in which the processed traveling direction grid 50 is superimposed on a captured image of the traveling direction field of view on the display device 21. This allows the occupants of the automobile 1 to manually drive the automobile 1 to the parking space and park the automobile 1 in the parking space when there is a malfunction in the vehicle sensor group 11 or the driving control unit and automatic driving is not possible.

[0098] The above-described embodiment is an example of a preferred embodiment of the present invention, but the present invention is not limited to this, and various modifications and changes are possible within the scope of the gist of the invention.

[0099] The above-described embodiment mainly describes an example in which the automobile 1 is manually or automatically driven to proceed to a parking space and stop there. The present invention is applicable not only to cases where the automobile 1 is manually or automatically driven to park in a parking space, but also to cases where the automobile 1 is traveling on a road, for example. For example, when the automobile 1 is being driven manually or automatically on a road or the like, the control unit of the manual driving assistance device 10 may cause the display device 21 to display an image of the forward view onto which a forward view grid 50 in the forward view that has been deformed in accordance with potential interfering objects is superimposed, as in the above-described embodiment.

[0100] In the third embodiment described above, the driving control device 17 directly controls the second frame 56 to determine whether the automobile 1 can be parked in a parking space by automatic driving. In addition, for example, the driving control device 17 may determine whether the automobile 1 can be parked in a parking space by automatic driving within the range corresponding to the second frame body 56, without directly using the second frame body 56 for control. The automobile 1 is provided with an exterior camera facing forward, for example. Such an exterior camera can capture images with a wide angle of view, for example, about 160 degrees. In this case, the cruise control device 17 can extract interfering objects from the captured image with the wide angle of view and determine whether they interfere with the vehicle itself. In this way, even though the second frame 56 is not directly used for control, the cruise control device 17 essentially determines whether the automobile 1 can park in a parking space by autonomous driving within the range corresponding to the second frame 56. [Explanation of symbols]

[0101] 1...Automobile (vehicle), 2...Other automobiles, 3...Boundary line, 4...Standing tree, 10...Manual driving assistance device, 11...Vehicle sensor group, 12...External camera, 13...Millimeter wave radar, 14...Infrared sensor, 15...Lidar, 16...GNSS receiver, 17...Driving control device, 18...Steering sensor, 19...Speed sensor, 20...Assistance button, 21...Display device, 22...Speaker, 23...Assistance image generation device, 24...Timer, 25...Interface unit, 26...Memory, 27...CPU, 28...Internal bus, 30...Captured image Image, 31...person, 32...stone, 34...branch, 40...interference virtual space, 41...person object, 42...stone object, 43...boundary line object, 44...branch object, 50, 51...travel direction grid, 52...first frame body, 53, 54, 55...deformed portion, 56...second frame body, 60...interference candidate object extraction unit, 61...interference virtual space generation unit, 62...grid generation unit, 63...grid interference determination unit, 64...grid processing unit, 65...grid image generation and output unit, 66...remaining distance determination unit, 67...alarm output unit

Claims

1. a group of vehicle sensors including a camera that captures images in the traveling direction of the vehicle; a display device capable of displaying an image captured by the camera in the field of view in the direction of travel; a control unit connected to the vehicle sensor group and the display device; and The control unit a grid generation process for generating a traveling direction grid that indicates a passing range of the vehicle when traveling, to be superimposed on the captured image; an interference candidate extraction process for extracting an interference candidate that may interfere with the vehicle when the vehicle travels in the traveling direction, based on the detection information of the vehicle sensor group; a grid processing process for changing a portion of the generated traveling direction grid that overlaps with the interference candidate object to a color or shape different from that of other portions of the traveling direction grid; a display process of converting the processed traveling direction grid into a planar image in a traveling direction field of view, outputting the planar image, and displaying an image on the display device in which the planar image in the traveling direction field of view of the processed traveling direction grid is superimposed on the captured image in the traveling direction field of view; To execute Manual driving assistance device for vehicles.

2. The control unit, in the interference candidate extraction process, extracting, as interfering candidates that may interfere with the vehicle, interfering candidates that are present in the passing range on the road surface on which the vehicle is traveling, and hollow interfering candidates that may interfere in the passing range when the vehicle is traveling; The manual driving assistance device for a vehicle according to claim 1.

3. The control unit, in the grid generation process, generating the traveling direction grid indicating a passing range of the vehicle when traveling based on a trajectory of a first frame body, the width of which is equal to or greater than the width and the height of which is equal to or greater than the height of the vehicle, traveling on a road surface in a virtual space corresponding to the captured image of the traveling direction field of view displayed on the display device; 3. The manual driving assistance device for a vehicle according to claim 2.

4. The control unit, in the grid processing, determining whether or not the interference candidate extracted in the interference candidate extraction process, which has the possibility of interfering with the vehicle, interferes with the traveling direction grid generated in the grid generation process in the virtual space; If the interference candidate object interferes in the virtual space, a portion of the traveling direction grid where the traveling direction grid and the interference candidate object overlap is changed to a color or shape different from that of other portions of the grid.

4. The manual driving assistance device for a vehicle according to claim 3.

5. The control unit, in the grid processing, deforming a shape of the interfering portion of the traveling direction grid so as to correspond to a grid image obtained by projecting the traveling direction grid in a state in which the interfering object is present at a position of the interfering object in a virtual space including the traveling direction; 5. The manual driving assistance device for a vehicle according to claim 4.

6. The control unit, in the grid processing, When the traveling direction grid is arranged in a state in which the interference candidate exists at the position of the interference candidate in a virtual space including the traveling direction, the shape of the interfering portion of the traveling direction grid is deformed so as to enclose the interference candidate from inside the traveling direction grid so that the traveling direction grid does not intersect with the interference candidate.

5. The manual driving assistance device for a vehicle according to claim 4.

7. a driving control unit that causes the vehicle to drive automatically based on detection information from the vehicle sensor group, The traveling control unit During travel control for automatically driving the vehicle and stopping the vehicle in a parking space, a determination is made as to whether parking is possible without interfering with the parking space in an area corresponding to a second frame body that is larger in width direction or height direction than a first frame body that has a width equal to or larger than the passing width and a height equal to or larger than the passing height of the vehicle, When parking is possible without interfering with the parking space, automatic driving control is performed to stop the vehicle in the parking space; The control unit When the driving control unit does not determine that parking in the parking space is possible, or when there is a malfunction in the vehicle sensor group or the driving control unit, a display process is executed to display on the display device an image in which the processed traveling direction grid is superimposed on an image captured in the traveling direction field of view. The manual driving assistance device for a vehicle according to any one of claims 1 to 6.

Images