Parking assistance method and parking assistance device

Abstract

To provide a parking support method and parking support device which allow a user to easily grasp the positional relationship between a travel route of a vehicle based on teaching travel and an object and a road around the vehicle.SOLUTION: A parking support method according to the present disclosure performing automatic travel of a vehicle on the basis of teaching travel by a driver comprises: a first acquisition step of acquiring a photographed image obtained by imaging the periphery of the vehicle during the teaching travel in time series together with the movement of the vehicle from an imaging device provided in the vehicle; a route recording step of recording the travel route of the vehicle in the teaching travel; a first expanded overhead image generation step of generating a first expanded overhead image in which a display object range including the travel route and the periphery of the travel route is overlooked from above on the basis of the photographed image; and a first display step of displaying the first expanded overhead image on a display part.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present disclosure relates to a parking assistance method and a parking assistance device. [Background technology]

[0002] Conventionally, parking assistance technologies that move a vehicle by automatic driving when parking the vehicle are known. One such parking assistance technology is a technology that learns a driving route based on teacher driving by the driver and performs parking assistance using the learning results. This technology is used when parking in a fixed parking position repeatedly, such as a parking lot at the user's home or workplace. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 3286306 Summary of the Invention [Problem to be solved by the invention]

[0004] However, it may be difficult for a user to objectively grasp the vehicle's driving route based on the supervised driving and the positional relationship between the objects and roads around the vehicle.

[0005] The present disclosure provides a parking assistance method and a parking assistance device that enable a user to easily grasp the positional relationship between a vehicle's driving path based on supervised driving and objects and roads around the vehicle. [Means for solving the problem]

[0006] The parking assistance method according to the present disclosure is a parking assistance method for automatically driving a vehicle based on teacher driving by a driver, and includes a first acquisition step of acquiring captured images of the surroundings of the vehicle during teacher driving in time series together with the movement of the vehicle from an imaging device provided in the vehicle, a route recording step of recording the driving route of the vehicle during teacher driving, a first extended overhead image generation step of generating a first extended overhead image of a display target range including the driving route and the surroundings of the driving route from above based on the captured images, a first display step of displaying the first extended overhead image on a display unit, and a start / end operation reception step of receiving an operation of starting and ending the recording of the teacher driving by a user. In the first display step, an overhead route image in which a driving route image showing the recorded driving route is superimposed on the first extended overhead image is displayed on the display unit. In addition, when an operation of ending the recording of the teacher driving is received in the first display step, the display of the overhead route image by the first display step is executed. In the first display step, the overhead route image is displayed before an operation for registering the teacher's run is accepted. In the first acquisition step, information on the distance between the vehicle and objects around the vehicle during teacher driving is acquired. In the first display step, an outline image showing a movement trajectory of the outline of the vehicle body during teacher driving is displayed superimposed on the first extended overhead image, and a portion of the outline image whose distance is equal to or less than a threshold is displayed on the overhead route image in a manner different from a portion whose distance is greater than the threshold. Effect of the Invention

[0007] According to the parking assistance method and parking assistance device disclosed herein, the user can easily grasp the vehicle's driving path based on teacher driving and the positional relationship between the objects and roads around the vehicle. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an example of a vehicle equipped with a parking assistance device according to a first embodiment. [Diagram 2] FIG. 2 is a diagram showing an example of a configuration in the vicinity of a driver's seat of a vehicle according to the first embodiment. [Diagram 3] FIG. 3 is a diagram illustrating an example of a hardware configuration of the parking assistance device according to the first embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of teacher running according to the first embodiment. [Diagram 5]FIG. 5 is a block diagram showing an example of functions included in the parking assistance device according to the first embodiment. [Figure 6] FIG. 6 is a diagram showing an example of a driving route image according to the first embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of an outline image according to the first embodiment. [Figure 8] FIG. 8 is a diagram illustrating an example of an overhead image according to the first embodiment. [Figure 9] FIG. 9 is a diagram illustrating an example of an extended overhead image according to the first embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of an overhead route image according to the first embodiment. [Figure 11] FIG. 11 is a flowchart showing an example of the flow of a driving route recording process executed by the parking assistance device according to the first embodiment. [Figure 12] FIG. 12 is a diagram illustrating an example of an overhead route image according to the second embodiment. [Figure 13] FIG. 13 is a diagram illustrating an example of an overhead route image according to the third embodiment. [Figure 14] FIG. 14 is a diagram showing an example of a travel route selection screen according to the fourth embodiment. [Figure 15] FIG. 15 is a flowchart showing an example of the flow of a driving route recording process executed by the parking assistance device according to the fourth embodiment. [Figure 16] FIG. 16 is a diagram illustrating an example of an overhead route image according to the fifth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, embodiments of a parking assistance method and a parking assistance device according to the present disclosure will be described with reference to the drawings.

[0010] (First embodiment) Fig. 1 is a diagram showing an example of a vehicle 1 equipped with a parking assistance device 100 according to the first embodiment. As shown in Fig. 1, the vehicle 1 includes a vehicle body 12 and two pairs of wheels 13 arranged along a predetermined direction on the vehicle body 12. The two pairs of wheels 13 include a pair of front tires 13f and a pair of rear tires 13r.

[0011] The front tire 13f shown in Fig. 1 is an example of a first wheel in this embodiment. The rear tire 13r is an example of a second wheel in this embodiment. Although the vehicle 1 shown in Fig. 1 has four wheels 13, the number of wheels 13 is not limited to this. For example, the vehicle 1 may be a two-wheeled vehicle.

[0012] The vehicle body 12 is coupled to wheels 13 and can move by the wheels 13. In this case, the predetermined direction in which the two pairs of wheels 13 are arranged is the traveling direction of the vehicle 1. The vehicle 1 can move forward or backward by switching gears (not shown) or the like. The vehicle 1 can also turn right or left by steering.

[0013] The vehicle body 12 has a front end F, which is an end on the front tire 13f side, and a rear end R, which is an end on the rear tire 13r side. The vehicle body 12 has a substantially rectangular shape when viewed from above, and the four corners of the substantially rectangular shape may be called ends. The vehicle 1 also includes a display device, a speaker, and an operation unit, which are not shown in FIG.

[0014] A pair of bumpers 14 are provided at the front and rear ends F, R of the vehicle body 12 near the lower end of the vehicle body 12. Of the pair of bumpers 14, the front bumper 14f covers the entire front surface and part of the side surface near the lower end of the vehicle body 12. Of the pair of bumpers 14, the rear bumper 14r covers the entire rear surface and part of the side surface near the lower end of the vehicle body 12.

[0015] At predetermined ends of the vehicle body 12, wave transmitting / receiving units 15f, 15r for transmitting and receiving sound waves such as ultrasonic waves are disposed. For example, one or more wave transmitting / receiving units 15f are disposed at the front bumper 14f, and one or more wave transmitting / receiving units 15r are disposed at the rear bumper 14r. Hereinafter, when the wave transmitting / receiving units 15f, 15r are not particularly limited, they are simply referred to as the wave transmitting / receiving unit 15. In addition, the number and positions of the wave transmitting / receiving units 15 are not limited to the example shown in FIG. 1. For example, the vehicle 1 may be provided with wave transmitting / receiving units 15 on the left and right sides.

[0016] In this embodiment, a sonar using ultrasonic waves will be described as an example of the wave transmitting and receiving unit 15, but the wave transmitting and receiving unit 15 may be a radar that transmits and receives electromagnetic waves. Alternatively, the vehicle 1 may be equipped with both a sonar and a radar. Also, the wave transmitting and receiving unit 15 may simply be called a sensor.

[0017] The wave transmitting / receiving unit 15 detects obstacles around the vehicle 1 based on the results of transmitting and receiving sound waves or electromagnetic waves. In addition, the wave transmitting / receiving unit 15 measures the distance between the vehicle 1 and obstacles around the vehicle 1 based on the results of transmitting and receiving sound waves or electromagnetic waves.

[0018] The vehicle 1 also includes a first imaging device 16a that images the area in front of the vehicle 1, a second imaging device 16b that images the area behind the vehicle 1, a third imaging device 16c that images the area to the left of the vehicle 1, and a fourth imaging device that images the area to the right of the vehicle 1. The fourth imaging device is not shown in the figure.

[0019] Hereinafter, when the first imaging device 16a, the second imaging device 16b, the third imaging device 16c, and the fourth imaging device are not particularly distinguished from each other, they will be simply referred to as imaging device 16. The positions and the number of imaging devices are not limited to the example shown in FIG. 1. For example, the vehicle 1 may also be equipped with only two imaging devices, the first imaging device 16a and the second imaging device 16b. Alternatively, the vehicle 1 may have other imaging devices in addition to the above examples.

[0020] The imaging device 16 is capable of capturing images of the surroundings of the vehicle 1, and is, for example, a camera that captures color images. The captured images captured by the imaging device 16 may be moving images or still images. The imaging device 16 may be a camera built into the vehicle 1, or may be a camera of a drive recorder that is attached to the vehicle 1 later.

[0021] The vehicle 1 is also equipped with a parking assistance device 100. The parking assistance device 100 is an information processing device that can be installed in the vehicle 1, and is, for example, an ECU (Electronic Control Unit) or an OBU (On Board Unit) provided inside the vehicle 1. Alternatively, the parking assistance device 100 may be an external device installed near the dashboard of the vehicle 1. The parking assistance device 100 may also function as a car navigation device or the like.

[0022] Next, a configuration of the vicinity of the driver's seat of the vehicle 1 according to this embodiment will be described. Fig. 2 is a diagram showing an example of a configuration of the vicinity of the driver's seat 130a of the vehicle 1 according to the first embodiment.

[0023] 2, the vehicle 1 includes a driver's seat 130a and a passenger seat 130b. In front of the driver's seat 130a, a windshield 180, a dashboard 190, a steering wheel 140, a display device 120, and operation buttons 141 are provided.

[0024] The display device 120 is a display provided on a dashboard 190 of the vehicle 1. As an example, the display device 120 is located in the center of the dashboard 190 as shown in FIG. 2. The display device 120 is, for example, a liquid crystal display or an organic EL (Electro Luminescence) display. The display device 120 may also function as a touch panel. The display device 120 is an example of a display unit in this embodiment.

[0025] The steering wheel 140 is provided in front of the driver's seat 130a and can be operated by the driver. The rotation angle of the steering wheel 140, i.e., the steering angle, is electrically or mechanically linked to the change in the direction of the front tires 13f, which are the steered wheels. The steered wheels may be the rear tires 13r, or both the front tires 13f and the rear tires 13r may be steered wheels.

[0026] The operation button 141 is a button that can accept an operation by a user. In this embodiment, the user is, for example, the driver of the vehicle 1. The operation button 141 accepts an operation from the driver to start parking assistance, for example, by accepting a press from the driver. The position of the operation button 141 is not limited to the example shown in FIG. 2, and may be provided on the steering wheel 140, for example. The operation button 141 is an example of an operation unit in this embodiment. In addition, when the display device 120 also serves as a touch panel, the display device 120 may be an example of an operation unit. In addition, an operation terminal that can transmit a signal to the vehicle 1 from outside the vehicle 1, such as a tablet terminal, a smartphone, a remote controller, or an electronic key (not shown), may be an example of the operation unit.

[0027] Next, a hardware configuration of the parking assistance device 100 will be described. Fig. 3 is a diagram showing an example of a hardware configuration of the parking assistance device 100 according to the first embodiment. As shown in Fig. 3, the parking assistance device 100 has a CPU (Central Processing Unit) 11A, a ROM (Read Only Memory) 11B, a RAM (Random Access Memory) 11C, an I / F (Interface) 11D, a HDD (Hard Disk Drive) 11E, etc., which are connected to each other via a bus 11F, and has a hardware configuration using a normal computer.

[0028] The CPU 11A is a calculation device that controls the entire ECU. The CPU 11A is an example of a processor in the parking assistance device 100 of this embodiment, and another processor or processing circuit may be provided instead of the CPU 11A. The ROM 11B stores programs and the like that realize various processes by the CPU 11A. The RAM 11C is, for example, a main storage device of the parking assistance device 100, and stores data necessary for various processes by the CPU 11A. The I / F 11D is an interface for transmitting and receiving data. The I / F 11D transmits and receives data to and from other devices provided in the vehicle 1, for example, the display device 120. The I / F 11D may also transmit and receive information to and from other ECUs mounted on the vehicle 1 via a CAN (Controller Area Network) or the like in the vehicle 1, or may communicate with an information processing device outside the vehicle 1 via a network such as the Internet.

[0029] 1 to 3, the display device 120 is illustrated as a device separate from the parking assistance device 100, but the display device 120 may be included in the parking assistance device 100.

[0030] The parking assistance device 100 of this embodiment learns a driving route based on the teacher driving by the driver, and performs parking assistance using the learning result. In other words, the parking assistance method executed by the parking assistance device 100 is a method of automatically driving the vehicle 1 based on the teacher driving by the driver. This parking assistance method is effective for reducing the driver's parking effort when repeatedly parking in a fixed parking position, such as the garage of the driver's home, a contracted parking position of an apartment building, or a specified parking position in a parking lot of the workplace, for example. This type of parking assistance is called home zone parking.

[0031] Fig. 4 is a diagram illustrating an example of teacher driving according to the first embodiment. In the example shown in Fig. 4, a parking space 910 located near a home 70 of a driver of the vehicle 1 is a target parking position of the vehicle 1. In addition, a garden 721 and a corner space 722 exist near the parking space 910. In the garden 721, there are trees 73a and 73b on the side facing the parking space 910.

[0032] In addition, around the parking space 910, there are other houses 71a and 71b, parking spaces 711a to 711c of the other houses 71a and 71b, and other vehicles 30a to 30c. The trees 73a and 73b in the yard 721, the other houses 71a and 71b, parking spaces 711a to 711c of the other houses 71a and 71b, and other vehicles 30a to 30c may become obstacles when the vehicle 1 is parked in the parking space 910. For this reason, the driver drives the vehicle 1 while avoiding these objects during teacher driving. Note that objects that may become obstacles for the vehicle 1 are not limited to these objects.

[0033] In the teacher driving, the driver drives the vehicle 1 by manual driving from a start position 900 on the road 40 to a parking space 910. The parking assistance device 100 records a driving route 80 from the start position 900 to the parking space 910 in the teacher driving. When the user determines the recorded driving route 80 as the driving route 80 for automatic driving, the parking assistance device 100 registers the driving route 80 as the driving route 80 for automatic driving. Note that the user may repeat the teacher driving multiple times until the desired driving route 80 is obtained.

[0034] After the registration of the driving route 80 based on such teacher driving has been completed, the parking assistance device 100 can park the vehicle 1 in the parking space 910 by automatically driving the vehicle 1 along the registered driving route 80.

[0035] It should be noted that the parking space 910 and its surrounding environment shown in FIG. 4 are merely examples and are not limiting.

[0036] Next, a detailed description will be given of the functions of the parking assistance device 100 of this embodiment. Fig. 5 is a block diagram showing an example of functions provided in the parking assistance device 100 according to the first embodiment.

[0037] As shown in FIG. 5, the parking assistance device 100 of this embodiment includes a reception unit 101, an acquisition unit 102, an extraction unit 103, a route recording unit 104, a route image generation unit 105, an extended overhead image generation unit 106, a display control unit 107, a registration processing unit 108, an estimation unit 109, a vehicle control unit 110, and a memory unit 111.

[0038] The storage unit 111 is configured with, for example, a ROM 11B, a RAM 11C, or a HDD 11E. Although one storage unit 111 is included in the parking assistance device 100 in FIG. 5, a plurality of storage media may function as the storage unit 111.

[0039] The storage unit 111 stores programs and data used in various processes executed by the parking assistance device 100. For example, the programs executed by the parking assistance device 100 of this embodiment are modularized to include the above-mentioned functional units (reception unit 101, acquisition unit 102, extraction unit 103, route recording unit 104, route image generating unit 105, extended overhead image generating unit 106, display control unit 107, registration processing unit 108, estimation unit 109, and vehicle control unit 110), and as actual hardware, the CPU 11A reads out and executes the programs from the storage unit 111, whereby the above-mentioned units are loaded onto the RAM 11C, and the reception unit 101, acquisition unit 102, extraction unit 103, route recording unit 104, route image generating unit 105, extended overhead image generating unit 106, display control unit 107, registration processing unit 108, estimation unit 109, and vehicle control unit 110 are generated on the RAM 11C. The processes realized by each functional unit of the parking assistance device 100 are also called steps.

[0040] The program executed by the parking assistance device 100 of this embodiment is provided as a file in an installable or executable format recorded on a computer-readable recording medium such as a flash memory, a CD-ROM, a flexible disk (FD), a CD-R, or a DVD (Digital Versatile Disk).

[0041] The program executed by the parking assistance device 100 of this embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. The program executed by the parking assistance device 100 of this embodiment may be provided or distributed via a network such as the Internet. The program executed by the parking assistance device 100 of this embodiment may be provided by being pre-installed in ROM 11B or the like.

[0042] The reception unit 101 receives various operations from the user. For example, when the operation button 141 is pressed, the reception unit 101 receives operations to start and end the recording of the teacher run by the user. In this embodiment, the start operation reception step, which is a process of receiving an operation to start the recording of the teacher run, and the end operation reception step, which is a process of receiving an operation to end the recording of the teacher run, are collectively referred to as the start and end operation reception step.

[0043] Furthermore, the reception unit 101 receives a user's operation to instruct whether or not to register the travel route 80 of the supervised travel as a travel route 80 for automatic travel after the supervised travel is completed.

[0044] Furthermore, for example, when the operation button 141 is pressed in a state in which the travel route 80 has already been registered, the reception unit 101 receives an operation to start parking assistance by the user.

[0045] There is no particular limitation on the means by which the reception unit 101 receives a user operation. For example, if the display device 120 is a touch panel, the reception unit 101 may receive an operation to start or end recording of a teacher driving or to start parking assistance when an image button on the touch panel is pressed.

[0046] The acquisition unit 102 acquires a plurality of captured images, which are captured in time series around the vehicle 1 during teacher running, together with the movement of the vehicle 1, from the plurality of imaging devices 16a to 16c. Note that since the captured images are images of the surroundings of the vehicle 1, they are referred to as surrounding images in this embodiment. This acquisition process is an example of a first acquisition step in this embodiment. In addition, in the first acquisition step, the acquisition unit 102 further acquires information regarding the distance between the vehicle 1 and objects around the vehicle 1 during teacher running.

[0047] The information on the distance between the vehicle 1 and an object around the vehicle 1 is, for example, the presence or absence of an obstacle detected by the wave transmitting / receiving unit 15, and the distance between the vehicle 1 and the obstacle around the vehicle 1 detected by the wave transmitting / receiving unit 15. The presence or absence of an obstacle detected by the wave transmitting / receiving unit 15, and the distance between the vehicle 1 and the obstacle around the vehicle 1 measured by the wave transmitting / receiving unit 15 may be referred to as environmental information on the environment around the vehicle 1. Note that the environmental information is not limited to this information.

[0048] Furthermore, the acquisition unit 102 acquires vehicle information of the vehicle 1 from various sensors of the vehicle 1 or other ECUs. The vehicle information includes, for example, information related to the speed, steering angle, and braking operation of the vehicle 1. Each piece of information included in the vehicle information of the vehicle 1 is stored in the storage unit 111 in association with the time at which each piece of information was detected. Note that the vehicle information of the vehicle 1 may further include the wheel speed or the number of rotations of the wheels 13, the acceleration of the vehicle 1 measured by a gyro sensor, etc.

[0049] The extraction unit 103 extracts feature points from the surrounding image. The method of extracting feature points by the extraction unit 103 is not particularly limited, and a known method may be applied. For example, the extraction unit 103 extracts feature points by a method such as FAST (Features from Accelerated Segment Test) or ORB (Oriented FAST and Rotated BRIEF). Furthermore, when learning the travel route 80, the extraction unit 103 may preferentially record feature points that satisfy a specified condition among the extracted feature points. For example, among a plurality of surrounding images consecutive in time series, the feature points extracted from an image in which the vehicle 1 has traveled a longer distance during image capture may be preferentially selected as feature points.

[0050] The route recording unit 104 records the travel route 80 of the vehicle 1 during the teacher driving. The route recording unit 104 estimates the position of the vehicle 1 during the teacher driving based on, for example, changes in feature points extracted from multiple surrounding images and vehicle information of the vehicle 1, and specifies the travel route 80 from the time-series changes in the position. The recording process of the travel route 80 is an example of a route recording step.

[0051] More specifically, the route recording unit 104 identifies a change in the position of the vehicle 1 based on a time series change in the feature points extracted from a plurality of surrounding images captured during the teacher driving. The route recording unit 104 may also correct the position of the vehicle 1 identified from the feature points based on the acquired vehicle information.

[0052] In this embodiment, the information in which the travel route 80 for automatic driving, the speed of the vehicle 1 traveling on the travel route 80, the steering angle, the braking operation, and feature points extracted from a plurality of surrounding images captured along with the movement of the vehicle 1 during teacher driving are associated in time series is referred to as travel route information. The route recording unit 104 stores the travel route information in the storage unit 111. The travel route information is used by the vehicle control unit 110 during automatic driving, which will be described later. Note that the recording method of the travel route 80 and the definition of the travel route information are not limited to this example.

[0053] In addition, the route recording unit 104 defines the environment around the vehicle 1 as a map based on feature points extracted from the surrounding images captured during teacher driving, and stores the map in the storage unit 111. The process of recording the driving route 80 based on teacher driving may be called a learning process.

[0054] The route image generating unit 105 generates a travel route image representing the travel route 80 recorded by the route recording unit 104 .

[0055] FIG. 6 is a diagram showing an example of a driving route image 801 according to the first embodiment. For example, as shown in FIG. 6, the driving route image 801 is an image that shows a line from a start position to an end position of a driving route 80. In the driving route image 801, for example, the driving route 80 is shown based on the movement trajectory of the center of the body 12 of the vehicle 1. In the example shown in FIG. 6, the tip of the line of the driving route image 801 is an arrow indicating the traveling direction of the vehicle 1. Note that the aspect of the driving route image 801 is not limited to this.

[0056] In addition, the route image generating unit 105 generates an outline image that represents the movement trajectory of the outline of the vehicle body 12 of the vehicle 1 during the teacher running.

[0057] FIG. 7 is a diagram showing an example of an outer contour image 820 according to the first embodiment. In FIG. 7, a travel route image 801 is also shown together with the outer contour image 820. The outer contour image 820 represents, for example, the movement trajectory of the outer contour of the vehicle body 12 of the vehicle 1 when the vehicle 1 travels along a travel route 80, as shown in FIG. 7. The route image generating unit 105 estimates the movement trajectory of the outer contour of the vehicle body 12, for example, based on the recorded travel route 80 and the size of the vehicle body 12. It is assumed that the size of the vehicle body 12 is stored in the storage unit 111, for example.

[0058] The contour image 820 of this embodiment includes at least a vehicle width image 821 that represents the movement trajectory of both the left and right ends of the body 12 of the vehicle 1. Furthermore, when the travel route 80 includes a turning point, the contour image includes an end image 822 that represents the position of the front end or rear end of the body 12 of the vehicle 1 at the turning point.

[0059] Returning to Fig. 5, the extended overhead image generating unit 106 generates an extended overhead image that shows an overhead view of the display target range including the travel route 80 and the periphery of the travel route 80 from above, based on the peripheral image. The display target range is, for example, a range including the start position 900 to the end position of the travel route 80 in the teacher's run. The extended overhead image is an example of a first extended overhead image in this embodiment. The process is an example of a first extended overhead image generating step.

[0060] 8 is a diagram showing an example of an overhead image 61 according to the first embodiment. The extended overhead image generating unit 106 converts a plurality of peripheral images captured in time series by the plurality of imaging devices 16a to 16c at a plurality of positions on the travel route 80 into images viewed from a virtual viewpoint and synthesizes the images, thereby generating a plurality of overhead images 61 of the surroundings of the vehicle 1 for each of a plurality of positions on the travel route 80. Such image processing of viewpoint conversion and synthesis is an example of an overhead image generating step.

[0061] The virtual viewpoint is, for example, located directly above the vehicle 1. For this reason, the overhead image is also called a top view. A known technique can be adopted as a method for generating the overhead image 61.

[0062] 8 is a composite image generated based on a plurality of peripheral images captured by the plurality of imaging devices 16a-16c when the vehicle 1 is located in a parking space 910, which is the end point of the travel route 80. Since the imaging ranges of the plurality of imaging devices 16a-16c change as the vehicle 1 moves, the overhead image 61 also differs when the vehicle 1 is located at a different position on the travel route 80.

[0063] The extended overhead image generating unit 106 then synthesizes the multiple overhead images 61 along the time series changes in the capture times of the peripheral images corresponding to each overhead image 61 to generate an image depicting the surroundings of the vehicle 1 along the entire travel route 80. Such an image depicts a wider range than each individual overhead image 61, and is therefore referred to as an extended overhead image in this embodiment. The extended overhead image may also be simply referred to as an overhead image.

[0064] 9 is a diagram showing an example of an extended overhead image 60 according to the first embodiment. The extended overhead image generating unit 106 generates one extended overhead image 60 by stitching together a plurality of overhead images 61 based on feature points included in the plurality of overhead images 61, for example. The extended overhead image 60 includes the road surface or ground on which the vehicle 1 travels from a start position 900 of the travel route 80 to a parking space 910 which is an end position, and a background depicting the periphery of the travel route 80.

[0065] 8 and 9, the extended overhead image generating unit 106 generates a plurality of overhead images 61 from a plurality of peripheral images, and then generates the extended overhead image 60 from the plurality of overhead images 61. However, the flow of the generation process is not limited to this. For example, the extended overhead image generating unit 106 may generate the extended overhead image 60 from a plurality of peripheral images.

[0066] 5, the display control unit 107 superimposes the driving route image 801 on the extended overhead image 60 and causes the display device 120 to display the superimposed image. Hereinafter, the extended overhead image 60 on which the driving route image 801 is superimposed is referred to as an overhead route image. This display process is an example of a first display step in this embodiment.

[0067] Fig. 10 is a diagram showing an example of an overhead route image 62 according to the first embodiment. In the overhead route image 62 shown in Fig. 10, a driving route image 801 and an outer boundary image 820 are superimposed on the extended overhead image 60. Note that the outer boundary image 820 is not essential, and it is sufficient that at least the driving route image 801 is superimposed and displayed on the extended overhead image 60.

[0068] In the example shown in Fig. 10, the display control unit 107 displays a rectangular background region 602 as the background of the extended overhead image 60. Note that the extended overhead image 60 may include the background region 602. Note that, although the background region 602 is displayed in black in Fig. 10, this is merely an example, and the display form is not particularly limited.

[0069] The bird's-eye view route image 62 is displayed, for example, when the user executes an operation to end the teacher's run. When the reception unit 101 receives an operation to end the recording of the teacher's run, the display control unit 107 causes the display device 120 to display the bird's-eye view route image 62.

[0070] For example, in the bird's-eye view route image 62, the road 40 and obstacles around the travel route 80, such as trees 73a and 73b, other houses 71a, other parking spaces 711a and 711c, and other vehicles 30a, are depicted together with the travel route image 801 and the outline image 820. This allows the user to confirm the positional relationship between the vehicle 1 and the surrounding obstacles and roads during teacher travel.

[0071] Furthermore, the display control unit 107 causes the display device 120 to display the overhead route image 62 at an angle such that the forward direction of the vehicle 1 at the end of the teacher run faces the vertical direction of the display device 120. In the example shown in Fig. 10, the vertical direction of the display device 120 is the Y direction, and the horizontal direction is the X direction. The vertical direction is also referred to as the upper side of the screen. The forward direction of the vehicle 1 means the direction in which the front end of the vehicle 1 faces.

[0072] As shown in FIG. 4, in this embodiment, the vehicle 1 moves forward and then reverses to enter the parking space 910 during teacher driving in the present embodiment. Therefore, the front of the vehicle 1 at the end of teacher driving faces the entrance side of the parking space 910, and the driver also faces the entrance side of the parking space 910. In other words, the display control unit 107 displays the overhead route image 62 on the display device 120 so that the front direction of the vehicle 1 at the end of teacher driving faces the Y direction, so that the direction in which the driver actually sees the scenery outside the vehicle 1 at the end of teacher driving matches the direction of the overhead route image 62 displayed on the display device 120. Therefore, the driver can easily grasp the relationship between the driving route 80 and the actual surrounding environment.

[0073] The display control unit 107 also displays a registration button 141a and a redo button 141b on the display device 120 together with the overhead route image 62. The registration button 141a is an image button that accepts a user operation to instruct the registration of the driving route 80 corresponding to the overhead route image 62. The redo button 141b is an image button that accepts a user operation to instruct the user not to register the driving route 80 corresponding to the overhead route image 62. When the registration button 141a and the redo button 141b are pressed, the reception unit 101 receives the operation corresponding to each button. Note that the method of receiving the operation to register or delete the driving route 80 is not limited to this.

[0074] Returning to FIG. 5, the registration processing unit 108 registers the driving route 80 generated by the route recording unit 104 in the storage unit 111. In this embodiment, the registration processing unit 108 registers only the driving route 80 selected by the user as the driving route 80 for automatic driving. For example, when the user presses the registration button 141a displayed together with the overhead route image 62, the overhead route image 62 registers the driving route 80 corresponding to the overhead route image 62 as the driving route 80 for automatic driving. Also, when the user presses the redo button 141b displayed together with the overhead route image 62, the overhead route image 62 deletes the driving route 80 corresponding to the overhead route image 62.

[0075] The estimation unit 109 estimates the position and orientation of the vehicle 1 based on the surrounding image when the vehicle 1 is automatically driven based on a driving route 80 by a vehicle control unit 110 described later.

[0076] For example, the estimation unit 109 estimates the position of the vehicle 1 by comparing feature points of a surrounding image captured during teacher driving with feature points of a current surrounding image. Note that the method of estimating the position and orientation of the vehicle 1 by the estimation unit 109 is not limited to this example.

[0077] The vehicle control unit 110 moves the vehicle 1 to the parking space 910 by automatic driving based on the driving route 80. The vehicle control unit 110 controls the steering, braking, and acceleration / deceleration of the vehicle 1 to cause the vehicle 1 to automatically drive along the registered driving route 80. This type of driving control method is also called reproducing the driving route 80. The automatic driving process by the vehicle control unit 110 is an example of a driving control step.

[0078] Furthermore, during autonomous driving, the vehicle control unit 110 may cause the vehicle 1 to perform an operation different from the teacher driving based on the vehicle information of the vehicle 1 and the environmental information acquired by the acquisition unit 102. As described above, the environmental information includes the presence or absence of the surroundings of the vehicle 1 and the distance between the vehicle 1 and the obstacles around the vehicle 1. For example, when an obstacle such as another vehicle approaches during autonomous driving, the vehicle control unit 110 executes control such as stopping the vehicle 1.

[0079] During automatic driving by the vehicle control unit 110, the driver may be seated in the driver's seat 130a of the vehicle 1 or may be alight from the vehicle 1.

[0080] Furthermore, when the vehicle 1 deviates from the driving route 80 during automatic driving, the vehicle control unit 110 may use feedback control to move the vehicle 1 back to the driving route 80. For example, the vehicle control unit 110 identifies a difference between the position of the vehicle 1 estimated by the estimation unit 109 and the driving route 80, and drives the vehicle 1 so as to reduce the difference.

[0081] In addition, while FIG. 5 illustrates the parking assistance device 100 as including the vehicle control unit 110, the vehicle control unit 110 may be realized by an ECU different from the parking assistance device 100.

[0082] Next, a flow of the parking assistance process executed by the parking assistance device 100 of this embodiment configured as above will be described.

[0083] FIG. 11 is a flowchart showing an example of the flow of a driving route recording process executed by the parking assistance device 100 according to the first embodiment.

[0084] First, the reception unit 101 judges whether or not a teacher run start operation by the user has been received (S101). If a teacher run start operation by the user has not been received (S101 "No"), the reception unit 101 waits while repeating the process of S101.

[0085] When the reception unit 101 receives an instruction to start the teacher driving by the user (S101 “Yes”), the process of recording the teacher driving starts. After the driver performs the instruction to start the teacher driving, the driver manually drives the vehicle 1 to move it to the parking space 910.

[0086] While the driver is driving as a teacher, the acquisition unit 102 acquires surrounding images from the multiple image capture devices 16a to 16c. The acquisition unit 102 also acquires the speed, steering angle, and braking operation of the vehicle 1 from various sensors of the vehicle 1 or from other ECUs. The acquisition unit 102 also acquires the distance measurement results of the distance between the vehicle 1 and surrounding obstacles from the wave transmitting and receiving unit 15 (S102).

[0087] The extraction unit 103 extracts feature points from the acquired multiple peripheral images (S103).

[0088] Then, the route recording unit 104 estimates the position of the vehicle 1 during the teacher drive based on the changes in feature points extracted from multiple surrounding images, as well as the speed, steering angle, braking operation, and distance measurement results, and records the time-series changes in the position as the driving route 80 (S104).

[0089] Then, the reception unit 101 judges whether or not an operation to end the teacher run has been received (S105). If an operation to end the teacher run by the user has not been received (S105 "No"), the reception unit 101 repeats the processes of S102 to S105.

[0090] When the reception unit 101 receives an operation to end the supervised driving by the user (S105 "Yes"), the route image generation unit 105 generates a driving route image 801 based on the driving route 80 recorded by the route recording unit 104 (S106).

[0091] Furthermore, the route image generating unit 105 generates an outline image 820 representing the movement trajectory of the outline of the vehicle body 12 of the vehicle 1 during the teacher running (S107).

[0092] Further, the extended overhead image generating unit 106 generates an extended overhead image 60 based on a plurality of peripheral images captured in time series by the plurality of image capturing devices 16a to 16c (S108).

[0093] Next, the display control unit 107 causes the display device 120 to display the overhead route image 62 in which the driving route image 801 and the contour image 820 are superimposed on the extended overhead image 60 (S109). In addition, the display control unit 107 causes the display device 120 to display a registration button 141a and a redo button 141b as shown in FIG.

[0094] Then, when the reception unit 101 receives an operation to redo the supervised driving by the user (S110 "Redo"), the process returns to S101. When the reception unit 101 receives an operation to register the supervised driving by the user (S110 "Register"), the registration processing unit 108 registers the driving route 80 in the storage unit 111 as the driving route 80 for automatic driving (S111). Here, the process of this flowchart ends.

[0095] In this manner, in the parking assistance method executed by the parking assistance device 100 of this embodiment, captured images of the surroundings of the vehicle 1 during teacher driving are acquired from the imaging device 16 provided in the vehicle 1 in time series with the movement of the vehicle 1, and the travel route 80 of the vehicle 1 during teacher driving is recorded. In this parking assistance method, an extended overhead image 60 is generated based on the captured images, which is an overhead view of the display target range including the travel route 80 and the surroundings of the travel route 80, and the generated extended overhead image 60 is displayed on the display device 120. Therefore, according to the parking assistance method and parking assistance device 100 of this embodiment, the user can easily grasp the positional relationship between the travel route 80 of the vehicle 1 based on teacher driving and the objects and road 40 around the vehicle 1. In addition, in the parking assistance device 100 of this embodiment, an overhead route image 62 in which a driving route image 801 is superimposed on an extended overhead image 60 is displayed on the display device 120, making it even easier for the user to understand the positional relationship between the driving route 80 of the vehicle 1 based on the teacher driving and the objects around the vehicle 1 and the road 40.

[0096] As a comparative example, in a normal bird's-eye view image, the current surroundings of the vehicle are depicted from an upward perspective, but the entire driving route and the area expanded to the surroundings are not depicted. Therefore, although it is suitable for the driver to check in real time, it may be insufficient to check the entire driving route. In contrast, in the parking assistance method executed by the parking assistance device 100 of this embodiment, the user can objectively grasp the entire driving route 80 during teacher driving and the surrounding conditions.

[0097] In addition, in the parking assistance method executed by the parking assistance device 100 of this embodiment, an outer contour image 820 showing the movement trajectory of the outer contour of the body 12 of the vehicle 1 during teacher driving is displayed superimposed on the extended overhead image 60. Therefore, according to the parking assistance method executed by the parking assistance device 100 of this embodiment, the driver can easily grasp the positional relationship and distance between the outer contour of the body 12 of the vehicle 1 and surrounding obstacles, etc. For example, even if the vehicle 1 reaches the parking space without contacting surrounding obstacles during teacher driving, the outer contour of the body 12 may be close to the surrounding obstacles, etc. The user can check the distance between the outer contour of the body 12 and the surrounding obstacles, etc. from an upper viewpoint by visually checking the outer contour image 820 on the extended overhead image 60.

[0098] For example, in the example shown in Fig. 10, the contour image 820 of the vehicle 1 during teacher driving is also depicted in the area corresponding to the corner space 722 shown in Fig. 4. Therefore, it is clear that the body 12 of the vehicle 1 passes through the corner space 722, although the center of the vehicle 1 does not pass through the corner space 722 during teacher driving. For example, it is assumed that there is nothing placed in the corner space 722 during teacher driving, and the corner space 722 may be used as a bicycle parking area. In this case, the user can avoid adopting the driving route 80 that passes through the corner space 722 for automatic driving by checking the movement trajectory of the body 12 of the vehicle 1 on the contour image 820 on the overhead route image 62.

[0099] Moreover, in the parking assistance method executed by the parking assistance device 100 of this embodiment, when an operation to start and end the recording of the teacher driving by the user is accepted, the overhead route image 62 is displayed. Therefore, according to the parking assistance method executed by the parking assistance device 100 of this embodiment, after the teacher driving is completed, the user can immediately check the positional relationship between the vehicle 1 and surrounding obstacles and roads during the teacher driving from the overhead route image 62. Furthermore, by making such confirmation easy, it is possible to support the user's decision-making as to whether to register a driving route based on the teacher driving or to redo the teacher driving after the teacher driving is completed.

[0100] Furthermore, in the parking assistance method executed by the parking assistance device 100 of this embodiment, the bird's-eye view route image 62 is displayed on the display device 120 at an angle such that the forward direction of the vehicle 1 at the end of the teacher driving faces upward on the screen of the display device 120. With this configuration, the direction in which the driver actually views the scenery outside the vehicle 1 at the end of the teacher driving matches the orientation of the bird's-eye view route image 62 displayed on the display device 120. This allows the driver to easily grasp the relationship between the driving route 80 and the actual surrounding environment.

[0101] Second embodiment In the first embodiment described above, the parking assistance device 100 superimposedly displayed the driving route image 801 and the contour image 820 on the extended overhead image 60. In the second embodiment, the parking assistance device 100 further changes the display mode of the contour image 820 according to the distance to the obstacles around the vehicle 1.

[0102] The vehicle 1 of this embodiment has the same configuration as that of the first embodiment described with reference to Figures 1 and 2. The hardware configuration of the parking assistance device 100 of this embodiment is the same as that of the first embodiment described with reference to Figure 3.

[0103] Similar to the first embodiment, the parking assistance device 100 of this embodiment includes a reception unit 101, an acquisition unit 102, an extraction unit 103, a route recording unit 104, a route image generating unit 105, an extended overhead image generating unit 106, a display control unit 107, a registration processing unit 108, an estimation unit 109, a vehicle control unit 110, and a storage unit 111. The reception unit 101, the acquisition unit 102, the extraction unit 103, the route recording unit 104, the extended overhead image generating unit 106, the display control unit 107, the registration processing unit 108, the estimation unit 109, the vehicle control unit 110, and the storage unit 111 have the same functions as those of the first embodiment.

[0104] The route image generating unit 105 of this embodiment has the same functions as those of the first embodiment, and generates a state of a portion of the contour image where the distance between the vehicle 1 and objects around the vehicle 1 during teacher driving is equal to or less than a threshold, in a state different from a portion where the distance is greater than the threshold. The value of the threshold for the distance between the vehicle 1 and objects around the vehicle 1 is not particularly limited, but may be, for example, 50 cm. The threshold may be a fixed value set in advance, or may be changeable by the user.

[0105] Fig. 12 is a diagram showing an example of an overhead route image 62a according to the second embodiment. In the example shown in Fig. 12, at a turning point of the vehicle 1 on the travel route 80, the distance between the vehicle 1 and surrounding obstacles is equal to or less than a threshold value.

[0106] The route image generating unit 105 generates an outline image 820a by, for example, using a solid line where the distance between the vehicle 1 and an object around the vehicle 1 is greater than a threshold, and using a double line where the distance between the vehicle 1 and an object around the vehicle 1 is equal to or less than the threshold. In the example shown in Fig. 12, the route image generating unit 105 uses a double line for an end image 822a corresponding to a location where the distance between the vehicle 1 and a surrounding obstacle is equal to or less than the threshold, and uses a solid line for a vehicle width image 821 corresponding to a location where the distance is greater than the threshold.

[0107] The display control unit 107 of this embodiment has the same functions as those of the first embodiment, and displays the end image 822a and the vehicle width image 821 generated by the route image generation unit 105 in a superimposed manner on the extended overhead image 60, thereby displaying parts of the contour image 820 where the distance to surrounding obstacles is less than a threshold value in a manner different from parts where the distance is greater than the threshold value.

[0108] The display mode is not limited to the example shown in FIG. 12, and the route image generating unit 105 may change the color of areas where the distance between the vehicle 1 and objects around the vehicle 1 is below a threshold value to a different color from other areas, or may add a warning mark or message.

[0109] In this way, in the parking assistance method executed by the parking assistance device 100 of this embodiment, the portions of the contour image 820 where the distance to the surrounding obstacles is equal to or less than the threshold are displayed in a different manner from the portions where the distance is greater than the threshold. Therefore, according to the parking assistance method executed by the parking assistance device 100 of this embodiment, by allowing the user to understand the portions where the distance between the vehicle 1 and the surrounding obstacles is close during teacher driving, it is possible to support the user's decision-making as to whether to register a driving route based on the teacher driving after the teacher driving ends, or to redo the teacher driving.

[0110] (Third embodiment) In the third embodiment, the parking assistance device 100 further displays a vehicle image representing the vehicle 1 on the overhead route image.

[0111] The vehicle 1 of this embodiment has the same configuration as that of the first embodiment described with reference to Figures 1 and 2. The hardware configuration of the parking assistance device 100 of this embodiment is the same as that of the first embodiment described with reference to Figure 3.

[0112] Similar to the first embodiment, the parking assistance device 100 of this embodiment includes a reception unit 101, an acquisition unit 102, an extraction unit 103, a route recording unit 104, a route image generating unit 105, an extended overhead image generating unit 106, a display control unit 107, a registration processing unit 108, an estimation unit 109, a vehicle control unit 110, and a storage unit 111. The reception unit 101, the acquisition unit 102, the extraction unit 103, the route recording unit 104, the route image generating unit 105, the extended overhead image generating unit 106, the registration processing unit 108, the estimation unit 109, the vehicle control unit 110, and the storage unit 111 have the same functions as those of the first embodiment.

[0113] The display control unit 107 of this embodiment has the same functions as those of the first embodiment, and displays a vehicle image representing the vehicle 1 at the position of the vehicle 1 at the end of the teacher run on the overhead route image. The vehicle image is, for example, a CG (Computer Graphics) image that imitates the shape of the vehicle 1. The vehicle image is, for example, stored in advance in the storage unit 111.

[0114] FIG. 13 is a diagram showing an example of an overhead route image 62b according to the third embodiment.

[0115] 13, the display control unit 107 of this embodiment superimposes and displays a vehicle image 83 on the extended overhead image 60 at a position corresponding to the parking space 910. The orientation of the vehicle image 83 is assumed to be the orientation of the vehicle 1 at the end of the teacher driving.

[0116] In this embodiment, the position of vehicle 1 at the end of the teacher driving is parking space 910. For example, when the driver checks the overhead route image 62b while remaining in vehicle 1 after the teacher driving is finished, the position of vehicle 1 in which the driver is riding is displayed on the overhead route image 62b, so that the driver can easily grasp the positional relationship between the position of vehicle 1 and driving route 80.

[0117] (Fourth embodiment) In the first embodiment described above, the parking assistance device 100 accepted a user's operation to register or delete the driving route 80 obtained in each teacher driving session. In this fourth embodiment, after the completion of multiple teacher driving sessions, the parking assistance device 100 registers the driving route 80 selected by the user from among the multiple driving routes 80 obtained in the multiple teacher driving sessions.

[0118] The vehicle 1 of this embodiment has the same configuration as the first embodiment described with reference to Figures 1 and 2. The hardware configuration of the parking assistance device 100 of this embodiment is the same as that of the first embodiment described with reference to Figure 3.

[0119] Similar to the first embodiment, the parking assistance device 100 of this embodiment includes a reception unit 101, an acquisition unit 102, an extraction unit 103, a route recording unit 104, a route image generating unit 105, an extended overhead image generating unit 106, a display control unit 107, a registration processing unit 108, an estimation unit 109, a vehicle control unit 110, and a storage unit 111. The acquisition unit 102, the extraction unit 103, the route recording unit 104, the route image generating unit 105, the extended overhead image generating unit 106, the estimation unit 109, the vehicle control unit 110, and the storage unit 111 have the same functions as those of the first embodiment.

[0120] The display control unit 107 of this embodiment has the same functions as those of the first embodiment, and after performing multiple supervised runs, displays multiple bird's-eye view route images based on each of the multiple supervised runs on the display device 120. This display process is an example of a second display step in this embodiment.

[0121] Fig. 14 is a diagram showing an example of a driving route selection screen 121 according to the fourth embodiment. The display control unit 107 causes the display device 120 to display the driving route selection screen 121 including a plurality of overhead route images 62c to 62f. The overhead route images 62c to 62f correspond to the first to fourth teacher runs, respectively. Note that the number of teacher runs and the overhead route images 62 shown in Fig. 14 is an example, and is not limited to four.

[0122] Each of the overhead route images 62c to 62f is an image in which driving route images 801c to 801f and contour images 820c to 820f are superimposed on the extended overhead images 60c to 60f. The contour images 820c to 820f include vehicle width images 821c to 820f and edge images 822c to 822f.

[0123] Furthermore, the display control unit 107 displays selection buttons 143a to 143d that allow the user to select each of the overhead route images 62c to 62f, and a registration button 142a, on the travel route selection screen 121. Each of the selection buttons 143a to 143d corresponds one-to-one to each of the overhead route images 62c to 62f.

[0124] The receiving unit 101 of this embodiment has the same functions as those of the first embodiment, and receives a user's operation of selecting one overhead route image 62 corresponding to the driving route 80 to be registered from among the multiple overhead route images 62c to 62f. This reception process is an example of a selection reception step in this embodiment.

[0125] For example, the user presses one of the selection buttons 143a to 143d and then presses the registration button 142a to select one overhead route image 62 corresponding to the driving route 80 to be registered. In the example shown in FIG. 14, the user selects the selection button 143c corresponding to the third supervised driving. When the user presses the registration button 142a in this state, the reception unit 101 receives the user's operation to register the driving route 80 obtained in the third supervised driving as a driving route 80 for automatic driving.

[0126] The registration processing unit 108 of this embodiment has the same functions as those of the first embodiment, and registers a driving route 80 selected by a user in a storage unit 111 as a driving route 80 for automatic driving.

[0127] FIG. 15 is a flowchart showing an example of the flow of a driving route recording process executed by the parking assistance device 100 according to the fourth embodiment.

[0128] The process from determining whether or not an operation to start a teacher run in S101 has been accepted to displaying an overhead route image 62 for each teacher run in S109 is the same as that in the first embodiment.

[0129] Then, if the reception unit 101 receives an operation to start the teacher driving again by the user after S109 (S201 "Start teacher driving again"), the process returns to S101. In this case, the driver causes the vehicle 1 to leave the parking space 910 and move the vehicle 1 to the starting position 900.

[0130] In addition, when the reception unit 101 receives an operation by the user to start selecting a driving route 80 (S201 “Start selecting a driving route”), the display control unit 107 causes the display device 120 to display a driving route selection screen 121 (S202).

[0131] Then, the reception unit 101 determines whether or not an operation of selecting the travel route 80 to be registered by the user has been received (S203). If the reception unit 101 has not received an operation of selecting the travel route 80 to be registered by the user (S203 “No”), the display control unit 107 continues to display the travel route selection screen 121.

[0132] Furthermore, when the reception unit 101 receives an operation by the user to select the driving route 80 to be registered, the registration processing unit 108 registers the driving route 80 selected by the user in the storage unit 111 as the driving route 80 for automatic driving (S204). Here, the processing of this flowchart ends.

[0133] In this way, in the parking assistance method executed by the parking assistance device 100 of this embodiment, after execution of multiple teacher-runs, multiple overhead route images 62c-62f based on each of the multiple teacher-runs are displayed on the display device 120, and a user operation is accepted to select the overhead route image 62 corresponding to the travel route 80 to be registered from the multiple overhead route images 62c-62f. Therefore, according to the parking assistance method executed by the parking assistance device 100 of this embodiment, the user can compare the multiple travel routes 80 obtained by multiple teacher-runs and select the desired travel route 80 to be registered.

[0134] Fifth embodiment In the above-mentioned first to fourth embodiments, the display of the overhead route images 62, 62a to 62f after teacher driving has been described, but in this fifth embodiment, the display of the overhead route image during automatic driving will be described.

[0135] The vehicle 1 of this embodiment has the same configuration as that of the first embodiment described with reference to Figures 1 and 2. The hardware configuration of the parking assistance device 100 of this embodiment is the same as that of the first embodiment described with reference to Figure 3.

[0136] Similar to the first embodiment, the parking assistance device 100 of this embodiment includes a reception unit 101, an acquisition unit 102, an extraction unit 103, a route recording unit 104, a route image generating unit 105, an extended overhead image generating unit 106, a display control unit 107, a registration processing unit 108, an estimation unit 109, a vehicle control unit 110, and a storage unit 111. The reception unit 101, the route recording unit 104, the route image generating unit 105, the registration processing unit 108, the estimation unit 109, the vehicle control unit 110, and the storage unit 111 have the same functions as those of the first embodiment.

[0137] The acquisition unit 102 of this embodiment has the same functions as those of the first embodiment, and acquires captured images of the surroundings of the vehicle 1 during automatic driving in time series with the movement of the vehicle 1, that is, surrounding images, from the multiple image capturing devices 16a to 16c. The acquisition process during automatic driving is an example of a second acquisition step in this embodiment.

[0138] The extraction unit 103 of this embodiment has the same functions as those of the first embodiment, and extracts feature points from surrounding images acquired during autonomous driving.

[0139] The extended overhead image generating unit 106 of this embodiment has the same functions as those of the first embodiment, and generates an overhead image 61 from a surrounding image acquired during autonomous driving.

[0140] Further, the extended overhead image generating unit 106 updates the extended overhead image 60 as the vehicle 1 travels during automatic driving by synthesizing the extended overhead image 60 during teacher driving stored in the storage unit 111 with the overhead image 61 based on the surrounding image captured during automatic driving. This synthesis process is an example of a synthesis step.

[0141] For example, the extended overhead image generating unit 106 specifies an image range of the extended overhead image 60 during teacher driving that corresponds to the overhead image 61 based on the peripheral image captured during automatic driving based on the feature points extracted by the extracting unit 103, and replaces the specified image range with the overhead image 61. By repeating such processing, the range of the extended overhead image 60 during teacher driving that is captured by the imaging device 16 along the progress of the vehicle 1 is sequentially replaced. Note that the method of synthesizing the extended overhead image 60 during teacher driving and the overhead image 61 based on the peripheral image captured during automatic driving is not limited to this.

[0142] In this embodiment, the extended overhead image 60 during teacher driving is called the first extended overhead image 60, and the image obtained by combining the first extended overhead image 60 with an overhead image 61 based on a surrounding image captured during automatic driving is called the second extended overhead image to distinguish between the two.

[0143] The display control unit 107 of this embodiment has the same functions as those of the first embodiment, and causes the display device 120 to display the second extended overhead image generated by the extended overhead image generating unit 106 during autonomous driving. This display process is an example of a third display step.

[0144] Fig. 16 is a diagram showing an example of an overhead route image 62g according to the fifth embodiment. In the second extended overhead image 63 shown in Fig. 16, the range in which the peripheral image is captured by the imaging device 16 of the vehicle 1 located near the start position 900 of the travel route 80 is replaced with the overhead image 61 generated during automatic traveling. In addition, in the second extended overhead image 63, the range in which the peripheral image is not yet captured by the imaging device 16 is displayed as it is as the first extended overhead image 60.

[0145] When the vehicle 1 is being driven automatically by the vehicle control unit 110, the display control unit 107 displays an overhead route image 62g in which the driving route image 802 and the vehicle image 83 are superimposed on the second extended overhead image 63, as shown in FIG. 16.

[0146] More specifically, the display control unit 107 displays the vehicle image 83 at a location on the second extended overhead image 63 that corresponds to the current position of the vehicle 1. The display control unit 107 moves the vehicle image 83 on the second extended overhead image 63 in accordance with the movement of the vehicle 1 due to autonomous driving.

[0147] 16, the display control unit 107 displays, in the driving route image 802, the range where the vehicle 1 has already traveled as a solid line, and the range where the vehicle 1 has not yet traveled as a dashed line. This display allows the user to know where the vehicle 1 is on the driving route 80.

[0148] 16 is an example, and the display mode does not need to be changed depending on whether the vehicle 1 is traveling or not. In addition, the display control unit 107 does not need to display the driving route image 802 during automatic driving.

[0149] 16, the display control unit 107 displays the overhead route image 62g so that the current forward direction of the vehicle 1 faces the Y direction of the display device 120. In this case, when the direction of the vehicle 1 changes, the orientation of the overhead route image 62g also changes.

[0150] The display direction is an example and may be changeable. For example, the display control unit 107 may have a first display mode in which the bird's-eye view route image 62g is displayed so that the current forward direction of the vehicle 1 faces the Y direction of the display device 120, and a second display mode in which the bird's-eye view route image 62g is displayed so that the forward direction of the vehicle 1 at the end point of the travel route 80 faces the Y direction of the display device 120. In the second display mode, the orientation of the bird's-eye view route image 62g is fixed regardless of changes in the orientation of the vehicle 1 during automatic travel.

[0151] For example, the reception unit 101 may receive an operation of selecting the first display mode or the second display mode by the user. The display control unit 107 displays the overhead route image 62g in the display mode selected by the user.

[0152] In this manner, in the parking assistance method executed by the parking assistance device 100 of this embodiment, the second extended overhead image 63 is generated by combining the overhead image 61 generated from the surrounding image captured during the automatic traveling of the vehicle 1 based on the traveling route 80 with the extended overhead image 60 during the teacher traveling. Then, in this parking assistance method, the generated second extended overhead image 63 is displayed on the display device 120. Therefore, according to the parking assistance method executed by the parking assistance device 100 of this embodiment, the second extended overhead image 63 reflecting the current surrounding conditions of the vehicle 1 can be presented to the driver during the automatic traveling of the vehicle 1 based on the traveling route 80.

[0153] For example, if an obstacle or the like that was not present during teacher driving is present around the vehicle 1, the parking assistance method executed by the parking assistance device 100 of this embodiment can depict the obstacle on the second extended overhead image 63.

[0154] (Variation 1) In each of the above-described embodiments, the display device 120 and the operation unit are mounted on the vehicle 1, but may be provided outside the vehicle 1. For example, the display device 120 and the operation unit may be a mobile terminal such as a tablet terminal or a smartphone capable of wireless or wired communication with the parking assistance device 100. Note that it is sufficient that the tablet terminal or the like is not directly connected to the parking assistance device 100 of the vehicle 1 and data and control signals can be transmitted and received via a server device or a cloud environment.

[0155] When the display device 120 and the operation unit are a tablet terminal or the like, the driver may use the tablet terminal or the like inside the vehicle 1, or may use the tablet terminal or the like after getting out of the vehicle 1.

[0156] Alternatively, in the case where the vehicle 1 is equipped with a head-up display, the head-up display may be an example of a display unit. For example, the windshield 180 of the vehicle 1 may become a head-up display capable of displaying an image by projecting an image onto it using a projection device (not shown).

[0157] (Variation 2) Further, the display timing at which the display control unit 107 of the parking assistance device 100 displays the overhead route images 62, 62a to 62g is not limited to the examples given in the above-mentioned embodiments.

[0158] For example, the parking assistance device 100 of this modification has a function of accepting a user's operation to correct the driving route 80. In this case, after the teacher driving is completed, when the reception unit 101 accepts an operation to instruct the user to display a correction screen for the driving route 80, the display control unit 107 causes the display device 120 to display the overhead route images 62, 62a to 62g.

[0159] The user's operation of instructing to display a correction screen for the driving route 80 is, for example, an operation of correcting the driving route on the overhead route images 62, 62a to 62g. The process of accepting this operation is an example of a correction operation accepting step.

[0160] More specifically, the user operates the driving route image 801 included in the overhead route images 62, 62a to 62g with a finger, a mouse, or the like to input a correction operation for the driving route 80. When the parking assistance device 100 receives a correction operation for the driving route 80 by the user, the parking assistance device 100 executes recalculation based on the correction operation and corrects the driving route 80.

[0161] The user may instruct the display of the correction screen for the driving route 80 while the user is in the vehicle 1, or may use a tablet terminal or the like as a display device and an operation unit after the user gets off the vehicle 1. In addition, the user may be able to perform correction operations not only for the driving route 80 immediately after it is recorded in the teacher driving, but also for the driving route 80 that has already been registered as a driving route for automatic driving.

[0162] (Variation 3) In each of the above-described embodiments, the parking assistance device 100 uses the driving trajectory of the vehicle 1 during teacher driving as the driving route 80. However, the parking assistance device 100 may generate the driving route 80 by applying corrections to the speed and driving position of the driving trajectory of the vehicle 1 during teacher driving.

[0163] In addition, in each of the above-described embodiments, the teacher driving is performed in the same manner as the vehicle 1 in automatic driving, in which the vehicle 1 drives from the starting position 900 to the parking space 910, but the teacher driving may be performed in the opposite direction to the automatic driving.

[0164] More specifically, the parking space 910 may be set as the starting point of the teacher driving, and the starting position 900 may be set as the end point of the teacher driving. In this case, the route recording unit 104 generates the driving route 80 for automatic driving by converting the route of the teacher driving in the reverse direction. In other words, the vehicle control unit 110 drives the vehicle 1 from the starting position 900 to the parking space 910 in the opposite direction to the teacher driving. This method is also called playing back the driving route in reverse.

[0165] (Variation 4) Further, some of the functions of the parking assistance device 100 in each of the above-described embodiments may be executed by an information processing device provided outside the vehicle 1. The information processing device provided outside the vehicle 1 is, for example, a mobile terminal such as a smartphone, a PC, a server device, etc. The information processing device may be provided in a cloud environment.

[0166] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope of the invention and its equivalents described in the claims, as well as in the scope and spirit of the invention. [Explanation of symbols]

[0167] 1 vehicle 12 Body 15, 15f, 15r Transmitter / receiver unit 16, 16a to 16d Imaging device 40 road 60,60c~60f Extended overhead image (first extended overhead image) 61 Aerial View 62,62a~62g Aerial view route images 63 Second extended overhead image 70 Home 80 Travel Route 83 Vehicle Images 100 Parking assistance device 101 Reception 102 Acquisition Department 103 Extraction part 104 Route Recording Section 105 Route image generation unit 106 Extended bird's-eye view image generation unit 107 Display control section 108 Registration processing section 109 Estimation section 110 Vehicle control unit 111 Storage section 120 Display device 121 Route selection screen 141 Operation Button 141a Registration button 141b Redo button 143a~143d Selection buttons 180 Windshield 801, 801c~801f, 802 Driving route images 820,820a,820c~820f Outer image 821,821c~820f vehicle width image 822, 822a, 822c~822f End image 900 starting position 910 Parking Space

Claims

1. A parking assistance method for automatically driving a vehicle based on a teacher driving by a driver, comprising: a first acquisition step of acquiring, from an imaging device provided on the vehicle, images of the surroundings of the vehicle during the teacher running in time series with the movement of the vehicle; a route recording step of recording a travel route of the vehicle during the teacher travel; a first extended overhead image generating step of generating a first extended overhead image, which is an overhead view of a display target range including the travel route and a periphery of the travel route, based on the captured image; a first display step of displaying the first extended overhead image on a display unit; a start / end operation receiving step of receiving an operation by a user to start and end recording of the teacher running; Including, In the first display step, an overhead route image in which a travel route image showing the recorded travel route is superimposed on the first extended overhead image is displayed on the display unit; When an operation to end the recording of the teacher driving is accepted, the display of the overhead view route image is executed by the first display step, The bird's-eye view route image is displayed before an operation for registering the teacher's run is accepted; In the first acquisition step, information regarding a distance between the vehicle and objects around the vehicle during the teacher running is further acquired, In the first display step, a contour image showing a movement trajectory of the contour of the vehicle body during the teacher driving is superimposed on the first extended overhead image and a portion of the contour image where the distance is equal to or less than a threshold is displayed on the overhead route image in a manner different from a portion where the distance is greater than the threshold. Parking assistance methods.

2. the display target range is a range including a start position to an end position of a travel route in the teacher travel, The parking assistance method according to claim 1 .

3. In the first display step, a vehicle image representing the vehicle is displayed on the overhead route image at a position of the vehicle at the end of the teacher running.

3. A parking assistance method according to claim 1 or 2.

4. In the first display step, the bird's-eye view route image is displayed on the display unit at an angle such that a forward direction of the vehicle at the end of the teacher running faces upward on a screen of the display unit. The parking assistance method according to claim 1 .

5. A correction operation receiving step of receiving a user's operation to correct the driving route on the overhead route image, A parking assistance method according to any one of claims 1 to 4.

6. a second display step of displaying, on the display unit, a plurality of overhead route images based on each of the plurality of teacher-runs after the plurality of teacher-runs have been executed; and a selection receiving step of receiving a user's operation of selecting an overhead route image corresponding to a travel route to be registered from among the plurality of overhead route images. A parking assistance method according to any one of claims 1 to 5.

7. a driving control step of moving the vehicle to a parking position by automatic driving based on the driving route; a second acquisition step of acquiring from the imaging device captured images of the surroundings of the vehicle during the autonomous driving in time series together with the movement of the vehicle; an overhead image generating step of generating an overhead image from the captured image acquired in the second acquisition step; a synthesis step of generating a second extended overhead image by synthesizing the overhead image with the first extended overhead image generated in the first extended overhead image generation step; and a third display step of displaying the second extended overhead image on the display unit. A parking assistance method according to any one of claims 1 to 6.

8. A parking assistance method for automatically driving a vehicle based on a teacher driving by a driver, comprising: acquiring, from an imaging device provided on the vehicle, captured images of the surroundings of the vehicle during the teacher running in time series with the movement of the vehicle; Recording a travel route of the vehicle during the teacher travel; generating a first extended overhead image in which a display target range including the travel route and a periphery of the travel route is viewed from above; Displaying the first extended overhead image on a display unit; accepting an operation by a user to start and end recording of the teacher driving; Including, displaying, on the display unit, an overhead route image in which a travel route image showing the recorded travel route is superimposed on the first extended overhead image; When an operation to end the recording of the teacher's driving is accepted, the overhead view route image is displayed; The bird's-eye view route image is displayed before an operation for registering the teacher's run is accepted; acquiring information regarding a distance between the vehicle and objects around the vehicle during the teacher driving; an outer contour image showing a movement trajectory of an outer contour of the vehicle body during the teacher driving is displayed superimposed on the first extended overhead image, and a portion of the outer contour image where the distance is equal to or less than a threshold is displayed on the overhead route image in a manner different from a portion where the distance is greater than the threshold; Parking assistance methods.

9. A parking assistance device that performs parking assistance for automatically driving a vehicle based on a teacher driving by a driver, an acquisition unit that acquires images of the surroundings of the vehicle during the teacher running from an imaging device provided in the vehicle in time series with the movement of the vehicle; a route recording unit that records a travel route of the vehicle during the teacher travel; an extended overhead image generating unit that generates a first extended overhead image based on the captured image, the first extended overhead image being a display target range including the travel route and a periphery of the travel route viewed from above; a display control unit that causes the first extended overhead image to be displayed on a display unit; A reception unit that receives an operation by a user to start and end recording of the teacher running; Equipped with the display control unit causes the display unit to display an overhead view route image in which a travel route image showing the recorded travel route is superimposed on the first extended overhead view image, when an operation to end recording of the teacher-running is accepted before an operation to register the teacher-running is accepted, The acquisition unit further acquires information regarding a distance between the vehicle and objects around the vehicle during the teacher traveling, The display control unit further causes an outer contour image representing a movement trajectory of an outer contour of the vehicle body during the teacher driving to be superimposed on the first extended overhead image, and causes a portion of the outer contour image where the distance is equal to or less than the threshold to be displayed on the overhead route image in a manner different from a portion where the distance is greater than the threshold. Parking assistance device.

Images