Parking support method and parking support device

Abstract

To provide a parking support method by which a user can easily grasp a positional relation between a travel route of a vehicle based on a teaching travel and objects and roads around the vehicle, and to provide a parking support device.SOLUTION: A parking support method performs automatic travel of a vehicle on the basis of a teaching travel by a driver, the method comprising: acquiring a captured image obtained by capturing the periphery of the vehicle at the time of teaching travel from an imaging device provided to the vehicle; recording a travel route of the vehicle in the teaching travel; generating a first bird's eye image looking down a first display target range at a predetermined point; generating a second bird' eye image looking down a second display target range including the travel route and the periphery of the travel route; and displaying the second bird's eye image on a display device. The second display target range is broader than the first display target range. The second bird's eye image includes an area displayed in black.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present disclosure relates to a parking assistance method and a parking assistance device.

Background Art

[0002] Conventionally, a parking assistance technology for moving a vehicle by automatic driving when parking the vehicle is known. One such parking assistance is a technology that learns a driving route based on a teacher drive by a driver and performs parking assistance using the learning result. This technology is used, for example, when repeatedly parking at a fixed parking position such as a parking lot at the user's home or workplace.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, it has sometimes been difficult for the user to objectively grasp the positional relationship between the driving route of the vehicle based on the teacher drive and the objects and roads around the vehicle.

[0005] The present disclosure provides a parking assistance method and a parking assistance device that enable the user to easily grasp the positional relationship between the driving route of the vehicle based on the teacher drive and the objects and roads around the vehicle.

Means for Solving the Problems

[0006] The parking assistance method according to the present disclosure is a parking assistance method for automatically driving a vehicle based on a teacher's driving by a driver. An imaging image obtained by imaging the surroundings of the vehicle during the teacher's driving is acquired from an imaging device provided in the vehicle, the driving route of the vehicle during the teacher's driving is recorded, a first overhead image that looks down on a first display target range at a predetermined point is generated, a second overhead image that looks down on a second display target range including the driving route and the periphery of the driving route is generated, and the second overhead image is displayed on a display device. The second display target range is wider than the first display target range. The second overhead image includes a region displayed in black.

Effect of the Invention

[0007] According to the parking assistance method and the parking assistance device according to the present disclosure, the user can easily grasp the positional relationship between the driving route of the vehicle based on the teacher's driving and the surrounding objects and roads of the vehicle.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Embodiments for Carrying Out the Invention

[0009] Hereinafter, embodiments of a parking support method and a parking support device according to the present disclosure will be described with reference to the drawings.

[0010] (First Embodiment) FIG. 1 is a diagram showing an example of a vehicle 1 including a parking support device 100 according to the first embodiment. As shown in FIG. 1, the vehicle 1 includes a vehicle body 12 and two pairs of wheels 13 arranged along a predetermined direction on the vehicle body 12. The two pairs of wheels 13 include a pair of front tires 13f and a pair of rear tires 13r.

[0011] The front tire 13f shown in FIG. 1 is an example of a first wheel in the present embodiment. Also, the rear tire 13r is an example of a second wheel in the present embodiment. Note that the vehicle 1 shown in FIG. 1 includes four wheels 13, but the number of wheels 13 is not limited to this. For example, the vehicle 1 may be a two-wheeled vehicle.

[0012] The vehicle body 12 is coupled to the wheels 13 and is movable by the wheels 13. In this case, the predetermined direction in which the two pairs of wheels 13 are arranged is the traveling direction of the vehicle 1. The vehicle 1 can move forward or backward by switching gears (not shown), etc. Also, the vehicle 1 can turn right or left by steering.

[0013] Also, the vehicle body 12 has a front end portion F which is the end portion on the front tire 13f side, and a rear end portion R which is the end portion on the rear tire 13r side. The vehicle body 12 is substantially rectangular in top view, and the four substantially rectangular corner portions may also be referred to as end portions. Also, although not shown in FIG. 1, the vehicle 1 includes a display device, a speaker, and an operation unit.

[0014] A pair of bumpers 14 are provided near the lower end of the vehicle body 12 at the front and rear end portions F and R of the vehicle body 12. Of the pair of bumpers 14, the front bumper 14f covers the entire front surface and a part of the side surface near the lower end portion of the vehicle body 12. Of the pair of bumpers 14, the rear bumper 14r covers the entire rear surface and a part of the side surface near the lower end portion of the vehicle body 12.

[0015] Transmitting and receiving units 15f and 15r for transmitting and receiving sound waves such as ultrasonic waves are arranged at a predetermined end portion of the vehicle body 12. For example, one or more transmitting and receiving units 15f are arranged on the front bumper 14f, and one or more transmitting and receiving units 15r are arranged on the rear bumper 14r. Hereinafter, when the transmitting and receiving units 15f and 15r are not particularly limited, they are simply referred to as the transmitting and receiving unit 15. Also, the number and position of the transmitting and receiving unit 15 are not limited to the example shown in FIG. 1. For example, the vehicle 1 may be provided with the transmitting and receiving unit 15 on the left and right sides.

[0016] In this embodiment, a sonar using ultrasonic waves will be described as an example of the transmitting and receiving unit 15, but the transmitting and receiving unit 15 may be a radar that transmits and receives electromagnetic waves. Alternatively, the vehicle 1 may be provided with both a sonar and a radar. Also, the transmitting and receiving unit 15 may be simply referred to as a sensor.

[0017] The transmitting / receiving unit 15 detects obstacles around the vehicle 1 based on the results of transmitting and receiving sound waves or electromagnetic waves. Also, the transmitting / receiving unit 15 measures the distance between the obstacles around the vehicle 1 and the vehicle 1 based on the results of transmitting and receiving sound waves or electromagnetic waves.

[0018] In addition, the vehicle 1 includes a first imaging device 16a that images the front of the vehicle 1, a second imaging device 16b that images the rear of the vehicle 1, a third imaging device 16c that images the left side of the vehicle 1, and a fourth imaging device that images the right side of the vehicle 1. The fourth imaging device is not shown in the figure.

[0019] Hereinafter, when the first imaging device 16a, the second imaging device 16b, the third imaging device 16c, and the fourth imaging device are not particularly distinguished, they are simply referred to as the imaging device 16. The position and number of the imaging devices are not limited to the example shown in FIG. 1. For example, the vehicle 1 may include only two imaging devices, namely, the first imaging device 16a and the second imaging device 16b. Alternatively, in addition to the above example, the vehicle 1 may further have other imaging devices.

[0020] The imaging device 16 can image the surrounding video of the vehicle 1, and is, for example, a camera that images a color image. Note that the captured image captured by the imaging device 16 may be a moving image or a still image. Also, the imaging device 16 may be a camera built into the vehicle 1, or may be a camera of a drive recorder retrofitted to the vehicle 1.

[0021] In addition, a parking assistance device 100 is mounted on the vehicle 1. The parking assistance device 100 is an information processing device that can be mounted on the vehicle 1, and is, for example, an ECU (Electronic Control Unit) provided inside the vehicle 1, or an OBU (On Board Unit). Alternatively, the parking assistance device 100 may be an external device installed near the dashboard of the vehicle 1. Note that the parking assistance device 100 may also serve as a car navigation device or the like.

[0022] Next, the configuration near the driver's seat of the vehicle 1 of the present embodiment will be described. FIG. 2 is a diagram showing an example of the configuration near the driver's seat 130a of the vehicle 1 according to the first embodiment.

[0023] As shown in FIG. 2, the vehicle 1 includes a driver's seat 130a and a passenger seat 130b. Further, a windshield 180, a dashboard 190, a steering wheel 140, a display device 120, and operation buttons 141 are provided in front of the driver's seat 130a.

[0024] The display device 120 is a display provided on the dashboard 190 of the vehicle 1. As an example, the display device 120 is located at the center of the dashboard 190 as shown in FIG. 2. The display device 120 is, for example, a liquid crystal display or an organic EL (Electro Luminescence) display. Further, the display device 120 may also serve as a touch panel. The display device 120 is an example of the display unit in the present embodiment.

[0025] Also, the steering wheel 140 is provided in front of the driver's seat 130a and can be operated by the driver. The rotation angle of the steering wheel 140, that is, the steering angle, is electrically or mechanically linked to the change in the direction of the front tire 13f which is the steered wheel. Note that the steered wheel may be the rear tire 13r, or both the front tire 13f and the rear tire 13r may be steered wheels.

[0026] The operation button 141 is a button that can receive operations by the user. In this embodiment, the user is, for example, the driver of the vehicle 1. By receiving the pressing from the driver, the operation button 141 receives an operation such as the start of parking support from the driver. Note that the position of the operation button 141 is not limited to the example shown in FIG. 2, and it may be provided, for example, on the steering wheel 140. The operation button 141 is an example of the operation unit in this embodiment. Further, when the display device 120 also serves as a touch panel, the display device 120 may be an example of the operation unit. Also, an operation terminal such as a tablet terminal, a smartphone, a remote controller, or an electronic key (not shown) that can transmit a signal to the vehicle 1 from outside the vehicle 1 may be used as an example of the operation unit.

[0027] Next, the hardware configuration of the parking support device 100 will be described. FIG. 3 is a diagram showing an example of the hardware configuration of the parking support device 100 according to the first embodiment. As shown in FIG. 3, in the parking support device 100, a CPU (Central Processing Unit) 11A, a ROM (Read Only Memory) 11B, a RAM (Random Access Memory) 11C, an I / F (interface) 11D, an HDD (Hard Disk Drive) 11E, etc. are interconnected by a bus 11F, and it has a hardware configuration using a normal computer.

[0028] The CPU 11A is an arithmetic unit that controls the entire ECU. Note that the CPU 11A is an example of a processor in the parking support device 100 of the present embodiment, and another processor or processing circuit may be provided in place of the CPU 11A. The ROM 11B stores programs and the like for realizing various processes by the CPU 11A. The RAM 11C is, for example, the main storage device of the parking support device 100, and stores data necessary for various processes by the CPU 11A. The I / F 11D is an interface for transmitting and receiving data. The I / F 11D transmits and receives data to and from other devices provided in the vehicle 1, such as the display device 120. Further, the I / F 11D may transmit and receive information to and from other ECUs mounted on the vehicle 1 via a CAN (Controller Area Network) or the like in the vehicle 1, or may communicate with an information processing device outside the vehicle 1 via a network such as the Internet.

[0029] In FIGS. 1 to 3, the display device 120 is illustrated as a device separate from the parking support device 100, but the display device 120 may be included in the parking support device 100.

[0030] The parking support device 100 of the present embodiment learns a driving route based on a teaching drive by the driver, and performs parking support using the learning result. In other words, the parking support method executed by the parking support device 100 is a method of automatically driving the vehicle 1 based on a teaching drive by the driver. Such a parking support method is effective for reducing the driver's parking effort when repeatedly parking at a fixed parking position, such as the driver's home garage, the contract parking position in an apartment building, or the specified parking position in a parking lot at the workplace. Such parking support is called home zone parking.

[0031] FIG. 4 is a diagram for explaining an example of the teacher driving according to the first embodiment. In the example shown in FIG. 4, a parking space 910 located near the home 70 of the driver of the vehicle 1 is the target parking position of the vehicle 1. Also, there are a garden 721 and a corner space 722 near the parking space 910. Among the gardens 721, there are trees 73a and 73b on the side facing the parking space 910.

[0032] In addition, around the parking space 910, there are other houses 71a and 71b, parking spaces 711a to 711c of the other houses 71a and 71b, and other vehicles 30a to 30c, etc. The trees 73a and 73b in the garden 721, the other houses 71a and 71b, the parking spaces 711a to 711c of the other houses 71a and 71b, and the other vehicles 30a to 30c, etc. can be obstacles when the vehicle 1 parks in the parking space 910. Therefore, the driver drives the vehicle 1 while avoiding these during the teacher driving. Note that the objects that can be obstacles to the vehicle 1 are not limited to these.

[0033] In the teacher driving, the driver drives the vehicle 1 manually from the start position 900 on the road 40 to the parking space 910. The parking support device 100 records the driving route 80 from the start position 900 to the parking space 910 in the teacher driving. When the user determines the recorded driving route 80 as the driving route 80 for automatic driving, the parking support device 100 registers the driving route 80 as the driving route 80 for automatic driving. Note that the user may repeat the teacher driving multiple times until the desired driving route 80 is obtained.

[0034] After the registration of the driving route 80 by such teacher driving is completed, the parking support device 100 can park the vehicle 1 in the parking space 910 by automatically driving the vehicle 1 along the registered driving route 80.

[0035] Note that the parking space 910 shown in FIG. 4 and the surrounding environment are examples and are not limited thereto.

[0036] Next, the details of the functions of the parking support device 100 of the present embodiment will be described. FIG. 5 is a block diagram showing an example of the functions provided by the parking support device 100 according to the first embodiment.

[0037] As shown in FIG. 5, the parking support device 100 of the present embodiment includes a reception unit 101, an acquisition unit 102, an extraction unit 103, a route recording unit 104, a route image generation unit 105, an extended overhead image generation unit 106, a display control unit 107, a registration processing unit 108, an estimation unit 109, a vehicle control unit 110, and a storage unit 111.

[0038] The storage unit 111 is configured by, for example, a ROM 11B, a RAM 11C, or an HDD 11E. Although FIG. 5 shows that one storage unit 111 is included in the parking support device 100, a plurality of storage media may function as the storage unit 111.

[0039] The storage unit 111 stores programs and data used in various processes executed by the parking support device 100. For example, the program executed by the parking support device 100 of the present embodiment has a module configuration including the above-described respective functional units (reception unit 101, acquisition unit 102, extraction unit 103, route recording unit 104, route image generation unit 105, extended overhead image generation unit 106, display control unit 107, registration processing unit 108, estimation unit 109, vehicle control unit 110). As actual hardware, the CPU 11A reads the program from the storage unit 111 and executes it, whereby the above-described respective units are loaded onto the RAM 11C, and the reception unit 101, acquisition unit 102, extraction unit 103, route recording unit 104, route image generation unit 105, extended overhead image generation unit 106, display control unit 107, registration processing unit 108, estimation unit 109, and vehicle control unit 110 are generated on the RAM 11C. Note that the processes realized by the respective functional units of the parking support device 100 are also referred to as steps.

[0040] The program executed by the parking support device 100 according to this embodiment is provided by being recorded on a computer-readable recording medium such as a flash memory, a CD-ROM, a flexible disk (FD), a CD-R, or a DVD (Digital Versatile Disk) in an installable format or an executable format file.

[0041] Also, the program executed by the parking support device 100 according to this embodiment may be configured to be stored on a computer connected to a network such as the Internet and downloaded via the network. Further, the program executed by the parking support device 100 according to this embodiment may be configured to be provided or distributed via a network such as the Internet. Also, the program executed by the parking support device 100 according to this embodiment may be configured to be provided by being pre-installed in a ROM 11B or the like.

[0042] The reception unit 101 receives various operations from the user. For example, when the operation button 141 is pressed, the reception unit 101 receives operations for starting and ending the recording of the teaching run by the user. In this embodiment, when collectively referring to the start operation reception step which is the process of receiving the operation for starting the recording of the teaching run and the end operation reception step which is the process of receiving the operation for ending the recording of the teaching run, it is called the start / end operation reception step.

[0043] Also, after the teaching run ends, the reception unit 101 receives an operation of the user for instructing whether to register the travel route 80 of the teaching run as the travel route 80 for automatic driving.

[0044] Also, for example, when the operation button 141 is pressed in a state where the travel route 80 is already registered, the reception unit 101 receives an operation of the user for starting parking support.

[0045] Note that the means by which the reception unit 101 receives the user's operation is not particularly limited. For example, when the display device 120 is a touch panel, the reception unit 101 may receive an operation to start and end the recording of the teaching run or to start the parking support when an image button on the touch panel is pressed.

[0046] The acquisition unit 102 acquires a plurality of captured images obtained by imaging the surroundings of the vehicle 1 in time series as the vehicle 1 moves from the plurality of imaging devices 16a to 16c. Since the captured image is an image that captures the periphery of the vehicle 1, it is referred to as a peripheral image in the present embodiment. This acquisition process is an example of the first acquisition step in the present embodiment. Further, in the first acquisition step, the acquisition unit 102 further acquires information regarding the distance between the vehicle 1 and an object around the vehicle 1 during the teaching run.

[0047] The information regarding the distance between the vehicle 1 and an object around the vehicle 1 is, for example, the presence or absence of an obstacle detected by the transmission / reception wave unit 15 and the length of the distance between the detected obstacle around the vehicle 1 and the vehicle 1. The presence or absence of an obstacle detected by the transmission / reception wave unit 15 and the distance between the detected obstacle around the vehicle 1 and the vehicle 1 measured by the transmission / reception wave unit 15 may be referred to as environmental information regarding the environment around the vehicle 1. Note that the environmental information is not limited to these pieces of information.

[0048] Further, the acquisition unit 102 acquires vehicle information of the vehicle 1 from various sensors of the vehicle 1 or other ECUs. The vehicle information includes, for example, information regarding the speed, steering angle, and braking operation of the vehicle 1. Each piece of information included in the vehicle information of the vehicle 1 is stored in the storage unit 111 in association with the time when each piece of information is detected. Note that the vehicle information of the vehicle 1 may further include the wheel speed or the number of rotations of the wheel 13, the acceleration of the vehicle 1 measured by a gyro sensor, etc.

[0049] The extraction unit 103 extracts feature points from the surrounding images. The method for extracting feature points by the extraction unit 103 is not particularly limited, and known methods may be applied. For example, the extraction unit 103 extracts feature points by a method such as FAST (Features from Accelerated Segment Test) or ORB (Oriented FAST and Rotated BRIEF). Further, when learning the driving route 80, the extraction unit 103 may preferentially record the feature points that satisfy the specified conditions among the extracted feature points. For example, among a plurality of surrounding images that are continuous in time series, the feature points extracted from the images in which the vehicle 1 has moved a longer distance during imaging may be preferentially selected as feature points.

[0050] The route recording unit 104 records the driving route 80 of the vehicle 1 during the teaching drive. The route recording unit 104 estimates the position of the vehicle 1 during the teaching drive based on, for example, the changes in the feature points extracted from a plurality of surrounding images and the vehicle information of the vehicle 1, and specifies the driving route 80 from the temporal change of the position. The recording process of the driving route 80 is an example of the route recording step.

[0051] More specifically, the route recording unit 104 specifies the change in the position of the vehicle 1 based on the temporal change of the feature points extracted from a plurality of surrounding images captured during the teaching drive. Further, the route recording unit 104 may correct the position of the vehicle 1 specified from the feature points based on the acquired vehicle information.

[0052] In the present embodiment, information in which the driving route 80 for automatic driving, the speed, steering angle, braking operation of the vehicle 1 traveling on the driving route 80, and the feature points extracted from a plurality of surrounding images captured as the vehicle 1 moves during the teaching drive are associated with each other in time series is referred to as driving route information. The route recording unit 104 stores the driving route information in the storage unit 111. The driving route information is used by the vehicle control unit 110 during the automatic driving described later. Note that the recording method of the driving route 80 and the definition of the driving route information are not limited to this example.

[0053] In addition, the route recording unit 104 defines the environment around the vehicle 1 as a map based on the feature points extracted from the surrounding images captured during the teacher's driving, and stores the map in the storage unit 111. Note that the recording process of the driving route 80 based on the teacher's driving may also be referred to as a learning process.

[0054] The route image generation unit 105 generates a driving route image representing the driving route 80 recorded by the route recording unit 104.

[0055] FIG. 6 is a diagram showing an example of the driving route image 801 according to the first embodiment. The driving route image 801 is, for example, an image representing the line from the start position to the end position of the driving route 80 as shown in FIG. 6. In the driving route image 801, for example, the driving route 80 is represented based on the movement locus of the center of the vehicle body 12 of the vehicle 1. Also, in the example shown in FIG. 6, the tip of the line that is the driving route image 801 is an arrow indicating the traveling direction of the vehicle 1. Note that the mode of the driving route image 801 is not limited to this.

[0056] In addition, the route image generation unit 105 generates an outline image representing the movement locus of the outline of the vehicle body 12 of the vehicle 1 during the teacher's driving.

[0057] FIG. 7 is a diagram showing an example of the outline image 820 according to the first embodiment. In FIG. 7, the driving route image 801 is also shown together with the outline image 820. The outline image 820 represents, for example, the movement locus of the outline of the vehicle body 12 of the vehicle 1 when the vehicle 1 travels on the driving route 80 as shown in FIG. 7. The route image generation unit 105 estimates the movement locus of the outline of the vehicle body 12 based on, for example, the recorded driving route 80 and the size of the vehicle body 12. Note that the size of the vehicle body 12 is assumed to be stored in the storage unit 111, for example.

[0058] The outline image 820 of the present embodiment includes at least a vehicle width image 821 representing the movement loci of both left and right end portions of the vehicle body 12 of the vehicle 1. Further, when the driving route 80 includes a turn, the outline image includes an end portion image 822 indicating the position of the front end portion or the rear end portion of the vehicle body 12 of the vehicle 1 at the turning point.

[0059] Returning to FIG. 5, the extended overhead image generation unit 106 generates an extended overhead image that looks down on the display target range including the driving route 80 and the surroundings of the driving route 80 from above, based on the surrounding images. The display target range is, for example, a range including from the start position 900 to the end position of the driving route 80 in the teacher driving. The extended overhead image is an example of the first extended overhead image in the present embodiment. This process is an example of the first extended overhead image generation step.

[0060] FIG. 8 is a diagram showing an example of the overhead image 61 according to the first embodiment. The extended overhead image generation unit 106 generates a plurality of overhead images 61 around the vehicle 1 at a plurality of positions on the driving route 80 by converting and synthesizing a plurality of surrounding images captured in time series by the plurality of imaging devices 16a to 16c as seen from a virtual viewpoint. Such image processing of viewpoint conversion and synthesis is an example of the overhead image generation step.

[0061] The virtual viewpoint is located, for example, directly above the vehicle 1. Therefore, the overhead image is also called a top view. A known technique can be adopted for the method of generating the overhead image 61.

[0062] The overhead image 61 shown in FIG. 8 is a composite image generated based on a plurality of surrounding images captured by the plurality of imaging devices 16a to 16c when the vehicle 1 is located in the parking space 910 which is the end point of the driving route 80. Since the imaging ranges of the plurality of imaging devices 16a to 16c change as the vehicle 1 moves, when the vehicle 1 is at different positions on the driving route 80, the overhead image 61 is also different.

[0063] Then, the extended overhead image generation unit 106 synthesizes the plurality of overhead images 61 along the time-series change of the imaging times of the surrounding images corresponding to each overhead image 61, thereby generating an image depicting the surroundings of the vehicle 1 over the entire driving route 80. Since such an image depicts a wider range than the individual overhead images 61, it is referred to as an extended overhead image in the present embodiment. Note that the extended overhead image may simply be referred to as an overhead image.

[0064] FIG. 9 is a diagram showing an example of the extended overhead image 60 according to the first embodiment. The extended overhead image generation unit 106 generates one extended overhead image 60, for example, by connecting a plurality of overhead images 61 based on feature points included in the plurality of overhead images 61. The extended overhead image 60 includes the road surface or ground on which the vehicle 1 travels from the start position 900 to the end position, which is the parking space 910, of the travel route 80, and a background depicting the periphery of the travel route 80.

[0065] In FIGS. 8 and 9, the extended overhead image generation unit 106 has been described as generating one extended overhead image 60 from the plurality of overhead images 61 after generating the plurality of overhead images 61 from the plurality of peripheral images, but the flow of the generation process is not limited to this. For example, the extended overhead image generation unit 106 may generate the extended overhead image 60 from the plurality of peripheral images.

[0066] Returning to FIG. 5, the display control unit 107 superimposes the travel route image 801 on the extended overhead image 60 and causes it to be displayed on the display device 120. Hereinafter, the extended overhead image 60 on which the travel route image 801 is superimposed is referred to as an overhead route image. Note that the display process is an example of the first display step in the present embodiment.

[0067] FIG. 10 is a diagram showing an example of the overhead route image 62 according to the first embodiment. In the overhead route image 62 shown in FIG. 10, the travel route image 801 and the outline image 820 are superimposed on the extended overhead image 60. Note that the outline image 820 is not essential, and at least the travel route image 801 may be superimposed and displayed on the extended overhead image 60.

[0068] In the example shown in FIG. 10, the display control unit 107 displays a rectangular background area 602 as the background of the extended overhead image 60. Note that the extended overhead image 60 may include the background area 602. In FIG. 10, the background area 602 is displayed in black, but this is an example, and the display mode is not particularly limited.

[0069] The display timing of the bird's-eye view route image 62 is, for example, when an operation to end the teaching run is performed by the user. When the reception unit 101 receives an operation to end the recording of the teaching run, the display control unit 107 causes the display device 120 to display the bird's-eye view route image 62.

[0070] For example, in the bird's-eye view route image 62, the road 40, the trees 73a, 73b which are obstacles around the driving route 80, other houses 71a, other parking spaces 711a, 711c, and other vehicles 30a, etc. are depicted together with the driving route image 801 and the outline image 820. Therefore, the user can confirm the positional relationship between the vehicle 1 and the surrounding obstacles and the road during the teaching run.

[0071] In addition, the display control unit 107 causes the display device 120 to display the bird's-eye view route image 62 at an angle such that the front direction of the vehicle 1 at the end of the teaching run faces the vertical direction of the display device 120. In the example shown in FIG. 10, the vertical direction of the display device 120 is the Y direction, and the horizontal direction is the X direction. The vertical direction is also referred to as the upper part of the screen. Note that the front direction of the vehicle 1 means the direction in which the front end of the vehicle 1 faces.

[0072] As shown in FIG. 4, in this embodiment, the vehicle 1 moves forward and then reverses into the parking space 910 during the teaching run. Therefore, the front of the vehicle 1 at the end of the teaching run faces the entrance side of the parking space 910, and the driver is also in a state of facing the entrance side of the parking space 910. That is, by causing the display control unit 107 to display the bird's-eye view route image 62 on the display device 120 such that the front direction of the vehicle 1 at the end of the teaching run faces the Y direction, the direction in which the driver actually views the scenery outside the vehicle 1 at the end of the teaching run and the orientation of the bird's-eye view route image 62 displayed on the display device 120 match. Therefore, the driver can easily grasp the relationship between the driving route 80 and the actual surrounding environment.

[0073] In addition, the display control unit 107 causes the display device 120 to display the registration button 141a and the retry button 141b together with the bird's-eye view route image 62. The registration button 141a is an image button that accepts a user operation for instructing registration of the travel route 80 corresponding to the bird's-eye view route image 62. The retry button 141b is an image button that accepts a user operation for instructing not to register the travel route 80 corresponding to the bird's-eye view route image 62. When the registration button 141a and the retry button 141b are pressed, the reception unit 101 accepts operations corresponding to the respective buttons. Note that the method of accepting operations for registering or deleting the travel route 80 is not limited to this.

[0074] Returning to FIG. 5, the registration processing unit 108 registers the travel route 80 generated by the route recording unit 104 in the storage unit 111. In the present embodiment, the registration processing unit 108 registers only the travel route 80 selected by the user as the travel route 80 for autonomous driving. For example, when the registration button 141a displayed together with the bird's-eye view route image 62 is pressed by the user, the travel route 80 corresponding to the bird's-eye view route image 62 is registered as the travel route 80 for autonomous driving. When the retry button 141b displayed together with the bird's-eye view route image 62 is pressed by the user, the travel route 80 corresponding to the bird's-eye view route image 62 is deleted.

[0075] When the vehicle 1 performs autonomous driving based on the travel route 80 by the vehicle control unit 110 described later, the estimation unit 109 estimates the position and orientation of the vehicle 1 based on the surrounding image.

[0076] For example, the estimation unit 109 estimates the position of the vehicle 1 by comparing the feature points of the surrounding image captured during the teacher driving with the feature points of the current surrounding image. Note that the method of estimating the position and orientation of the vehicle 1 by the estimation unit 109 is not limited to this example.

[0077] The vehicle control unit 110 moves the vehicle 1 to the parking space 910 by means of automatic driving based on the driving route 80. The vehicle control unit 110 automatically drives the vehicle 1 along the registered driving route 80 by controlling the steering, braking, and acceleration / deceleration of the vehicle 1. Such a driving control method is also referred to as reproducing the driving route 80. The process of automatic driving by the vehicle control unit 110 is an example of a driving control step.

[0078] In addition, during automatic driving, the vehicle control unit 110 may cause the vehicle 1 to perform an operation different from the teaching driving based on the vehicle information of the vehicle 1 and the environmental information acquired by the acquisition unit 102. As described above, the environmental information includes the presence or absence around the vehicle 1 and the distance between the obstacles around the vehicle 1 and the vehicle 1. For example, when an obstacle such as another vehicle approaches during automatic driving, the vehicle control unit 110 executes control such as stopping the vehicle 1.

[0079] During automatic driving by the vehicle control unit 110, the driver may be seated in the driver's seat 130a of the vehicle 1, or may have gotten out of the vehicle 1.

[0080] In addition, when the vehicle 1 deviates from the driving route 80 during automatic driving, the vehicle control unit 110 may move the vehicle 1 back to the driving route 80 by means of feedback control. For example, the vehicle control unit 110 identifies the difference between the position of the vehicle 1 estimated by the estimation unit 109 and the driving route 80, and drives the vehicle 1 so as to reduce the difference.

[0081] In addition, in FIG. 5, the parking support device 100 is illustrated as including the vehicle control unit 110, but the vehicle control unit 110 may be realized by another ECU different from the parking support device 100.

[0082] Next, the flow of the parking support process executed by the parking support device 100 of the present embodiment configured as described above will be described.

[0083] FIG. 11 is a flowchart showing an example of the flow of the travel route recording process executed by the parking support device 100 according to the first embodiment.

[0084] First, the reception unit 101 determines whether or not it has received a start operation of the teaching drive by the user (S101). If the reception unit 101 has not received the start operation of the teaching drive by the user (S101 “No”), the reception unit 101 repeats the process of S101 and waits.

[0085] When the reception unit 101 has received the start operation of the teaching drive by the user (S101 “Yes”), the process of recording the teaching drive starts. Further, after the driver performs the start operation of the teaching drive, the driver manually drives the vehicle 1 and moves it to the parking space 910.

[0086] During the teaching drive by the driver, the acquisition unit 102 acquires peripheral images from the plurality of imaging devices 16a to 16c. Further, the acquisition unit 102 acquires the speed, steering angle, and braking operation of the vehicle 1 from various sensors of the vehicle 1 or other ECUs. Further, the acquisition unit 102 acquires the ranging result of the distance between the vehicle 1 and surrounding obstacles from the transmission / reception unit 15 (S102).

[0087] The extraction unit 103 extracts feature points from the acquired plurality of peripheral images (S103).

[0088] Then, based on the changes in the feature points extracted from the plurality of peripheral images, as well as the speed, steering angle, braking operation, and ranging result, the route recording unit 104 estimates the position of the vehicle 1 during the teaching drive, and records the temporal change of the position as the travel route 80 (S104).

[0089] Then, the reception unit 101 determines whether or not it has received an end operation of the teaching drive (S105). If the reception unit 101 has not received the end operation of the teaching drive by the user (S105 “No”), the reception unit 101 repeats the processes of S102 to S105.

[0090] When the reception unit 101 receives an end operation of the teaching run by the user (S105 “Yes”), the route image generation unit 105 generates a route image 801 based on the travel route 80 recorded by the route recording unit 104 (S106).

[0091] Also, the route image generation unit 105 generates an outline image 820 representing the movement locus of the outline of the vehicle body 12 of the vehicle 1 in the teaching run (S107).

[0092] Also, the extended overhead image generation unit 106 generates an extended overhead image 60 based on a plurality of peripheral images captured in time series by the plurality of imaging devices 16a to 16c (S108).

[0093] Next, the display control unit 107 causes the display device 120 to display an overhead route image 62 in which the route image 801 and the outline image 820 are superimposed on the extended overhead image 60 (S109). Also, as shown in FIG. 10, the display control unit 107 causes the display device 120 to display the registration button 141a and the retry button 141b.

[0094] When the reception unit 101 receives a retry operation of the teaching run by the user (S110 “retry”), the process returns to the process of S101. When the reception unit 101 receives a registration operation of the teaching run by the user (S110 “registration”), the registration processing unit 108 registers the travel route 80 in the storage unit 111 as a travel route 80 for automatic driving (S111). Here, the processing of this flowchart ends.

[0095] As described above, in the parking support method executed by the parking support device 100 of the present embodiment, an imaging image obtained by imaging the surroundings of the vehicle 1 in time series as the vehicle 1 moves is acquired from the imaging device 16 provided in the vehicle 1, and the travel route 80 of the vehicle 1 during the teaching run is recorded. In this parking support method, based on the imaging image, an extended bird's-eye view image 60 obtained by looking down on the display target range including the travel route 80 and the surroundings of the travel route 80 from above is generated, and the generated extended bird's-eye view image 60 is displayed on the display device 120. Therefore, according to the parking support method and the parking support device 100 of the present embodiment, the user can easily grasp the positional relationship between the travel route 80 of the vehicle 1 based on the teaching run and the surrounding objects and the road 40 of the vehicle 1. Further, in the parking support device 100 of the present embodiment, by displaying on the display device 120 a bird's-eye view route image 62 in which a travel route image 801 is superimposed on the extended bird's-eye view image 60, it becomes even easier for the user to grasp the positional relationship between the travel route 80 of the vehicle 1 based on the teaching run and the surrounding objects and the road 40 of the vehicle 1.

[0096] As a comparative example, in a normal bird's-eye view image, the state around the current vehicle is depicted from a viewpoint looking down from above, but the range covering the entire travel route and the area enlarged to the surroundings thereof are not depicted. Therefore, although it is suitable for the driver to check in real time, there are cases where it is insufficient to check the entire travel route. On the other hand, in the parking support method executed by the parking support device 100 of the present embodiment, the user can objectively grasp the entire travel route 80 and the surrounding state during the teaching run.

[0097] In addition, in the parking support method executed by the parking support device 100 of the present embodiment, an outline image 820 representing the movement locus of the outline of the vehicle body 12 of the vehicle 1 during the teaching run is superimposed and displayed on the extended bird's-eye view image 60. Therefore, according to the parking support method executed by the parking support device 100 of the present embodiment, the driver can easily grasp the positional relationship and distance between the outline of the vehicle body 12 of the vehicle 1 and surrounding obstacles or the like. For example, even when the vehicle 1 reaches the parking space without contacting surrounding obstacles during the teaching run, the distance between the outline of the vehicle body 12 and surrounding obstacles or the like may be close. The user can confirm the distance between the outline of the vehicle body 12 and surrounding obstacles or the like from the top view point by visually recognizing the outline image 820 on the extended bird's-eye view image 60.

[0098] For example, in the example shown in FIG. 10, the outline image 820 of the vehicle 1 is also drawn in a region corresponding to the corner space 722 shown in FIG. 4 during the teaching run. Therefore, although the center of the vehicle 1 does not pass through the corner space 722 during the teaching run, it becomes clear that the vehicle body 12 of the vehicle 1 passes through the corner space 722. For example, assuming that nothing was placed in the corner space 722 during the teaching run and the corner space 722 may be used as a parking area. In this case, the user can avoid adopting the driving route 80 passing through the corner space 722 for automatic driving by confirming the movement locus of the vehicle body 12 of the vehicle 1 with the outline image 820 on the overhead route image 62.

[0099] Also, in the parking support method executed by the parking support device 100 of the present embodiment, when receiving the operations for starting and ending the recording of the teaching drive by the user, the bird's-eye view route image 62 is displayed. Therefore, according to the parking support method executed by the parking support device 100 of the present embodiment, after the end of the teaching drive, the user can immediately confirm, by means of the bird's-eye view route image 62, the positional relationship between the vehicle 1 and the surrounding obstacles and roads in the teaching drive. Further, by facilitating such confirmation, it is possible to support the user's decision-making as to whether to register the driving route based on the teaching drive or to repeat the teaching drive after the end of the teaching drive.

[0100] Also, in the parking support method executed by the parking support device 100 of the present embodiment, the bird's-eye view route image 62 is caused to be displayed on the display device 120 at an angle such that the forward direction of the vehicle 1 at the end of the teaching drive faces the upper part of the screen of the display device 120. With such a configuration, at the end of the teaching drive, the direction in which the driver actually views the scenery outside the vehicle 1 coincides with the orientation of the bird's-eye view route image 62 displayed on the display device 120. Therefore, the driver can easily grasp the relationship between the driving route 80 and the actual surrounding environment.

[0101] (Second Embodiment) In the above-described first embodiment, the parking support device 100 superimposed and displayed the driving route image 801 and the outline image 820 on the extended bird's-eye view image 60. In this second embodiment, the parking support device 100 further changes the display mode of the outline image 820 according to the distance to the obstacles around the vehicle 1.

[0102] The vehicle 1 of the present embodiment has the same configuration as that of the first embodiment described with reference to FIGS. 1 and 2. Also, the hardware configuration of the parking support device 100 of the present embodiment is the same as that of the first embodiment described with reference to FIG. 3.

[0103] Similar to the first embodiment, the parking support device 100 of this embodiment includes a reception unit 101, an acquisition unit 102, an extraction unit 103, a route recording unit 104, a route image generation unit 105, an extended overhead image generation unit 106, a display control unit 107, a registration processing unit 108, an estimation unit 109, a vehicle control unit 110, and a storage unit 111. The reception unit 101, the acquisition unit 102, the extraction unit 103, the route recording unit 104, the extended overhead image generation unit 106, the display control unit 107, the registration processing unit 108, the estimation unit 109, the vehicle control unit 110, and the storage unit 111 have the same functions as those in the first embodiment.

[0104] In addition to having the same functions as those in the first embodiment, the route image generation unit 105 of this embodiment generates the aspect of the part where the distance between the vehicle 1 and the objects around the vehicle 1 during the teacher driving is less than or equal to the threshold in the outline image in a different aspect from the part where the distance is farther than the threshold. The value of the threshold for the distance between the objects around the vehicle 1 and the vehicle 1 is not particularly limited, and may be, for example, 50 cm or the like. Note that the threshold may be a preset fixed value or may be changeable by the user.

[0105] FIG. 12 is a diagram showing an example of the overhead route image 62a according to the second embodiment. In the example shown in FIG. 12, at the turning position of the vehicle 1 on the travel route 80, the distance between the surrounding obstacles and the vehicle 1 is less than or equal to the threshold.

[0106] For example, the route image generation unit 105 generates an outline image 820a with a solid line at the part where the distance between the objects around the vehicle 1 and the vehicle 1 is farther than the threshold, and a double line at the part where the distance between the objects around the vehicle 1 and the vehicle 1 is less than or equal to the threshold. In the example shown in FIG. 12, the route image generation unit 105 uses a double line for the end image 822a corresponding to the part where the distance between the surrounding obstacles and the vehicle 1 is less than or equal to the threshold, and a solid line for the vehicle width image 821 corresponding to the part where the distance is farther than the threshold.

[0107] Then, the display control unit 107 of the present embodiment has the same functions as those of the first embodiment, and by superimposing and displaying the end image 822a and the vehicle width image 821 generated by the route image generation unit 105 on the extended bird's-eye view image 60, a portion of the outer contour image 820 where the distance to surrounding obstacles is equal to or less than the threshold is displayed in a manner different from that of a portion where the distance is farther than the threshold.

[0108] Note that the display mode is not limited to the example shown in FIG. 12. The route image generation unit 105 may make the color of a portion where the distance between an object around the vehicle 1 and the vehicle 1 is equal to or less than the threshold different from that of other portions, or may add a warning mark or message.

[0109] Thus, in the parking support method executed by the parking support device 100 of the present embodiment, a portion of the outer contour image 820 where the distance to surrounding obstacles is equal to or less than the threshold is displayed in a manner different from that of a portion where the distance is farther than the threshold. Therefore, according to the parking support method executed by the parking support device 100 of the present embodiment, in the teaching run, a user can be made to grasp a portion where the distance between the vehicle 1 and surrounding obstacles is close, so that after the end of the teaching run, it is possible to support the user's decision-making on whether to register a driving route based on the teaching run or to repeat the teaching run.

[0110] (Third Embodiment) In this third embodiment, the parking support device 100 further displays a vehicle image representing the vehicle 1 on the bird's-eye view route image.

[0111] The vehicle 1 of the present embodiment has the same configuration as that of the first embodiment described with reference to FIGS. 1 and 2. Also, the hardware configuration of the parking support device 100 of the present embodiment is the same as that of the first embodiment described with reference to FIG. 3.

[0112] The parking support device 100 of this embodiment includes a reception unit 101, an acquisition unit 102, an extraction unit 103, a route recording unit 104, a route image generation unit 105, an extended overhead image generation unit 106, a display control unit 107, a registration processing unit 108, an estimation unit 109, a vehicle control unit 110, and a storage unit 111, similar to the first embodiment. The reception unit 101, the acquisition unit 102, the extraction unit 103, the route recording unit 104, the route image generation unit 105, the extended overhead image generation unit 106, the registration processing unit 108, the estimation unit 109, the vehicle control unit 110, and the storage unit 111 have the same functions as those in the first embodiment.

[0113] The display control unit 107 of this embodiment, while having the same functions as those in the first embodiment, displays a vehicle image representing the vehicle 1 at the position of the vehicle 1 at the end of the teaching run on the overhead route image. The vehicle image is, for example, a CG (Computer Graphics) image simulating the shape of the vehicle 1. Assume that the vehicle image is stored in advance in the storage unit 111, for example.

[0114] FIG. 13 is a diagram showing an example of an overhead route image 62b according to the third embodiment.

[0115] As shown in FIG. 13, the display control unit 107 of this embodiment superimposes and displays the vehicle image 83 at a position corresponding to the parking space 910 on the extended overhead image 60. Also, assume that the orientation of the vehicle image 83 is the orientation of the vehicle 1 at the end of the teaching run.

[0116] In this embodiment, the position of the vehicle 1 at the end of the teaching run is the parking space 910. For example, when the driver checks the overhead route image 62b while remaining in the vehicle 1 after the end of the teaching run, the position of the vehicle 1 in which the driver is riding is displayed on the overhead route image 62b, so that the driver can easily grasp the positional relationship between the position of the vehicle 1 and the travel route 80.

[0117] (Fourth Embodiment) In the above-described first embodiment, the parking support device 100 accepted a user operation to register or delete the travel route 80 obtained in the teaching run for each teaching run. In this fourth embodiment, the parking support device 100 registers the travel route 80 selected by the user among the plurality of travel routes 80 obtained in the plurality of teaching runs after the completion of the plurality of teaching runs.

[0118] The vehicle 1 of this embodiment has the same configuration as the first embodiment described with reference to FIGS. 1 and 2. Further, the hardware configuration of the parking support device 100 of this embodiment is the same as that of the first embodiment described with reference to FIG. 3.

[0119] The parking support device 100 of this embodiment includes a reception unit 101, an acquisition unit 102, an extraction unit 103, a route recording unit 104, a route image generation unit 105, an extended bird's-eye view image generation unit 106, a display control unit 107, a registration processing unit 108, an estimation unit 109, a vehicle control unit 110, and a storage unit 111, in the same manner as the first embodiment. The acquisition unit 102, the extraction unit 103, the route recording unit 104, the route image generation unit 105, the extended bird's-eye view image generation unit 106, the estimation unit 109, the vehicle control unit 110, and the storage unit 111 have the same functions as those of the first embodiment.

[0120] The display control unit 107 of this embodiment has the same function as that of the first embodiment, and after executing a plurality of teaching runs, causes the display device 120 to display a plurality of bird's-eye view route images based on each of the plurality of teaching runs. This display process is an example of the second display step in this embodiment.

[0121] FIG. 14 is a diagram showing an example of a travel route selection screen 121 according to the fourth embodiment. The display control unit 107 causes the display device 120 to display a travel route selection screen 121 including a plurality of bird's-eye view route images 62c to 62f. Each of the bird's-eye view route images 62c to 62f corresponds to the first to fourth teaching runs, respectively. Note that the number of teaching runs and the bird's-eye view route images 62 shown in FIG. 14 are examples and are not limited to four.

[0122] Each of the overhead path images 62c to 62f is an image in which the travel path images 801c to 801f and the outline images 820c to 820f are superimposed on the extended overhead images 60c to 60f. The outline images 820c to 820f include vehicle width images 821c to 820f and end images 822c to 822f.

[0123] Further, the display control unit 107 causes selection buttons 143a to 143d, which allow the user to select each of the overhead path images 62c to 62f, and a registration button 142a to be displayed on the travel path selection screen 121. Each of the selection buttons 143a to 143d corresponds one-to-one with the overhead path images 62c to 62f, respectively.

[0124] The reception unit 101 of the present embodiment has the same functions as those of the first embodiment, and receives an operation of the user who selects one overhead path image 62 corresponding to the travel path 80 to be registered from among the plurality of overhead path images 62c to 62f. The reception process is an example of the selection reception step in the present embodiment.

[0125] For example, the user presses one of the selection buttons 143a to 143d and then presses the registration button 142a, thereby selecting one overhead path image 62 corresponding to the travel path 80 to be registered. In the example shown in FIG. 14, the user selects the selection button 143c corresponding to the third teaching run. When the user presses the registration button 142a in this state, the reception unit 101 receives an operation of the user to register the travel path 80 obtained in the third teaching run as the travel path 80 for automatic driving.

[0126] The registration processing unit 108 of the present embodiment has the same functions as those of the first embodiment, and registers the travel path 80 selected by the user as the travel path 80 for automatic driving in the storage unit 111.

[0127] FIG. 15 is a flowchart showing an example of the flow of the travel path recording process executed by the parking support device 100 according to the fourth embodiment.

[0128] The process from the determination of whether to accept the start operation of the instructor driving in S101 to the display of the overhead route image 62 for each instructor driving in S109 is the same as that in the first embodiment.

[0129] Then, when the reception unit 101 receives an operation from the user to start the instructor driving again after S109 (S201 "Start instructor driving again"), the process returns to S101. Also, in this case, the driver takes the vehicle 1 out of the parking space 910 and moves the vehicle 1 to the start position 900.

[0130] Also, when the reception unit 101 receives an operation from the user to start selecting the travel route 80 (S201 "Start selection of travel route"), the display control unit 107 causes the travel route selection screen 121 to be displayed on the display device 120 (S202).

[0131] Then, the reception unit 101 determines whether it has received an operation from the user to select the travel route 80 to be registered (S203). If the reception unit 101 has not received an operation from the user to select the travel route 80 to be registered (S203 "No"), the display control unit 107 continues to display the travel route selection screen 121.

[0132] Also, when the reception unit 101 receives an operation from the user to select the travel route 80 to be registered, the registration processing unit 108 registers the travel route 80 selected by the user in the storage unit 111 as the travel route 80 for autonomous driving (S204). Here, the processing of this flowchart ends.

[0133] Thus, in the parking support method executed by the parking support device 100 of the present embodiment, after a plurality of teaching runs are executed, a plurality of bird's-eye view path images 62c to 62f based on each of the plurality of teaching runs are displayed on the display device 120, and an operation of a user who selects a bird's-eye view path image 62 corresponding to the driving path 80 to be registered is received from among the plurality of bird's-eye view path images 62c to 62f. Therefore, according to the parking support method executed by the parking support device 100 of the present embodiment, the user can compare a plurality of driving paths 80 obtained by a plurality of teaching runs and select a desired driving path 80 as a registration target.

[0134] (Fifth Embodiment) In the above-described first to fourth embodiments, the display of the bird's-eye view path images 62, 62a to 62f after the teaching run has been described. In this fifth embodiment, the display of the bird's-eye view path image during the automatic driving will be described.

[0135] The vehicle 1 of the present embodiment has the same configuration as that of the first embodiment described with reference to FIGS. 1 and 2. Further, the hardware configuration of the parking support device 100 of the present embodiment is the same as that of the first embodiment described with reference to FIG. 3.

[0136] The parking support device 100 of the present embodiment includes a reception unit 101, an acquisition unit 102, an extraction unit 103, a path recording unit 104, a path image generation unit 105, an extended bird's-eye view image generation unit 106, a display control unit 107, a registration processing unit 108, an estimation unit 109, a vehicle control unit 110, and a storage unit 111, similar to the first embodiment. The reception unit 101, the path recording unit 104, the path image generation unit 105, the registration processing unit 108, the estimation unit 109, the vehicle control unit 110, and the storage unit 111 have the same functions as those of the first embodiment.

[0137] The acquisition unit 102 of the present embodiment, in addition to having the same functions as those of the first embodiment, acquires imaging images, that is, peripheral images, of the surroundings of the vehicle 1 during automatic driving in time series as the vehicle 1 moves, from a plurality of imaging devices 16a to 16c. The acquisition process during automatic driving is an example of the second acquisition step in the present embodiment.

[0138] The extraction unit 103 of the present embodiment has the same functions as those of the first embodiment, and extracts feature points from the surrounding images acquired during automatic driving.

[0139] The extended bird's-eye view image generation unit 106 of the present embodiment has the same functions as those of the first embodiment, and generates a bird's-eye view image 61 from the surrounding images acquired during automatic driving.

[0140] In addition, the extended bird's-eye view image generation unit 106 updates the extended bird's-eye view image 60 during automatic driving by synthesizing the bird's-eye view image 61 based on the surrounding images captured during automatic driving with the extended bird's-eye view image 60 during teacher driving stored in the storage unit 111. The said synthesis process is an example of a synthesis step.

[0141] For example, based on the feature points extracted by the extraction unit 103, the extended bird's-eye view image generation unit 106 specifies an image range corresponding to the bird's-eye view image 61 based on the surrounding images captured during automatic driving among the extended bird's-eye view images 60 during teacher driving, and replaces the specified image range with the bird's-eye view image 61. By repeating such processing, the range imaged by the imaging device 16 along the progress of the vehicle 1 among the extended bird's-eye view images 60 during teacher driving is sequentially replaced. Note that the method for synthesizing the extended bird's-eye view image 60 during teacher driving and the bird's-eye view image 61 based on the surrounding images captured during automatic driving is not limited to this.

[0142] In the present embodiment, the extended bird's-eye view image 60 during teacher driving is referred to as the first extended bird's-eye view image 60, and an image synthesized from the first extended bird's-eye view image 60 and the bird's-eye view image 61 based on the surrounding images captured during automatic driving is distinguished by being called the second extended bird's-eye view image.

[0143] The display control unit 107 of the present embodiment has the same functions as those of the first embodiment, and causes the display device 120 to display the second extended bird's-eye view image generated by the extended bird's-eye view image generation unit 106 during automatic driving. The said display process is an example of a third display step.

[0144] FIG. 16 is a diagram showing an example of an overhead path image 62g according to the fifth embodiment. In the second extended overhead image 63 shown in FIG. 16, the range in which the peripheral image is captured by the imaging device 16 of the vehicle 1 located near the start position 900 of the travel path 80 has replaced the overhead image 61 generated during the automatic driving. Further, in the range of the second extended overhead image 63 where the peripheral image has not been captured by the imaging device 16 yet, the first extended overhead image 60 remains displayed.

[0145] When the vehicle 1 is automatically driven by the vehicle control unit 110, the display control unit 107 displays an overhead path image 62g in which a travel path image 802 and a vehicle image 83 are superimposed on the second extended overhead image 63 as shown in FIG. 16.

[0146] More specifically, the display control unit 107 displays the vehicle image 83 at a position corresponding to the current position of the vehicle 1 on the second extended overhead image 63. The display control unit 107 moves the vehicle image 83 on the second extended overhead image 63 as the vehicle 1 moves by automatic driving.

[0147] In the example shown in FIG. 16, the display control unit 107 displays the range where the vehicle 1 has traveled on the travel path image 802 as a solid line and the range where the vehicle 1 has not traveled yet as a dashed line. With such a display, the user can grasp at which position on the travel path 80 the vehicle 1 is located.

[0148] Note that the display mode of the travel path image 802 shown in FIG. 16 is an example, and it is not necessary to change the display mode depending on whether the vehicle 1 is traveling or not. Further, the display control unit 107 does not necessarily display the travel path image 802 during the automatic driving.

[0149] In the example shown in FIG. 16, the display control unit 107 displays the overhead path image 62g so that the forward direction of the current vehicle 1 faces the Y direction of the display device 120. In this case, when the direction of the vehicle 1 changes, the orientation of the overhead path image 62g also changes.

[0150] Note that the display direction is an example and can be changed. For example, the display control unit 107 may include a first display mode in which the overhead path image 62g is displayed so that the forward direction of the current vehicle 1 faces the Y direction of the display device 120, and a second display mode in which the overhead path image 62g is displayed so that the forward direction of the vehicle 1 at the end point of the travel path 80 faces the Y direction of the display device 120. In the case of the second display mode, the orientation of the overhead path image 62g is fixed regardless of the change in the orientation of the vehicle 1 during automatic driving.

[0151] For example, the reception unit 101 may be configured to receive an operation for selecting the first display mode or the second display mode by the user. The display control unit 107 displays the overhead path image 62g in the display mode selected by the user.

[0152] Thus, in the parking support method executed by the parking support device 100 of the present embodiment, the second extended overhead image 63 is generated by synthesizing the overhead image 61 generated from the peripheral image captured during the automatic driving of the vehicle 1 based on the travel path 80 with the extended overhead image 60 during the teacher driving. Then, in the parking support method, the generated second extended overhead image 63 is displayed on the display device 120. Therefore, according to the parking support method executed by the parking support device 100 of the present embodiment, during the automatic driving of the vehicle 1 based on the travel path 80, the second extended overhead image 63 reflecting the state around the current vehicle 1 can be presented to the driver.

[0153] For example, when an obstacle that did not exist during the teacher driving exists around the vehicle 1, according to the parking support method executed by the parking support device 100 of the present embodiment, the obstacle can be depicted on the second extended overhead image 63.

[0154] (Modification Example 1) In each of the above-described embodiments, the display device 120 and the operation unit are mounted on the vehicle 1, but they may be provided outside the vehicle 1. For example, the display device 120 and the operation unit may be a mobile terminal such as a tablet terminal or a smartphone that can communicate with the parking support device 100 wirelessly or by wire. Note that it is sufficient if the tablet terminal or the like and the parking support device 100 of the vehicle 1 do not directly connect and can transmit and receive data and control signals via a server device or a cloud environment.

[0155] When the display device 120 and the operation unit are a tablet terminal or the like, the driver may use the tablet terminal or the like inside the vehicle 1, or may use the tablet terminal or the like in a state of getting out of the vehicle 1.

[0156] Alternatively, when the vehicle 1 is provided with a head-up display, the head-up display may be an example of the display unit. For example, an image may be projected onto the front glass 180 of the vehicle 1 by a projection device (not shown) to form a head-up display capable of displaying an image.

[0157] (Modification Example 2) In addition, the display timing at which the display control unit 107 of the parking support device 100 displays the overhead path images 62, 62a to 62g is not limited to the examples given in the above-described embodiments.

[0158] For example, the parking support device 100 of this modification example is assumed to have a function of receiving a modification operation of the travel route 80 by the user. In this case, after the end of the teaching run, when the reception unit 101 receives an operation instructing the display of a modification screen of the travel route 80 by the user, the display control unit 107 causes the overhead path images 62, 62a to 62g to be displayed on the display device 120.

[0159] The operation instructing the display of the modification screen of the travel route 80 by the user is, for example, an operation of modifying the travel route on the overhead path images 62, 62a to 62g. The process of receiving the operation is an example of a modification operation reception step.

[0160] More specifically, the user inputs a correction operation for the travel route 80 by operating the travel route image 801 included in the bird's-eye view route images 62, 62a to 62g with a finger or a mouse or the like. When the parking support device 100 receives a correction operation for the travel route 80 by the user, it executes recalculation based on the correction operation and corrects the travel route 80.

[0161] Note that the operation for instructing the display of the correction screen for the travel route 80 by the user may be executed while the user is in the vehicle 1, or may be executed after the user gets out of the vehicle 1 using a tablet terminal or the like as a display device and an operation unit. Further, not only the travel route 80 immediately after being recorded in the teaching run, but also the travel route 80 that has already been registered as an automatic travel route may be subject to a correction operation by the user.

[0162] (Modification Example 3) Note that in each of the above-described embodiments, the parking support device 100 uses the travel locus of the vehicle 1 in the teaching run as the travel route 80, but the parking support device 100 may generate the travel route 80 by correcting the travel locus of the vehicle 1 in the teaching run with respect to the speed and the travel position.

[0163] Also, in each of the above-described embodiments, the teaching run is such that the vehicle 1 is run from the start position 900 to the parking space 910 in the same manner as the travel of the vehicle 1 in the automatic run, but the teaching run may be in the reverse direction to the automatic run.

[0164] More specifically, it is also possible to use the parking space 910 as the starting point of the teaching run and the start position 900 as the end point of the teaching run. In this case, the route recording unit 104 generates the travel route 80 for automatic travel by converting the route of the teaching run in the reverse direction. In other words, the vehicle control unit 110 runs the vehicle 1 from the start position 900 to the parking space 910 in the reverse direction to the teaching run. Such a method is also referred to as reverse reproduction of the travel route.

[0165] (Modification Example 4) Further, a part of the functions of the parking support device 100 in each of the above-described embodiments may be executed by an information processing device provided outside the vehicle 1. Examples of the information processing device provided outside the vehicle 1 include mobile terminals such as smartphones, PCs, server devices, and the like. Note that the information processing device may be provided in a cloud environment.

[0166] Although some embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are also included in the invention described in the claims and the equivalent scope thereof.

Explanation of Reference Numerals

[0167] 1 Vehicle 12 Vehicle Body 15, 15f, 15r Transmission / Reception Wave Section 16, 16a~16d Imaging Device 40 Road 60, 60c~60f Expanded Overhead Image (First Expanded Overhead Image) 61 Overhead Image 62, 62a~62g Overhead Route Image 63 Second Expanded Overhead Image 70 Home 80 Travel Route 83 Vehicle Image 100 Parking Support Device 101 Reception Section 102 Acquisition Section 103 Extraction Section 104 Route Recording Section 105 Route Image Generation Section 106 Expanded Overhead Image Generation Section 107 Display Control Section 108 Registration Processing Section 109 Estimation Section 110 Vehicle Control Section 111 Storage Section 120 display device 121 Travel route selection screen 141 Operation button 141a Registration button 141b Redo button 143a - 143d Selection buttons 180 Windshield 801, 801c - 801f, 802 Travel route images 820, 820a, 820c - 820f Outline images 821, 821c - 820f Vehicle width images 822, 822a, 822c - 822f End images 900 Starting position 910 Parking space

Claims

1. A parking assistance method for performing automatic driving of a vehicle based on a teacher's drive by a driver, comprising: acquiring a captured image obtained by imaging the surroundings of the vehicle during the teacher's drive from an imaging device provided on the vehicle; recording the driving route of the vehicle during the teacher's drive; generating a first bird's-eye view image that looks down on a first display target range at a predetermined point; generating a second bird's-eye view image that looks down on a second display target range including the driving route and the periphery of the driving route; displaying the second bird's-eye view image on a display device; wherein the second display target range is wider than the first display target range; wherein the second bird's-eye view image includes a region displayed in black; A parking assistance method.

2. The parking assistance method according to claim 1, wherein the second display target range is a range including from the start position to the end position of the driving route during the teacher's drive. The parking assistance method according to claim 1.

3. displaying a vehicle image representing the vehicle at the position of the vehicle at the end of the teacher's drive on the second bird's-eye view image; The parking assistance method according to claim 1 or 2.

4. changing the orientation of the second bird's-eye view image based on a change in the direction of the vehicle; displaying the second bird's-eye view image on the display device at an angle such that the front direction of the vehicle faces upward on the screen of the display device; The parking assistance method according to claim 1.

5. displaying an image representing the driving route of the vehicle during the teacher's drive; The parking assistance method according to claim 1.

6. generating an image representing the movement locus of the outer contour of the vehicle body of the vehicle during the teacher's drive; displaying an image representing the driving route of the vehicle during the teacher's drive and an image representing the movement locus of the outer contour of the vehicle body of the vehicle during the teacher's drive; The parking assistance method according to claim 1.

7. generating a plurality of the first bird's-eye view images that look down on the first display target range at a plurality of the predetermined points on the driving route based on the captured image; generating the second bird's-eye view image based on the plurality of the first bird's-eye view images; The parking assistance method according to claim 1.

8. A parking assistance device for executing parking assistance for performing automatic driving of a vehicle based on a teacher's drive by a driver, comprising: an acquisition unit that acquires a captured image obtained by imaging the surroundings of the vehicle during the teacher's drive from an imaging device provided on the vehicle; a route recording unit that records the driving route of the vehicle during the teacher's drive; An overhead image generation unit that generates a first overhead image that looks down on a first display target range at a predetermined point, and a second overhead image that looks down on a second display target range including the driving route and the periphery of the driving route, A display control unit that causes the display device to display the second overhead image, The second display target range is wider than the first display target range, The second overhead image includes a region displayed in black, Parking assistance device.

9. The second display target range is a range including from the start position to the end position of the driving route in the teaching drive, The parking assistance device according to claim 8.

10. The display control unit displays a vehicle image representing the vehicle at the position of the vehicle at the end of the teaching drive on the second overhead image, The parking assistance device according to claim 8 or 9.

11. The display control unit, Based on the change in the direction of the vehicle, changes the orientation of the second overhead image, Displays the second overhead image on the display device at an angle at which the forward direction of the vehicle faces upward on the screen of the display device, The parking assistance device according to claim 8.

12. The display control unit displays an image representing the driving route of the vehicle in the teaching drive, The parking assistance device according to claim 8.

13. Further includes a route image generation unit that generates an image representing the movement locus of the outer contour of the vehicle body of the vehicle in the teaching drive, The display control unit displays an image representing the driving route of the vehicle in the teaching drive and an image representing the movement locus of the outer contour of the vehicle body of the vehicle in the teaching drive, The parking assistance device according to claim 8.

14. The overhead image generation unit generates a plurality of the first overhead images that look down on the first display target range at a plurality of the predetermined points on the driving route based on the captured image, and generates the second overhead image based on the plurality of the first overhead images, The parking assistance device according to claim 8.

Images