Parking assist system

Abstract

This invention provides a parking assistance device that enables parking assistance to a parking target position specified by the user. [Solution] A vehicle surrounding image showing the area around the vehicle 2 is displayed on the liquid crystal display 4. Within the vehicle surrounding image, a parking frame image 47 is displayed that identifies a parking target position specified by the user at an arbitrary position. If an obstacle that may affect parking assistance to the parking target position is detected, the system is configured to ask the user whether to start parking assistance to the parking target position.

Description

Technical Field

[0005] , ,

[0001] The present invention relates to a parking assistance device that performs parking assistance for a vehicle.

Background Art

[0002] Conventionally, when a vehicle parks, a camera image of the surrounding environment of the vehicle is displayed, a warning is given to people or obstacles located around, or some or all of the user's driving operations are executed on the vehicle side, etc. Thus, a parking assistance device that performs parking assistance has been proposed.

[0003] Here, in order to perform the above-described parking assistance, it is necessary to select a parking target position that is the target for parking the vehicle. Also, regarding the parking target position, it is possible to detect and select an empty space on the device side. However, when the parking target position is selected on the device side, for example, when there are obstacles (such as bicycles, color cones (registered trademarks), etc.) that can be removed by the user within or around the parking target position desired by the user, there is a problem that that position cannot be selected as the parking target position. Therefore, a technique for allowing the user to arbitrarily specify the parking target position has also been proposed. For example, in Japanese Patent No. 6801787, a technique is proposed in which an image around the vehicle is displayed on a display device and the user arbitrarily specifies a parking target position within the displayed image around the vehicle.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In the above-mentioned Patent Document 1, it is possible for the user to set the parking target position even if there are obstacles that can be removed by the user in or around the parking target position. However, on the other hand, it is also possible that the user may unintentionally specify a parking target position that has obstacles within or around it. As a result, problems have occurred, for example, when providing parking assistance to a parking target position specified by the user, such as the parking assistance being interrupted midway due to the detection of obstacles, or obstacles being located near the vehicle after parking against the user's wishes.

[0006] The present invention was made to solve the aforementioned problems of the conventional invention, and aims to provide a parking assistance device that enables parking assistance to a parking target location specified by the user by prompting the user whether to start parking assistance to the parking target location when an obstacle affecting parking assistance to the parking target location is detected. [Means for solving the problem]

[0007] To achieve the above objective, the parking assistance device according to the present invention is a parking assistance device that assists in parking a vehicle, and displays a vehicle surrounding image showing the area around the vehicle on a display device, displays location identification information that identifies a parking target position specified by the user at an arbitrary position in the vehicle surrounding image, and when an obstacle that affects parking assistance to the parking target position is detected, the device asks the user whether to start parking assistance to the parking target position. Furthermore, "vehicle surroundings image" may be a real-world image of the area around the vehicle captured by an imaging device such as a camera, or it may be a virtual image of the area around the vehicle recreated using computer graphics (CG). In addition, if it is a real-world image, it may be the captured image itself, or an image that has been processed from the captured image. For example, it may be an image created by combining images taken by multiple cameras, or an overhead or bird's-eye view image with a shifted viewpoint. Furthermore, "obstacles affecting parking assistance" may be obstacles located within the target parking position, obstacles located within a specified range from the target parking position, or obstacles on or around the parking path from the vehicle's current position to the target parking position. [Effects of the Invention]

[0008] According to the parking assistance device of the present invention having the above configuration, when an obstacle affecting parking assistance to a parking target position is detected, the user is asked whether to start parking assistance to the parking target position. If the user has recognized the obstacle and specified a parking target position, it becomes possible to provide parking assistance to the parking target position specified by the user. On the other hand, if the user has specified a parking target position without recognizing the obstacle, it is also possible to make the user recognize the obstacle. [Brief explanation of the drawing]

[0009] [Figure 1] This is a schematic diagram of the vehicle according to this embodiment. [Figure 2] This is a block diagram showing the configuration of the parking assistance device according to this embodiment. [Figure 3] This is a flowchart of the parking assistance processing program according to this embodiment. [Figure 4] This diagram illustrates the method for converting captured images into overhead views. [Figure 5] This diagram explains how to generate an overhead view image. [Figure 6] This diagram illustrates a method for setting parking target positions using an overhead view image. [Figure 7] This diagram shows the obstacle detection area. [Figure 8] This is an example of a display screen showing an inquiry form on an LCD display. [Figure 9] This is an example of a display screen showing an inquiry form on an LCD display. [Figure 10]This diagram illustrates how to adjust the parking target position. [Figure 11] This figure shows an example of the display screen on the LCD when the "Move Start" switch is displayed. [Modes for carrying out the invention]

[0010] Hereinafter, one embodiment of the parking assistance device according to the present invention will be described in detail with reference to the drawings. First, the vehicle 2 equipped with the parking assistance device 1 according to this embodiment will be described below. Figure 1 is a schematic diagram of the vehicle 2 according to this embodiment.

[0011] Here, Vehicle 2 may be, for example, an automobile powered by an internal combustion engine (internal combustion engine vehicle), an automobile powered by an electric motor (electric vehicle, fuel cell vehicle, etc.), or an automobile powered by both (hybrid vehicle). Furthermore, there is no restriction on the type of vehicle; it may be a regular passenger car, a large commercial truck, a bus, construction machinery, etc. Also, although the following explanation will refer to it as a four-wheeled vehicle, it may also be a two-wheeled or three-wheeled vehicle.

[0012] However, Vehicle 2 shall be a vehicle capable of not only manual driving based on the user's driving operations, but also assisted driving through automated driving assistance, in which the vehicle drives automatically without user operation.

[0013] Furthermore, the automated driving assistance may be performed only under specific circumstances, such as when parking or exiting a parking space, or it may be performed on all sections of road, or it may be configured to be performed only while the vehicle is traveling on a specific section of road (for example, a highway with gates (regardless of whether they are manned or unmanned, tolled or free) at the boundary). In this embodiment, the automated driving assistance will be provided at least for the vehicle's journey to the parking space when parking (i.e., parking assistance).

[0014] In the vehicle control for automatic driving support in this embodiment, for example, the current position of the vehicle, the lane in which the vehicle is traveling, and the positions of surrounding obstacles are detected at any time, and the vehicle control such as steering, drive source, and brake is automatically performed so that the vehicle travels at a speed according to the generated speed plan along the generated driving trajectory. Particularly when performing parking support, as will be described later, the user is allowed to specify a parking target position at an arbitrary position, or a previously registered parking target position is read from the memory and designated as the current parking target position. Then, a parking trajectory to the designated parking target position is calculated, and the vehicle is automatically controlled to enter the parking target position along the calculated parking trajectory and complete the parking. Further, in addition to the above normal parking support, for example, it also supports long-range parking in which parking is performed targeting a distant parking space such as a garage at home or a monthly-paid parking space in a parking lot. In the parking support for long-range parking, the vehicle control for moving to a preset distant parking target position and completing the parking is automatically performed. However, it is also possible to automatically perform only the steering operation and perform the control of the drive source and the brake based on manual operation. Alternatively, only the instruction for the parking operation may be given, and the parking operation may be performed manually by the user. Further, when performing the above parking support, the in-vehicle display also displays the scenery around the vehicle (either a real scene or a CG image), and also displays the parking target position specified by the user and the obstacles around it.

[0015] Also, as shown in FIG. 1, the vehicle 2 includes an operation unit 3 that receives operations from the occupant, a liquid crystal display 4 that displays an aerial image, an overhead image, or other information related to driving support around the vehicle to the occupant, a speaker 5 that outputs voice guidance related to driving support, front cameras 6, rear cameras 7, and side cameras 8A and 8B for imaging the periphery of the vehicle, ultrasonic sensors 9A to 9L that detect obstacles around the vehicle, and a driving support ECU (Electronic Control Unit) 10 that performs various arithmetic processes based on the input information. Here, the parking support device 1 includes the above driving support ECU 10.

[0016] The following describes each component of the vehicle 2. First, the operation unit 3 is arranged, for example, on the front surface of the steering wheel (also referred to as a steering wheel), and includes operation buttons and the like that are operated when starting automatic driving support. By operating the operation unit 3, the user can switch between manual driving in which the vehicle travels based on the user's driving operation and assisted driving by automatic driving support in which the vehicle automatically travels without the user's driving operation. In particular, in this embodiment, the operation unit 3 also includes a touch panel provided on the front surface of the liquid crystal display 4. The user operates the touch panel to specify a parking target position during parking support and to answer inquiries from the parking support device 1 as described later. Further, the operation unit 3 may include a microphone and a voice recognition device.

[0017] The liquid crystal display 4 is a type of display device, is installed on the instrument panel of the vehicle 2, and displays an overhead image or a bird's-eye view image of the vehicle surroundings generated by performing viewpoint conversion and synthesis processing on captured images taken by the front camera 6, rear camera 7, and side cameras 8A and 8B as a vehicle surroundings image showing the surroundings of the vehicle during the execution of automatic driving support. However, a virtual image that reproduces the surroundings of the vehicle in CG instead of a real-scene image may be displayed. Further, when starting parking support, position specifying information for specifying a parking target position is also displayed superimposed on the vehicle surroundings image. Further, an inquiry screen including a sentence asking whether to start parking support for the parking target position specified by the user is also displayed. Incidentally, the liquid crystal display 4 may be used in common with the navigation device.

[0018] Also, the speaker 5 is installed on the instrument panel of the vehicle 2 and outputs guidance voices, warning sounds, and the like related to driving support. Incidentally, the speaker 5 may be used in common with the navigation device.

[0019] Also, the front camera 6 is an imaging device having a camera using a solid-state imaging device such as a CCD, and is installed, for example, above the front bumper of the vehicle 2 or behind the rearview mirror with the optical axis direction facing forward in the traveling direction of the vehicle.

[0020] The rear camera 7 is an imaging device that also has a camera using a solid-state image sensor such as a CCD, and is mounted, for example, near the center above the license plate attached to the rear of the vehicle 2, with the optical axis facing the rear of the vehicle.

[0021] Furthermore, the side cameras 8A and 8B are imaging devices that also have cameras using solid-state image sensors such as CCDs, and are mounted, for example, on the left and right side mirrors of vehicle 2, with the optical axis facing the side of the vehicle.

[0022] The driver assistance ECU 10 generates overhead and bird's-eye views of the vehicle's surroundings by performing viewpoint transformation and synthesis processing on the images captured by the front camera 6, rear camera 7, and side cameras 8A and 8B. Furthermore, during automated driving assistance, it performs image recognition processing on the captured images to detect lane markings, parking space markings, and obstacles (other vehicles, pedestrians, bicycles, walls, guardrails, and other structures) around the vehicle, and performs automated driving assistance based on the detection results. In particular, when performing parking assistance, it checks whether there are any obstacles that would affect parking assistance to the parking target position based on the obstacle detection results, and prompts the user for clarification as necessary.

[0023] On the other hand, ultrasonic sensors 9A to 9L are arranged at predetermined intervals on the front, rear, and sides of the vehicle, respectively. They transmit ultrasonic waves as probe waves around the vehicle 2 and detect objects that reflected the probe waves by receiving reflected waves from objects around the vehicle. Specifically, they are a type of distance measuring sensor capable of detecting the distance (measured distance value) to the object that reflected the probe waves by measuring the time from transmission to reception. Furthermore, ultrasonic sensors 9A to 9L are configured to generate an output signal (including the distance to the detected object) corresponding to the reception result of the received wave and output it to the control unit. The objects to be detected by ultrasonic sensors 9A to 9L include, for example, people, bicycles, other vehicles, walls, and other obstacles that the vehicle 2 needs to avoid when driving, or obstacles that form a parking space. In addition, millimeter-wave sensors or laser sensors may be used as distance measuring sensors instead of ultrasonic sensors.

[0024] Furthermore, while the installation position and direction of each ultrasonic sensor 9A to 9L can be set as appropriate, in this embodiment, in order to make the detection range of the target object encompass all directions in front of, behind, and to the left and right of the vehicle's direction of travel, for example, ultrasonic sensors 9A to 9D are installed on the front of the vehicle 2 facing the direction of travel so that the direction of transmission of the probe wave is in front of the vehicle's direction of travel. Ultrasonic sensors 9E and 9F are installed on the left side of the vehicle 2 facing left so that the direction of transmission of the probe wave is to the left of the vehicle's direction of travel. Ultrasonic sensors 9G and 9H are installed on the right side of the vehicle 2 facing right so that the direction of transmission of the probe wave is to the right of the vehicle's direction of travel. Ultrasonic sensors 9I to 9L are installed on the rear of the vehicle 2 facing the opposite direction of travel so that the direction of transmission of the probe wave is to the rear of the vehicle. The height of each ultrasonic sensor 9A to 9L from the ground surface is approximately the same.

[0025] In this embodiment, among the ultrasonic sensors 9A to 9L, the ultrasonic sensors 9A to 9D on the front of the vehicle 2 and the ultrasonic sensors 9I to 9L on the rear of the vehicle 2 are installed in positions where they can receive reflected waves as indirect waves from adjacent sensors. By receiving both direct and indirect waves, it is possible to determine not only the distance to the object but also the specific position of the object (relative position to the vehicle) using triangulation. The ultrasonic sensors 9E to 9H on the sides are installed spaced apart from each other and cannot receive indirect waves, but as the vehicle moves, it is possible to determine the specific position of the object (relative position to the vehicle) using triangulation with respect to the distance measured at the previous position, the distance measured at the current position, and the distance traveled in between.

[0026] On the other hand, the driver assistance ECU 10 is an electronic control unit that performs various processes related to automated driving assistance. The driver assistance ECU 10 is connected to the aforementioned control unit 3, liquid crystal display 4, speaker 5, front camera 6, rear camera 7, side cameras 8A, 8B, and ultrasonic sensors 9A~9L via an in-vehicle network such as CAN. It is also connected to various sensors mounted on the vehicle 2, such as a vehicle speed sensor, acceleration sensor, gyro sensor, steering sensor, and shift position sensor, as well as in-vehicle devices such as the navigation system. The detailed configuration of the driver assistance ECU 10 will be described later.

[0027] In addition to the components shown in Figure 1, Vehicle 2 also has other basic components as Vehicle 2, but only the configuration related to the control of automated driving assistance (especially parking assistance) and the control related to said configuration will be explained.

[0028] Next, we will explain in detail the driver assistance ECU 10, which is part of the parking assistance system 1 provided by the vehicle 2 described above. Figure 2 is a block diagram showing the configuration of the parking assistance system 1 according to this embodiment.

[0029] As shown in Figure 2, the driver assistance ECU (Electronic Control Unit) 10 is an electronic control unit that controls the entire parking assistance device 1, and includes a CPU 31 as a calculation device and control device, a RAM 32 which is used as working memory when the CPU 31 performs various calculations and stores driving trajectory data when the driving trajectory is calculated, a ROM 33 which stores control programs as well as the parking assistance processing program (see Figure 3) described later, and a flash memory 34 which stores programs read from the ROM 33. The driver assistance ECU 10 also executes various functions as a processing algorithm. For example, it has the function of displaying a vehicle surrounding image showing the area around the vehicle on the liquid crystal display 4, a function of displaying location identification information that identifies a parking target position specified by the user at an arbitrary position in the vehicle surrounding image, a function of asking the user whether to start parking assistance for the parking target position when an obstacle affecting parking assistance to the parking target position is detected, and a function of having the user specify a new parking target position or correcting the parking target position to a position unaffected by obstacles if the user's response to the inquiry is to deny starting parking assistance.

[0030] Furthermore, the driver assistance ECU 10 is connected to various sensors 36 for detecting the vehicle's behavior, such as a vehicle speed sensor, acceleration sensor, gyro sensor, steering sensor, and shift position sensor, as well as to various drive units 37 of the vehicle, such as the steering, brakes, accelerator, and transmission. Based on the detection results of these sensors 36, the ECU 10 detects the vehicle's current behavior and controls each drive unit 37 to provide automatic driving assistance for the vehicle 2. Specifically, in the case of parking assistance, the ECU 10 continuously detects the vehicle's current position and the positions of surrounding obstacles, and controls the steering, drive source, brakes, etc., to drive along the generated parking path. In particular, if an obstacle is detected within a predetermined distance from the vehicle by ultrasonic sensors 9A to 9L while driving with parking assistance, the ECU 10 controls the vehicle to decelerate and stop, or to perform avoidance control to avoid the obstacle. However, it is also possible to perform only the steering operation automatically, and the control of the drive source and brakes based on manual operation. Alternatively, it is also possible to only give instructions for parking operations, and have the user perform the parking operation manually.

[0031] Furthermore, ROM33 includes vehicle information DB35, which stores various information about vehicle 2. For example, it stores the installation positions (height from the ground, left-right position) and detection axes (optical axis for cameras) of cameras and ultrasonic sensors 9A-9L installed on vehicle 2, as well as the overall length, vehicle width, wheelbase, and minimum turning radius. This information is entered in advance by the occupants or personnel from the vehicle manufacturer.

[0032] Next, the parking assistance processing program executed by the driver assistance ECU 10 in the parking assistance device 1 having the above configuration will be explained with reference to Figure 3. Figure 3 is a flowchart of the parking assistance processing program according to this embodiment. Here, the parking assistance processing program is executed after the ACC power supply (accessory power supply) of the vehicle 2 is turned ON and the start conditions for starting parking assistance are met, and is a program that provides parking assistance, especially when the vehicle is parked, as one of the automatic driving assistance functions. The program shown in the flowchart in Figure 3 below is stored in the RAM 32 and ROM 33 of the parking assistance device 1 and is executed by the CPU 31.

[0033] Furthermore, the conditions for initiating parking assistance may include the user selecting to start parking assistance by operating the control unit 3, the detection of the vehicle entering a parking lot, the determination of arrival at a set destination, or the detection of approaching a parking lot. However, in this embodiment, the conditions for initiating parking assistance are defined as the vehicle 2 stopping near the parking space designated by the user as the parking target position, and the user selecting to start parking assistance by operating the control unit 3. In particular, in the following description, it will be assumed that the position where the user stops the vehicle 2 is a position where the parking target position specified by the user can be imaged by at least one of the front camera 6, rear camera 7, or side cameras 8A, 8B.

[0034] First, in step 1 (hereinafter abbreviated as S), the CPU 31 generates an overhead view image of the area around the vehicle, looking down vertically from above, based on real-time images captured by the front camera 6, rear camera 7, and side cameras 8A and 8B. The method for generating the overhead view image is described below. As shown in Figure 4, the real-time images captured by each camera are projected onto a virtual projection plane, which is a horizontal plane corresponding to the height of the ground surface. The images projected onto the virtual projection plane are then converted into images viewed from a virtual viewpoint looking vertically down from above the vehicle 2, thereby generating the overhead view image for each camera. The conversion to an image viewed from a virtual viewpoint (viewpoint conversion) is performed by first converting each coordinate in the image coordinate system, which is set along a plane perpendicular to the optical axis of the camera, to each coordinate in the ground coordinate system, which is set along the ground surface, and then to each coordinate in the overhead view image coordinate system. The conversion formulas used for each coordinate conversion are already publicly known, so their explanation is omitted. Then, as shown in Figure 5, the overhead image 46 is generated by combining (stitching together) the overhead image 41 obtained by changing the viewpoint of the image captured by the front camera 6, the overhead image 42 obtained by changing the viewpoint of the image captured by the rear camera 7, the overhead image 43 obtained by changing the viewpoint of the image captured by the side camera 8A, and the overhead image 44 obtained by changing the viewpoint of the image captured by the side camera 8B. Furthermore, an illustration image 45 schematically showing the vehicle is inserted between each of the overhead images 41 to 44.

[0035] Next, in S2, the CPU 31 displays the overhead image 46 created in S1 on the liquid crystal display 4.

[0036] In the following explanation, we will describe an example of displaying an overhead view image 46, which shows the area around the vehicle viewed vertically downwards from above, as an image of the area around the vehicle. However, it is also possible to display a bird's-eye view image, which shows the area around the vehicle viewed diagonally downwards from above. Furthermore, in the bird's-eye view image, the viewpoint can be set to, for example, the inside of the vehicle, in which case a diagram showing the area around the vehicle as seen from inside the vehicle can be displayed as the image of the area around the vehicle. Alternatively, it is also possible to display real-time images captured by any of the front camera 6, rear camera 7, and side cameras 8A, 8B without any processing.

[0037] Next, in S3, the CPU 31 obtains the parking target position specified by the user at an arbitrary location. Here, the parking target position may be obtained by having the user specify an arbitrary location using the overhead image 46 displayed on the LCD display 4, or it may be obtained by reading a pre-registered parking target position from the flash memory 34.

[0038] The following describes an example of a method for allowing the user to specify a parking target position using an overhead view image 46 displayed on the LCD display 4. First, the CPU 31 displays a parking frame image (position identification information) 47 on the overhead view image 46. The parking frame image 47 is rectangular in shape, is an object image approximately the same size as the vehicle 2, and is displayed on the LCD display 4 to indicate the parking target position. Figure 6 is an example of the display screen shown on the LCD display 4 immediately after the start of the parking assistance processing program. The CPU 31 changes the position and orientation of the parking frame image 47 in response to the user's touch panel operation on the LCD display 4. This allows the user to place the parking frame image 47 at any position within the vehicle surroundings image displayed on the LCD display 4, and the user can set the parking target position arbitrarily by moving the parking frame image 47. In addition, a complete button 48 is also displayed on the overhead view image 46 along with the parking frame image 47. The user moves the parking frame image 47 to the desired parking target position and then operates the complete button 48. As a result, the position where the parking space image 47 is displayed at the time the complete button 48 is pressed is set as the target parking position. In addition to specifying the target parking position, the user may also specify the parking method (e.g., parallel parking, straight parking, forward parking, reverse parking, etc.).

[0039] The CPU 31 then stores the coordinates of the parking target position, which were set using the overhead image 46 and parking space image 47 displayed on the liquid crystal display 4 shown in Figure 6, into the flash memory 34 as the parking target position for this parking assistance operation. The coordinates to be stored may be in any coordinate system, and coordinate transformations may be performed using well-known and conventional techniques.

[0040] Furthermore, the CPU 31 may store the parking target position set using the overhead image 46 and parking space image 47 displayed on the liquid crystal display 4 shown in Figure 6 as a registered parking position in the flash memory 34 when a user performs a registration operation. This allows the user to specify a parking target position not only using the overhead image 46 and parking space image 47 displayed on the liquid crystal display 4 shown in Figure 6, but also by reading a previously registered parking position from the flash memory 34 and specifying that registered parking position as the current parking target position.

[0041] Next, in S4, the CPU 31 performs image recognition processing on the overhead view image generated in S1 to detect obstacles included in the overhead view image. The types of obstacles to be detected are not particularly limited and include moving objects such as pedestrians, bicycles, and other vehicles, as well as stationary objects such as utility poles, walls, and steps. On the other hand, only specific types of obstacles such as pedestrians and bicycles may be targeted for detection. In this embodiment, image recognition processing is performed on the overhead view image generated in S1, but the image recognition processing may also be performed on the camera image that is used as the source for the overhead view image.

[0042] In S4, the process for detecting obstacles can be, for example, to perform brightness correction based on the brightness difference between the road surface and the obstacle, then perform binarization to separate the obstacle from the image, geometric processing to correct distortion, and smoothing processing to remove noise from the image, thereby detecting the boundary line between the road surface and the obstacle. Furthermore, detection may also be performed using known template matching processing or feature point detection processing. In addition, the image recognition processing on the captured image is not limited to the above example, and may be performed using, for example, machine learning.

[0043] Furthermore, in addition to image recognition processing, ultrasonic sensors 9A to 9L may also be used in combination to detect obstacles. While detection by ultrasonic sensors 9A to 9L can pinpoint the location of obstacles around the vehicle, it is generally not possible to identify the category (type) of the detected obstacles. However, by using a camera in conjunction, it may be possible to identify the type of obstacle detected by ultrasonic sensors 9A to 9L.

[0044] Subsequently, in S5, the CPU 31 uses the obstacle detection results from S4 to determine whether any of the detected obstacles are "obstacles that affect parking assistance to the parking target position (hereinafter referred to as influencing obstacles)." Here, influencing obstacles include, for example, obstacles within the parking target position, i.e., the parking frame image 47, or obstacles located between the vehicle 2 and the parking target position, i.e., obstacles located on or around the parking track. In addition, the CPU 31 may also recognize as influencing obstacles any object that the vehicle 2 cannot or cannot avoid when parking at the parking target position.

[0045] In the following explanation, we will describe an example in which the CPU 31 sets an area within a predetermined distance from the parking target position as an obstacle detection area and determines obstacles within the obstacle detection area as influential obstacles. For example, in the example shown in Figure 7, the area extending a predetermined distance in all directions from the parking frame image 47 is set as the obstacle detection area 50. As a result, the CPU 31 determines that obstacle 51, which is located within the obstacle detection area 50, is an influential obstacle. On the other hand, obstacle 52 is outside the obstacle detection area 50 and is therefore determined not to be an influential obstacle.

[0046] If it is determined that there is an influencing obstacle (S5: Yes), the process proceeds to S6. Conversely, if it is determined that there is no influencing obstacle (S5: No), the process proceeds to S8.

[0047] In this embodiment, since obstacle detection is performed using the overhead image 46, obstacles determined to be influential obstacles in S5 will basically be included as obstacle images in the overhead image 46 displayed on the liquid crystal display 4. That is, obstacles 51 and 52 included in the overhead image 46 displayed on the liquid crystal display 4 become obstacle images that indicate the location of the obstacles in the overhead image 46. Therefore, if influential obstacles exist, the user can understand the location and type of influential obstacles by looking at the overhead image 46. Alternatively, a model image that mimics the appearance of the influential obstacles may be superimposed and displayed at the location of the influential obstacles in the overhead image 46. This makes it possible to clearly indicate the presence and location of the influential obstacles to the user, even if the influential obstacles in the overhead image 46 are displayed distorted.

[0048] In S6, the CPU 31 asks the user whether to start parking assistance for the target parking position. The user's inquiry may be made by displaying an inquiry screen containing a message asking whether to start parking assistance for the target parking position on the LCD display 4, or by outputting the same message as an audio from the speaker 5.

[0049] The following describes an example of an inquiry using the liquid crystal display 4. Here, Figure 8 is an example of a display screen shown on the liquid crystal display 4 when it is determined that there is an obstacle 51 within the obstacle detection area 50, as shown in Figure 7. In Figure 8, the CPU 31 displays a confirmation dialog 55 in front of the overhead image 46 displayed on the liquid crystal display 4, and makes an inquiry with a sentence asking whether to start parking assistance for the parking target position. It is desirable that the CPU 31 displays the confirmation dialog 55 in such a position and size that the obstacle 51, which was determined to be an influencing obstacle in S5, and the parking frame image 47, which is the parking target position, can be recognized on the same screen. This allows the user to recognize the inquiry from the confirmation dialog 55 and at the same time confirm the positional relationship between the obstacle 51 and the parking frame image 47. However, the confirmation dialog 55 may also be displayed in the center of the screen at a fixed size.

[0050] Furthermore, the CPU 31 may change the content of the inquiry to the user regarding whether to start parking assistance for the parking target position, depending on the location of the influencing obstacle. For example, Figure 9 shows the case where an influencing obstacle 51 is located within the parking frame image 47, i.e., within the parking target position, and the content of the inquiry is changed. Specifically, the content is to inform the user that an obstacle is located within the parking target position. In addition, the CPU 31 may relatively emphasize the obstacle 51 by displaying an object image along the contour of the obstacle 51, making it easier for the user to recognize the obstacle.

[0051] Next, in S7, the CPU 31 obtains the user's response to the inquiry made in S6 and determines whether the user's response affirms or denies the commencement of parking assistance. As shown in Figures 8 and 9, the confirmation dialog 55 displayed on the liquid crystal display 4 includes an operation button 55A for obtaining that the user's response affirms the commencement of parking assistance, and an operation button 55B for obtaining that the user's response denies the commencement of parking assistance. Thus, the CPU 31 can obtain the response when the user operates either operation button 55A or operation button 55B via touch panel operation on the liquid crystal display 4. The user's response may also be obtained by voice; in this case, the user's voice is collected by a microphone installed in the vehicle and analyzed to obtain the user's response.

[0052] If the user's response is determined to be affirmative (S7: Yes), the system proceeds to S8. Conversely, if the user's response is determined to be negative (S7: No), the system returns to S2. After obtaining the user's response, the CPU 31 removes the confirmation dialog 55 from the display screen of the LCD display 4.

[0053] Furthermore, if the user's response affirms the initiation of parking assistance despite the presence of an influencing obstacle, it means that parking assistance will be provided to the parking target location specified by the user. In such cases, for example, the user may plan to remove the obstacle before the vehicle starts moving after the response, or the user may perceive the obstacle as having no impact on the vehicle even when approached.

[0054] On the other hand, if the user's response is negative and does not initiate parking assistance, the system returns to S2. However, if the current parking target position was obtained by reading a pre-registered parking target position from the flash memory 34, the parking assistance device 1 then automatically corrects the parking target position and displays the parking frame image 47 again in the overhead image 46 at a position corresponding to the corrected parking target position.

[0055] For example, Figure 10 illustrates how the parking target position is modified in a scenario where the user rejects the initiation of parking assistance. In Figure 10, obstacle 51, which is an influencing obstacle, is located within the obstacle detection area 50, while obstacle 52, which is determined not to be an influencing obstacle, is located outside the obstacle detection area 50. In such a scenario, the CPU 31 moves the parking frame image 47, i.e., the parking target position, away from obstacle 51. If obstacles 51 and 52 are not located within the obstacle detection area 50 at the new parking target position after the move, the CPU 31 uses the new parking target position as the proposed modification for the parking target position. In other words, the CPU 31 modifies the parking target position to a position unaffected by the obstacles. Even if obstacles 51 and 52 are located within the obstacle detection area 50 at the new parking target position after the move, if the intrusion into the obstacle detection area 50 is minor compared to before the move, it may still be used as the proposed modification for the parking target position. As described above, the CPU 31 generates proposed modifications for the parking target position.

[0056] Subsequently, in S5, the CPU 31 determines whether or not influencing obstacles still exist after the parking target position has been corrected. If it determines that influencing obstacles exist, it again asks the user whether to start parking assistance for the parking target position (Figures 8 and 9). However, the inquiry regarding the corrected parking target position may be displayed on a separate screen different from Figures 8 and 9, and for example, the parking target position before correction may also be displayed.

[0057] On the other hand, if it is determined in S7 that the user's response denies the commencement of parking assistance, and the user has obtained the current parking target position by specifying an arbitrary position, the parking target position setting screen shown in Figure 6 is displayed again on the liquid crystal display 4, and the user is prompted to specify a new parking target position.

[0058] Furthermore, the process of the user specifying a new parking target position is basically performed in the same way as S3 described above. That is, as shown in Figure 6, the parking frame image 47 and the complete button 48 are displayed superimposed on the overhead image 46, and the CPU 31 changes the position and orientation of the parking frame image 47 in response to the user's touch panel operation on the LCD display 4. Then, the position where the parking frame image 47 is displayed when the complete button 48 is pressed is set as the new modified parking target position. In this way, the user can modify the position of the parking frame image 47 within the vehicle surrounding image displayed on the LCD display 4, and the user can set a new parking target position by moving the parking frame image 47.

[0059] Subsequently, in S5, the CPU 31 determines whether or not there are any influencing obstacles at the newly designated parking target location. If it determines that there are influencing obstacles, it again asks the user whether to start parking assistance for the parking target location (Figures 8 and 9). However, the inquiry regarding the newly designated parking target location may be displayed on a separate screen different from Figures 8 and 9, and for example, the previously designated parking target location may also be displayed.

[0060] Furthermore, although not limited to the above example, if it is determined in S7 that the user's response denies the commencement of parking assistance, it is possible to appropriately choose whether to have the user specify a new parking target position or to automatically correct the parking target position. In addition, if it is determined in S7 that the user's response denies the commencement of parking assistance, and the user does not subsequently specify a new parking target position, or if there is insufficient parking space and the parking target position cannot be automatically corrected, the parking assistance processing program may be terminated after informing the user that parking assistance cannot be commenced.

[0061] On the other hand, in S8, which is executed when it is determined in S7 that the user's response affirms the commencement of parking assistance, the CPU 31 displays a move start switch 61 on the liquid crystal display 4 to initiate movement to the parking target position. Figure 11 shows an example of the display screen of the liquid crystal display 4 when the move start switch 61 is displayed. If the CPU 31 determines that the move start switch 61 has been operated by touch panel operation on the liquid crystal display 4 (S8: Yes), it proceeds to S9. This makes it possible to start the movement of the vehicle to the parking target position triggered by the user's instruction to start moving the vehicle. On the other hand, if it is determined that the move start switch 61 has not been operated (S8: No), it waits until it is operated. That is, the vehicle will not start moving until the move start switch 61 is operated.

[0062] In this embodiment, after responding to the inquiry in S6 with a response affirming the commencement of parking assistance, the vehicle is moved by operating the move start switch 61. As a result, if the influencing obstacle is one that can be removed by the user (e.g., a bicycle, a traffic cone, etc.), after responding to the inquiry with a response affirming the commencement of parking assistance, the user has the opportunity to get out of the vehicle and move the obstacle. Then, by operating the move start switch 61 after moving the obstacle, parking assistance can be performed without being affected by the influencing obstacle.

[0063] Subsequently, in S9, the CPU 31 calculates a parking trajectory, which is the driving path for parking vehicle 2, based on the parking target position acquired in S3 (the corrected parking target position if it has been modified, or the newly specified parking target position if it has been newly specified) and parking items such as parallel parking, perpendicular parking, forward parking, and reverse parking, and provides parking assistance such as automatic control or operation guidance. For example, in parking assistance, the current position of the vehicle and the positions of surrounding obstacles are detected at all times, and vehicle control such as steering, drive source, and brakes is performed so that the vehicle drives along the generated parking trajectory. In particular, if an obstacle is detected within a predetermined distance from the vehicle by ultrasonic sensors 9A to 9L while driving with parking assistance, control is performed to decelerate and stop, or avoidance control is performed to avoid the obstacle.Therefore, if the user chooses to start parking assistance toward the parking target position when an influencing obstacle exists, the influencing obstacle may be detected by ultrasonic sensors 9A to 9L, and the above stop control may be performed, canceling the parking assistance, or avoidance control may be performed.

[0064] Furthermore, for parking assistance, only steering may be automated, while the drivetrain and brakes may be controlled manually. Alternatively, only parking instructions may be given, with the user performing the parking maneuver manually.

[0065] As described in detail above, according to the parking assistance device 1 and the computer program executed by the parking assistance device 1 according to this embodiment, a vehicle surrounding image showing the area around the vehicle 2 is displayed on the liquid crystal display 4 (S2), a parking frame image 47 that identifies a parking target position specified by the user at an arbitrary position is displayed in the vehicle surrounding image (S3), and if an obstacle that affects parking assistance to the parking target position is detected, the user is asked whether to start parking assistance to the parking target position (S6). Therefore, if the user has recognized an obstacle and specified a parking target position, it is possible to provide parking assistance to the parking target position specified by the user. On the other hand, if the user has specified a parking target position without recognizing an obstacle, it is also possible to make the user recognize the obstacle. Furthermore, if the user's response to the inquiry is one that rejects the commencement of parking assistance, the system will either prompt the user to specify a new parking target location or modify the parking target location to a location unaffected by obstacles. This allows the system to continue parking assistance after modifying the parking target location to a new location unaffected or minimally affected by the obstacles if obstacles are detected at the target location. Furthermore, the inquiry is made by displaying an inquiry screen containing a message asking whether to start parking assistance for the target parking location on the LCD display 4 along with the location identification information (S6). This allows the user to see the currently designated target parking location and then decide whether or not to perform parking assistance for that location. Furthermore, if the user's response to the inquiry affirms the commencement of parking assistance, the vehicle will begin moving towards the parking target position (S9) triggered by the user's subsequent instruction to begin moving the vehicle. Therefore, if the obstacle is one that can be removed by the user, after responding to the inquiry affirming the commencement of parking assistance, the user can get out of the vehicle and move the obstacle. Then, by giving the instruction to begin moving after the obstacle has been moved, parking assistance can be performed without being affected by the obstacle.

[0066] [Note] The embodiments described above also disclose the following inventions. In the following description, the names and expressions of corresponding components in the embodiments, as well as the reference numerals used in the drawings, are indicated in parentheses for reference. However, the components of each invention are not limited to these indications.

[0067] (Invention A) The parking assistance device (1) according to claim 1, wherein the designation of the parking target position is performed by the user arbitrarily designating a position in the vehicle surrounding image (46) displayed on the display device (4).

[0068] According to this, users will be able to specify a parking target location after checking the surrounding conditions of the vehicle in the vehicle surroundings image.

[0069] (Invention B) A parking assistance device (1) according to claim 1, comprising a storage medium (34) for registering parking target positions previously designated by the user at any location, wherein the designation of the parking target position is performed by reading the registered parking target position from the storage medium.

[0070] According to this, when specifying the same parking target location as in the past, the operation of specifying the parking target location can be omitted, thereby improving user convenience.

[0071] (Invention C) The location identification information is an image of a parking frame (47, 60) superimposed on the parking target position in the vehicle surrounding image (46), Obstacle images (51, 52) indicating the obstacle are displayed at the location of the obstacle in the vehicle surrounding image. The parking assistance device according to claim 1, which, when the obstacle image is displayed within or around the image of the parking space, makes the inquiry on the grounds that an obstacle affecting parking assistance to the parking target position has been detected.

[0072] According to this, if an obstacle image is displayed in the vicinity of the location information or if it overlaps with the location information, a query will be made. Therefore, if the user has specified the parking target location after recognizing the positional relationship between the parking target location and the obstacle, it will be possible to provide parking assistance to the parking target location specified by the user.

[0073] It should be noted that the present invention is not limited to the embodiments described above, and various improvements and modifications are possible without departing from the spirit of the invention. For example, in this embodiment, the vehicle is started to move by first responding to the inquiry in S6 or S9 with a response affirming the start of parking assistance, and then operating the move start switch 61. However, the vehicle may also start moving at the time the response affirming the start of parking assistance is given to the inquiry in S6 or S9.

[0074] Furthermore, in the inquiry screens shown in Figures 8 and 9, a confirmation dialog 55 is displayed in front of the overhead image 46 to ask whether to start parking assistance for the parking target position. However, it is also possible to display only the confirmation dialog 55 without displaying the overhead image 46.

[0075] Furthermore, in this embodiment, when parking assistance is performed, the vehicle surrounding image displayed on the liquid crystal display 4 is an overhead image generated from images captured by the front camera 6, rear camera 7, and side cameras 8A and 8B. However, the vehicle surrounding image displayed on the liquid crystal display 4 may be a schematicly generated virtual landscape image (for example, a 3D map image) rather than an image captured by the cameras.

[0076] Furthermore, in this embodiment, the driver assistance ECU 10 of the parking assistance device 1 executes the processing of the parking assistance processing program (Figure 3), but the execution entity can be changed as appropriate. For example, the control unit of the liquid crystal display 4, the vehicle control ECU, the control unit of the navigation device, or other in-vehicle devices may be used to execute the processing. [Explanation of symbols]

[0077] 1…Parking assist system, 2…Vehicle, 3…Control unit, 4…LCD display (display device), 10…Driver assistance ECU, 31…CPU, 34…Flash memory, 46…Overhead view image (image of the area around the vehicle), 47…Parking space image (location identification information), 51, 52…Obstacles, 55…Confirmation dialog (inquiry screen), 60…Parking space image after correcting the parking target position (location identification information)

Claims

1. A parking assist device that assists in parking a vehicle, The vehicle's surroundings are shown in the display device, In the aforementioned vehicle surroundings image, location identification information is displayed to identify the parking target location specified by the user at any location. A parking assistance device that, when an obstacle affecting parking assistance to the parking target position is detected, prompts the user to inquire whether to start parking assistance to the parking target position.

2. If the user's response to the aforementioned inquiry is to deny the commencement of the parking assistance, The parking assistance device according to claim 1, which either causes the user to newly specify the parking target position or modifies the parking target position to a position that is not affected by the obstacle.

3. The parking assistance device according to claim 1, which performs the aforementioned inquiry by displaying an inquiry screen on the display device that includes a sentence asking whether to start parking assistance for the parking target position.

4. If the user's response to the aforementioned inquiry affirms the commencement of the parking assistance, The parking assistance device according to any one of claims 1 to 3, wherein the vehicle starts moving to the parking target position in response to a user instruction to start moving the vehicle thereafter.

Images