Parking support device, parking support method, and parking support program

Abstract

To provide a parking support device, a parking support method, and a parking support program that are capable of further reducing a risk in automatic parking.SOLUTION: A parking support device includes: a feature point detector that, when automatic parking of a vehicle is to be executed, determines whether a first number of a plurality of feature points extracted from a camera image captured at a start position of the automatic parking is equal to or greater than a third number greater than a second number of a plurality of feature points extracted from a camera image captured at the start position during learning traveling; and a traveling controller that controls the vehicle so that the automatic parking is not executed when the first number is equal to or greater than the third number.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to a parking assistance device, a parking assistance method, and a parking assistance program. [Background technology]

[0002] Conventionally, there is a technology for estimating a position while considering an imaging environment when a vehicle performs automatic parking. For example, Patent Document 1 discloses a position estimation system that does not perform position estimation when the number of feature amounts included in a feature amount group of a map is equal to or less than a threshold value in a position estimation process. This position estimation system can prevent the vehicle 1 from being parked by the automatic parking function based on a position with relatively poor estimation accuracy. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-133318 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, even if the number of features included in the feature group of the map is greater than the threshold, there is still a possibility of erroneous recognition of objects around the vehicle. The technology of Patent Document 1 does not take such cases into consideration, so automatic parking is performed while the possibility of erroneous recognition remains.

[0005] The present disclosure is intended to solve the above-mentioned problems, and aims to provide a parking assistance device, a parking assistance method, and a parking assistance program that further reduce risks in automatic parking. [Means for solving the problem]

[0006] In order to solve the above problems, one aspect of a parking assistance device according to the present disclosure includes a feature point detection unit that determines whether a first number of a plurality of feature points extracted from a camera image taken at a start position for automatic parking when a vehicle is automatically parked is equal to or greater than a third number that is greater than a second number of a plurality of feature points extracted from a camera image taken at the start position during learning driving, and a driving control unit that controls the vehicle so as not to perform the automatic parking if the first number is equal to or greater than the third number.

[0007] Furthermore, one aspect of the parking assistance method according to the present disclosure includes a feature point detection step of determining whether a first number of a plurality of feature points extracted from a camera image taken at a start position for automatic parking when a vehicle is automatically parked is equal to or greater than a third number that is greater than a second number of a plurality of feature points extracted from a camera image taken at the start position during learning driving, and a driving control step of controlling the vehicle so as not to perform the automatic parking if the first number is equal to or greater than the third number.

[0008] In addition, one aspect of the parking assistance program according to the present disclosure causes a computer to execute a feature point detection step of determining whether a first number of a plurality of feature points extracted from a camera image taken at a start position for automatic parking when the vehicle is automatically parked is equal to or greater than a third number that is greater than a second number of a plurality of feature points extracted from a camera image taken at the start position during learning driving, and a driving control step of controlling the vehicle so as not to perform the automatic parking if the first number is equal to or greater than the third number. Effect of the Invention

[0009] According to the present disclosure, it is possible to reduce risks in automatic parking. [Brief description of the drawings]

[0010] [Figure 1] 1 is a diagram showing a vehicle to which the parking assistance device according to the present embodiment can be applied. [Diagram 2]1 is a block diagram showing a parking assistance device according to an embodiment of the present invention; [Diagram 3] 1 is a diagram showing a hardware configuration of a parking assistance device according to an embodiment of the present invention; [Figure 4] 4 is a flowchart of a learning run in the present embodiment. [Diagram 5] FIG. 2 is a diagram showing characteristic points of a parking space in the present embodiment. [Figure 6] 4 is a flowchart of automatic parking in the present embodiment. [Figure 7] FIG. 11 is a diagram showing a comparison of feature points according to the present embodiment. [Figure 8] FIG. 2 is a diagram showing characteristic points of a parking space in the present embodiment. [Figure 9] FIG. 11 is a diagram showing a comparison of feature points according to the present embodiment. [Figure 10] FIG. 2 is a diagram showing characteristic points of a parking space in the present embodiment. [Figure 11] FIG. 11 is a diagram showing a comparison of feature points according to the present embodiment. [Figure 12] FIG. 2 is a diagram showing characteristic points of a parking space in the present embodiment. [Figure 13] FIG. 11 is a diagram showing a comparison of feature points according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that each embodiment described below shows a specific example of the present disclosure. Therefore, each component, the arrangement position and connection form of each component, each step and the order of each step shown in the following embodiments are merely examples and are not intended to limit the present disclosure. In addition, among the components in the following embodiments, components that are not described in the independent claims will be described as optional components.

[0012] In addition, each drawing is a schematic diagram and is not necessarily a precise illustration. In each drawing, the same reference numerals are used for substantially the same configurations, and duplicated explanations are omitted or simplified.

[0013] 1 is a diagram showing a vehicle to which the parking assistance device of this embodiment can be applied. In the learning type automatic parking, the vehicle 1 is manually parked during learning driving and the parking start position and parking route are stored in the vehicle 1, and during automatic parking, the vehicle 1 is automatically driven from the parking start position along the parking route to park the vehicle 1.

[0014] More specifically, when a vehicle 1 equipped with the parking assistance device 100 is learning to park manually, it extracts feature points contained in the camera image captured by the camera 2 and generates a map including information on the feature points and information on the parking route, and when automatically parking, it uses the map to estimate the position and attitude of the vehicle 1 while driving.

[0015] Cameras 2 are provided at four locations on the front, rear, left and right sides of the body of vehicle 1. Each camera 2 is equipped with a fisheye lens and has a horizontal viewing range of 180 degrees or more (see dashed lines). Parking assistance device 100 identifies the position of an object captured in the camera image and the direction seen from vehicle 1 based on the extracted feature points.

[0016] 2 is a block diagram showing a parking assistance device 100 according to the present embodiment. The parking assistance device 100 includes a state management unit 110, a map generation unit 120, an image processing unit 130, a feature point detection unit 140, a position estimation unit 150, a driving control unit 160, a storage unit 170, and a notification unit 180. A camera 2, an operation device 10, an HMI (Human Machine Interface) device 20, and a vehicle control device 30 are also connected to the parking assistance device 100.

[0017] The camera 2 outputs a camera image captured around the vehicle 1. The image processing unit 130 receives the camera image and performs operations such as generating a display image to be displayed on the HMI device 20. The display image is output from the notification unit 180 to the HMI device 20.

[0018] The notification unit 180 displays a message on the HMI device 20 and outputs a voice message from a speaker (not shown). The state management unit 110 accepts a user's operation via the operation device 10 and controls functions of the parking assistance device 100 in response to the user's operation. The operation device 10 is, for example, a touch panel of the navigation device 40.

[0019] The state management unit 110 receives position information from the main body (not shown) of the navigation device 40. When performing learning driving, the state management unit 110 adds the position information to a map and records it in the storage unit 170, and when performing automatic parking, it compares the position (GPS) information of the navigation device 40 with the position information added to the map to select the map to be used.

[0020] The image processing unit 130 generates a detection image in addition to a display image. The detection image is, for example, an image in which changes in brightness and color are emphasized. The feature point detection unit 140 extracts information on feature points from the detection image. For example, feature points are points located at corners or edges, not points on the faces or sides of an object captured in a camera image. The feature point information includes information on the color and shape of the object captured in the camera image, and information on the position of the feature point on the camera image.

[0021] When starting a learning drive, the map generating unit 120 stores the feature points detected at the start position of the learning drive in the storage unit 170. Information on the feature points at the start position is necessary to identify the position and attitude of the vehicle 1 when starting automatic parking.

[0022] When the vehicle 1 starts traveling, the map generating unit 120 tracks the positions of the feature points on the camera image. For example, as shown in Fig. 1, the map generating unit 120 compares feature points captured in a camera image at time A with feature points captured in a camera image at a later time B, and detects pairs of feature points whose feature point positions, colors, and shape information match. This is called tracking.

[0023] The change in the position of the paired feature points on the camera image occurs due to the movement and attitude change of vehicle 1 between time A and time B, so the three-dimensional coordinates of the feature points can be identified using the principles of triangulation.

[0024] The map generating unit 120 further generates a map including information on characteristic points and information on parking routes. The storage unit 170 stores the map information.

[0025] When performing automatic parking, the feature point detection unit 140 compares the information of the feature points detected by the feature point detection unit 140 (information on the color and shape of the object captured in the camera image and information on the position of the feature points on the camera image) with the information of the feature points registered on the map, and detects feature points that match the map.

[0026] Since the position of a feature point on a camera image indicates the angle at which the feature point is visible from the vehicle 1, the position estimation unit 150 can estimate the position and attitude of the vehicle 1 from the feature point on the map and the angle of the feature point on the camera image that matches it. This process is called self-position estimation, and the process of identifying the matching feature point is called matching processing. Self-position estimation and matching processing can be performed using existing methods, so detailed explanations will be omitted.

[0027] The driving control unit 160 communicates with the vehicle control device 30 during learning driving, acquires information on the number of rotations of the wheels and the steering angle, and calculates the amount of movement and the direction of movement of the vehicle. The number of rotations of the wheels and the steering angle, or the amount of movement and the direction of movement of the vehicle are registered on the map as parking route information.

[0028] The driving control unit 160 outputs an instruction signal to the vehicle control device 30 to reproduce the steering angle during learning driving when performing automatic parking. In other words, the parking route during automatic parking is a reproduction of the parking route during learning driving.

[0029] If the position or attitude of the vehicle deviates from that of the learning drive, the driving control unit 160 first controls the steering angle to change the course in a direction that intersects with the path used for learning drive, and when the vehicle overlaps with the path used for learning drive, controls the steering angle so that the vehicle's attitude matches that of the learning drive. In other words, the driving control unit 160 estimates the position and attitude of the vehicle, and feedback controls the steering angle so that the vehicle's course follows the path used for learning drive.

[0030] Fig. 3 is a diagram showing a hardware configuration of parking assistance device 100 in this embodiment. The functions of parking assistance device 100 may be implemented in the hardware shown in Fig. 3. Parking assistance device 100 may be a computer including CPU 101, ROM 102, RAM 103, I / O (input / output interface) 104, and IMP (Image Processor) 105, with each element being connected via a bus.

[0031] Parking assistance device 100 may accommodate multiple elements on one chip, or one element may be composed of multiple chips. A bus may not be a single one, but may be a combination of multiple types of buses. For example, CPU 101, ROM 102, RAM 103, and IMP 105 may be accommodated on one chip and connected by a parallel bus, and I / O 104 may be composed of multiple chips and connected to the chip accommodating CPU 101 via a serial bus.

[0032] The CPU 101 controls the entire parking assistance device 100. The functions of each unit of the parking assistance device 100 may be implemented in the form of a program executed by the CPU 101. The ROM 102 and the RAM 103 correspond to the storage unit 170. The ROM 102 is a non-volatile memory, and the RAM 103 is a volatile memory used for temporary storage as a working area for the CPU 101. For example, the RAM 103 temporarily stores camera images such as display images and detection images, information on detected feature points, and the like.

[0033] The IMP 105 is a processor that is specialized for image processing and parallel processing and has improved processing performance, and the processing of the image processing unit 130, the feature point detection unit 140, the position estimation unit 150, and the like may be executed by the IMP 105.

[0034] 4 is a flowchart of the learning drive in this embodiment. Information on the automatic parking route traveled by the driver is registered in the storage unit 170 as map information in association with the GPS information and characteristic points of the route registration start position.

[0035] In the learning run, the map generating unit 120 registers the GPS coordinates of the starting position in the storage unit 170 (step S1), and further registers information on detected characteristic points (step S2).

[0036] At the start position of the learning drive in steps S1 and S2, the map generating unit 120 acquires the GPS coordinates at the start position from the navigation device 40, adds the information to the map information, and registers it in the storage unit 170 so that the map to be used can be specified from the GPS coordinates during automatic parking. In addition, the map generating unit 120 registers information on the characteristic points detected at the start position in the storage unit 170 so as to be used for initial self-position estimation for automatic parking.

[0037] The map generating unit 120 tracks the characteristic points and identifies their coordinates, and registers this information together with route information such as steering angle and amount of movement as map information (step S3).

[0038] In step S3, the map generating unit 120 tracks characteristic points between the start position and the parking position, collects the coordinates of the characteristic points that have been tracked, adds the information to the map, and registers it in the storage unit 170. The map generating unit 120 also registers in the storage unit 170 route information such as the steering angle, vehicle speed, gear position, moving direction, and moving distance, as well as obstacle detection information.

[0039] Thereafter, the map generating unit 120 determines whether the learning run is complete (step S4). If the learning run is not complete (step S4, NO), the process proceeds to step S3 and subsequent steps. If the learning run is complete (step S4, YES), the power supply of the vehicle 1 is turned off (step S5), and the learning run ends.

[0040] 5 is a diagram showing feature points 202, 212 of a parking space 201 extracted from a camera image. The diagram on the left shows an example of feature points 202 extracted from a camera image 200 captured during a learning drive at night, and the diagram on the right shows an example of feature points 212 extracted from a camera image 210 captured during automatic parking in the daytime.

[0041] Camera images 200 and 210 show a parking space 201 in front of a house, and feature points 202 registered on an automatic parking route are shown. A total of four feature points 202 are extracted from camera image 200 taken at night. A total of 13 feature points 212 are displayed from camera image 210 taken during the day. Note that line 203 is a virtual line for grasping the positions of feature points 202 and 212.

[0042] As night turns into day and the illuminance increases, the number of feature points 212 becomes greater than the number of feature points 202.

[0043] Next, automatic parking that uses the registered map and route information to reproduce a route will be described. Fig. 6 is a flowchart of automatic parking in this embodiment.

[0044] First, the state management unit 110 uses the GPS information to determine whether the vehicle 1 has arrived at a registered position that is registered as the starting position of the learning journey (step S11).

[0045] In this determination, the status management unit 110 may determine whether or not the vehicle 1 has returned home based on whether the driving direction during the learning drive matches the driving direction of the vehicle 1, or whether the vehicle 1 has approached the area of ​​the driving route during the learning drive from an area other than the area of ​​the driving route during the learning drive, and may execute the processing of step S11 if the vehicle 1 has returned home.

[0046] If the vehicle 1 has not arrived at the registered position (step S11, NO), the process of step S11 is repeated.

[0047] When the vehicle 1 arrives at the registered position (step S11, YES), the state management unit 110 determines whether or not the registration of the map information by the learning drive was successful (step S12).

[0048] The map information used when performing automatic parking is generated by extracting multiple feature points from camera images taken during learning driving at the start position of automatic parking, and executing a series of processes up to registering the extracted feature points in memory unit 170. However, since such map information generation process may be completed after the ignition of vehicle 1 is turned off, it is possible that the map information may not be generated due to some reason, resulting in failure.

[0049] If the process of generating the map information is completed to the end, the state management unit 110 determines that the registration of the map information is successful and turns on the completion flag. If the process of generating the map information is not completed to the end, the state management unit 110 determines that the registration of the map information is unsuccessful and turns off the completion flag.

[0050] As will be explained later, if the registration of map information fails during learning driving, automatic parking is likely to fail, so the execution of automatic parking is canceled.

[0051] If the registration of the map information during the learning drive is successful (step S12, YES), the feature point detection unit 140 detects feature points in a camera image taken around the vehicle 1 (step S13). In addition, the notification unit 180 may notify the driver that the registration of the map information during the learning drive is successful.

[0052] Next, the feature point detection unit 140 compares the feature points around the vehicle 1 detected in the processing of step S13 with the feature points registered as map information, and determines whether the first number of feature points around the vehicle 1 detected in the processing of step S13 is equal to or greater than a third number that is greater than the second number of feature points registered as map information (step S14).

[0053] The third number is, for example, twice the second number. Here, the third number is twice the second number, but is not limited to this, and may be a number obtained by multiplying the second number by a value greater than 1. The third number may also be a number obtained by adding a predetermined number to the second number.

[0054] Then, if the first number is equal to or greater than the third number (step S15, YES), the driving control unit 160 controls the vehicle 1 so as not to perform automatic parking (step S16).

[0055] 7 is a diagram showing a comparison of feature points 202, 212 in this embodiment. The diagram on the left corresponds to image 200, and shows feature points 202 registered during a learning drive at night. The diagram on the right corresponds to image 210, and shows feature points 212 obtained during automatic parking in the daytime.

[0056] In this case, since the number of feature points 212 is more than twice the number of feature points 202, the driving control unit 160 controls the vehicle 1 not to perform automatic parking (step S16). At that time, the notification unit 180 may notify the driver that automatic parking is not possible by displaying a message such as "Automatic parking is not possible" on the HMI device 20 or outputting a voice message from a speaker (not shown).

[0057] If the number of feature points around vehicle 1 detected in the processing of step S13 becomes greater than the number of feature points registered as map information, there is a possibility that uncorresponding feature points will be mistakenly matched with each other, and there is a risk of misrecognizing objects around vehicle 1. Therefore, by not performing automatic parking, accidents due to misrecognition can be prevented.

[0058] In step S15, if the first number is not equal to or greater than the third number (step S15, NO), the driving control unit 160 controls the vehicle 1 to perform automatic parking (step S17).

[0059] In step S12, if the registration of map information through the learning drive has failed (step S12, NO), the notification unit 180 notifies the driver by displaying a message on the HMI device 20 stating "The previous registration process failed" or by outputting a voice message from a speaker (not shown) (step S18).

[0060] After that, the state management unit 110 causes the vehicle 1 to again execute the process of registering map information through the learning drive as shown in FIG. 4 (step S19).

[0061] 8 is a diagram showing feature points 242, 252 of parking space 201 in this embodiment. The diagram on the left shows an example of feature points 242 extracted from camera image 240 captured during a learning drive in the daytime, and the diagram on the right shows an example of feature points 252 extracted from camera image 250 captured during automatic parking at night.

[0062] A total of 13 feature points 242 are extracted from the camera image 240 taken during the day, and a total of 4 feature points 252 are extracted from the camera image 250 taken at night.

[0063] 9 is a diagram showing a comparison of feature points 242, 252 in this embodiment. The diagram on the left corresponds to image 240, and shows feature points 242 registered during a daytime learning drive. The diagram on the right corresponds to image 250, and shows feature points 252 obtained during automatic driving at night.

[0064] In this case, automatic parking is possible because 30% or more of the feature points 252 match the feature points 242. Note that the reference value for matching is not limited to 30%, and may be set to another value.

[0065] 10 is a diagram showing feature points 282, 292 of parking space 201 in this embodiment. The diagram on the left shows feature points 282 extracted from camera image 280 taken during a learning drive in the daytime, and the diagram on the right shows feature points 292 extracted from camera image 290 taken during automatic parking in the daytime.

[0066] A total of 13 feature points 282 are extracted from the camera image 280. Also, a total of 13 feature points 292 are extracted from the camera image 290.

[0067] 11 is a diagram showing a comparison of feature points 282, 292 in this embodiment. The diagram on the left corresponds to image 280, and shows feature points 282 registered during a daytime learning drive. The diagram on the right corresponds to image 290, and shows feature points 292 obtained during daytime automatic parking.

[0068] In this case too, since 30% or more of the feature points 292 match the feature points 282, automatic parking is possible.

[0069] 12 is a diagram showing feature points 322 of parking space 201 in this embodiment. The diagram on the left shows feature points 322 extracted from a camera image 320 taken during a learning drive in the daytime, and the diagram on the right shows a camera image 330 taken during automatic parking in pitch black at night.

[0070] A total of 13 feature points 322 are extracted from the camera image 320. No feature points can be extracted from the camera image 330.

[0071] 13 is a diagram showing a comparison of feature points 322 in this embodiment. The diagram on the left corresponds to image 320, and shows feature points 322 registered during a daytime learning drive. The diagram on the right corresponds to image 330, and does not show feature points obtained during automatic parking at night.

[0072] In this case, it cannot be said that the feature points obtained during automatic parking at night match the feature points 322 by 30% or more, so automatic parking is deemed impossible.

[0073] Although the embodiments have been described above, the present disclosure is not limited to the above-described embodiments.

[0074] For example, in the above embodiment, each component may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory.

[0075] Furthermore, the general or specific aspects of the present invention may be realized as an apparatus, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or may be realized as any combination of an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0076] In addition, the present disclosure also includes forms obtained by applying various modifications to each embodiment that a person skilled in the art may conceive, or forms realized by arbitrarily combining the components and functions of each embodiment within the scope that does not deviate from the spirit of the present disclosure. [Industrial Applicability]

[0077] The present disclosure can be used in a parking assistance device, a parking assistance method, and a parking assistance program. [Explanation of symbols]

[0078] 1 vehicle 2 Camera 10 Operating device 20 HMI device 30 Vehicle control device 40 Navigation devices 100 Parking assistance device 101 CPU 102 ROM 103 RAM 104 I / O 105 IMP 110 Status Management Unit 120 Map Generation Unit 130 Image processing section 140 Feature point detection unit 150 Position estimation part 160 Driving control unit 170 Storage section 180 Notification Department 200 images 201 Parking Space 202,212,242,252,282,292,322 feature points

Claims

1. a feature point detection unit that determines whether a first number of a plurality of feature points extracted from a camera image captured at a start position of automatic parking during automatic parking of a vehicle is equal to or greater than a third number that is greater than a second number of a plurality of feature points extracted from a camera image captured at the start position during learning driving; A driving control unit that controls the vehicle so as not to perform the automatic parking when the first number is equal to or greater than the third number; A parking assistance device comprising:

2. 2. The parking assistance device according to claim 1, further comprising an alarm unit that notifies a user of a failure when a series of processes from extracting a plurality of feature points from a camera image taken during a learning drive at the start position to registering the extracted feature points fails.

3. a feature point detection step of determining whether a first number of a plurality of feature points extracted from a camera image taken at a start position of automatic parking during automatic parking of the vehicle is equal to or greater than a third number that is greater than a second number of a plurality of feature points extracted from a camera image taken at the start position during learning driving; a driving control step of controlling the vehicle so as not to perform the automatic parking when the first number is equal to or greater than the third number; A parking assistance method comprising:

4. 4. The parking assistance method according to claim 3, further comprising a notification step of notifying a failure when a series of processes from extracting a plurality of feature points from a camera image taken during a learning drive at the start position to registering the extracted feature points fails.

5. a feature point detection step of determining whether a first number of a plurality of feature points extracted from a camera image taken at a start position of automatic parking during automatic parking of the vehicle is equal to or greater than a third number that is greater than a second number of a plurality of feature points extracted from a camera image taken at the start position during learning driving; a driving control step of controlling the vehicle so as not to perform the automatic parking when the first number is equal to or greater than the third number; A parking assistance program that causes a computer to execute the above.

Images