Parking assistance method and parking assistance device

Abstract

In the parking assist method, a color image, i.e., a surrounding image, obtained by capturing the surroundings of the host vehicle is acquired (S1), the surrounding image is converted into a first multi-gray scale image in a manner that increases the luminance difference between a region in which the proportion of green components is small and a region in which the proportion of green components is large (S2-S6), a feature point is extracted from the first multi-gray scale image (S7), and a relative position of the host vehicle with respect to a target parking position is calculated based on the extracted feature point (S32).

Description

TECHNICAL FIELD

[0001] The present application relates to a parking assist method and a parking assist device. BACKGROUND

[0002] In Patent Literature 1, a driving control device is described which extracts feature points from an image that has been captured around a target parking position before and stores the feature points, calculates a relative position of the target parking position with respect to the host vehicle based on the position of the stored object and the position of the object extracted from an image that has been captured around the host vehicle at the time of automatic parking, and causes the host vehicle to automatically move to the target parking position based on the calculated relative position.

[0003] PRIOR ART DOCUMENTS

[0004] PATENT LITERATURE

[0005] Patent Literature 1: Japanese Patent Application Publication No. 2017-138664 SUMMARY

[0006] PROBLEMS TO BE SOLVED BY THE INVENTION

[0007] When the image captured around the target parking position or the image captured around the host vehicle contains a vegetation area, there is a possibility that the calculation accuracy of the target parking position decreases because feature points such as blurred outlines in the vegetation area are extracted.

[0008] An object of the present application is to improve the calculation accuracy of the target parking position in the case where a captured image contains a vegetation area in a parking assist in which the host vehicle is assisted to park at a target parking position based on feature points extracted from the captured image.

[0009] SOLUTION TO PROBLEM

[0010] According to one embodiment of the present invention, a parking assistance method for assisting a host vehicle to park at a target parking position is provided. The method includes: extracting, in advance, a feature point around the target parking position from an image obtained by capturing an environment of the host vehicle and storing the feature point as a learned feature point in a storage device; capturing the environment of the host vehicle to obtain an image while the host vehicle is moving to the target parking position; extracting a feature point around the host vehicle from the image of the environment of the host vehicle as an environment feature point; calculating a relative position of the host vehicle with respect to the target parking position based on a relative positional relationship between the learned feature point and the target parking position and a relative positional relationship between the environment feature point and the host vehicle; calculating a target travel trajectory from a current position of the host vehicle to the target parking position based on the calculated relative position of the host vehicle with respect to the target parking position, and assisting the host vehicle to move along the target travel trajectory. When extracting a target feature point from at least one of the learned feature point and the environment feature point, a color image obtained by capturing the environment of the host vehicle, i.e., an environment image, is converted into a first multi-grayscale image in such a manner that a luminance difference between a region having a small proportion of green color component and a region having a large proportion of green color component in the environment image is increased, and the target feature point is extracted from the first multi-grayscale image.

[0011] Effects of the Invention

[0012] According to the present invention, in a parking assistance for assisting a host vehicle to park at a target parking position based on a feature point extracted from a captured image, the calculation accuracy of the target parking position can be improved in a case where a vegetation region is included in the captured image. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 is a diagram showing an outline of a structure of a parking assistance device.

[0014] Figure 2A is a diagram for explaining one example of a process of storing a learned feature point.

[0015] Figure 2B is a diagram for explaining one example of a process when a parking assistance is performed.

[0016] Figure 3 is a block diagram showing one example of a functional structure of a controller of Figure 1

[0017] Figure 4 is a diagram showing one example of a grayscale image generated from an environment image when an illuminance of an environment of a host vehicle is equal to or higher than a predetermined threshold value.

[0018] Figure 5 is a diagram showing one example of a first intermediate image generated from an environment image when an illuminance of an environment of a host vehicle is equal to or higher than a predetermined threshold value.

[0019] ​Figure 6 is a drawing showing one example of a second intermediate image.

[0020] Figure 7 is a drawing showing one example of a multi-tone image obtained by the second method.

[0021] Figure 8 is a drawing showing one example of a multi-tone image obtained by the third method.

[0022] Figure 9A is a drawing showing one example of a gray scale image generated from a surrounding image at the time of a threshold value defined by insufficient illuminance around the host vehicle and a partially enlarged image.

[0023] Figure 9B is a drawing showing a gray scale image of Figure 4 and a partially enlarged image.

[0024] Figure 9C is a drawing showing a multi-tone image of Figure 8 and a partially enlarged image.

[0025] Figure 10A is an explanatory drawing of a first example of a process of storing learned feature points.

[0026] Figure 10B is an explanatory drawing of a first example of a multi-tone image generation process.

[0027] Figure 10C is an explanatory drawing of a second example of a multi-tone image generation process.

[0028] Figure 10D is an explanatory drawing of a third example of a multi-tone image generation process.

[0029] Figure 10E is an explanatory drawing of a second example of a process of storing learned feature points.

[0030] Figure 11A is an explanatory drawing of a first example of a process when a parking assistance is implemented.

[0031] Figure 11B is an explanatory drawing of a second example of a process when a parking assistance is implemented. DETAILED DESCRIPTION

[0032] (CONSTITUTION)

[0033] Referring to Figure 1The host vehicle 1 has a parking assist device 10 that assists the host vehicle 1 in parking at a target parking position. The parking assist device 10 assists the host vehicle 1 in traveling along a target travel trajectory from a current position to the target parking position. For example, automatic driving of the host vehicle 1 (i.e., control of all or a part of the steering angle, driving force, and braking force of the host vehicle, to automatically implement all or a part of the control of the host vehicle 1 to travel along the target travel trajectory) can be performed to cause the host vehicle 1 to travel along the target travel trajectory to the target parking position. The host vehicle 1 can also be assisted in parking by displaying the target travel trajectory and the current position of the host vehicle 1 on a display device that can be visually confirmed by an occupant of the host vehicle 1.

[0034] The positioning device 11 measures the current position of the host vehicle 1. The positioning device 11 has, for example, a global navigation satellite system (GNSS) receiver. The human-machine interface (HMI) 12 is an interface device between the parking assist device 10 and an occupant, and has a display device, a speaker or buzzer, and an operation member. The shift switch (shift SW) 13 is a switch used by the driver or the parking assist device 10 to switch gears. The outside sensor 14 detects objects within a prescribed distance range from the host vehicle 1. The outside sensor 14 detects the relative position of an object existing around the host vehicle 1 to the host vehicle 1, the distance between the host vehicle 1 and the object, the direction in which the object exists, and the like, to detect the environment around the host vehicle 1. The outside sensor 14 can include, for example, a camera that captures the environment around the host vehicle 1. In order to correspond to parking assistance of the host vehicle 1 in an environment in which the illuminance around the host vehicle 1 is less than a prescribed threshold value (e.g., at night), as the camera included in the outside sensor 14, a day and night camera that can capture both in the visible light region and in the infrared region can be used. Alternatively, a day camera that can capture in the visible light region and a camera that can capture in the infrared region can be used. In the present embodiment, an example in which a day and night camera is used will be described. Hereinafter, the camera of the outside sensor 14 will be simply referred to as a "camera". The outside sensor 14 can also include a range finder such as a laser range finder, a radar, a LiDAR (Light Detection and Ranging), or the like. The vehicle sensor 15 detects various information (vehicle information) of the host vehicle 1. The vehicle sensor 15 can include, for example, a vehicle speed sensor, a wheel speed sensor, a three-axis acceleration sensor (G sensor), a steering angle sensor, a steering angle sensor, a gyro sensor, and a yaw rate sensor.

[0035] The controller 16 is an electronic control unit that performs the parking assist control of the host vehicle 1. The controller 16 includes a processor 20, a storage device 21, and the like as peripheral components. The storage device 21 can have a semiconductor storage device, a magnetic storage device, an optical storage device, and the like. The functions of the controller 16 are realized, for example, by the processor 20 executing a computer program stored in the storage device 21. The steering actuator 18a controls the steering direction and the steering amount of the steering mechanism in accordance with a control signal of the controller 16. The accelerator actuator 18b controls the accelerator opening of the drive device (engine, drive motor) in accordance with a control signal of the controller 16. The brake actuator 18c operates the brake device in accordance with a control signal of the controller 16.

[0036] The infrared projector 19 irradiates infrared rays to the surroundings of the host vehicle 1 in accordance with a control signal of the controller 16 when assisting the parking of the host vehicle 1 in an environment where the illuminance of the surroundings of the host vehicle 1 is less than a prescribed threshold value. The infrared projector can be, for example, an infrared light emitting diode that is provided on the left and right sides of the host vehicle 1 and emits infrared rays obliquely downward to the left and right sides of the host vehicle 1, respectively, and irradiates the road surface around the left and right sides of the host vehicle 1 with the infrared rays.

[0037] Next, the parking assist control performed by the parking assist device 10 will be described. Referring to FIG. 2, the parking assist device 10 includes the controller 16, the camera 17, the infrared projector 19, the HMI 12, and the like. Figure 2A In the parking assist using the parking assist device 10, feature points are extracted from an image that captures the surroundings of a position at which the host vehicle 1 is parked, that is, a target parking position 30, and are stored in the storage device 21 in advance. Hereinafter, the feature points stored in the storage device 21 will be referred to as "learned feature points". In the parking assist using the parking assist device 10, the learned feature points are used to determine whether the host vehicle 1 is parked at the target parking position 30. Figure 2A The circular icons in FIG. 1 represent the learned feature points. For example, the parking assist device 10 extracts the feature points around the target parking position 30 from a surrounding image obtained by capturing the surroundings of the host vehicle 1 when the host vehicle 1 is located near the target parking position 30 (for example, when parked at the target parking position 30 by manual driving). For example, edge points in which the brightness of pixels adjacent to edges or corners of objects such as road markings, road boundaries, and obstacles on a captured image obtained by capturing using the camera changes by a prescribed amount or more, or points having a feature in shape are detected as the feature points. When the learned feature points are stored in the storage device 21, the driver, for example, operates a "parking position learning switch" prepared as an operating member of the HMI 12. Note that in a case where the illuminance of the surroundings of the host vehicle 1 is less than a prescribed threshold value, the learned feature points are extracted from a captured image obtained by capturing in a state where infrared rays are irradiated to the surroundings of the host vehicle 1 from the infrared projector 19.

[0038] The parking assist device 10 stores the relative positional relationship between the learned feature point and the target parking position 30. For example, the driver can input information that the host vehicle 1 is located at the target parking position 30 using the HMI 12. The parking assist device 10 can also calculate the relative positional relationship between the learned feature point and the target parking position 30 based on the position of the learned feature point detected when the host vehicle 1 is located at the target parking position 30. For example, the parking assist device 10 can also store the coordinates of the learned feature point and the target parking position 30 in a coordinate system (hereinafter referred to as "map coordinate system") with a fixed point as a reference point. In this case, the current position in the map coordinate system measured by the positioning device 11 when the host vehicle 1 is located at the target parking position 30 can also be stored as the target parking position 30. Alternatively, instead of the map coordinate system, the relative positions of the target parking position 30 with respect to each of the learned feature points can be stored.

[0039] Figure 2B This is a diagram illustrating one example of the process when the parking assist is implemented. The parking assist device 10 implements the parking assist of the host vehicle 1 when the host vehicle 1 is located near the target parking position 30. For example, the parking assist device 10 determines whether the driver has performed a shift operation for reversing when the host vehicle 1 is located near the target parking position 30. The mark 31 is a reversing position. The parking assist device 10 can determine that the shift operation for reversing has been performed and start the parking assist when the gear is shifted from the forward gear (D range) to the reverse gear (R range) or from the R range to the D range. The parking assist can also be started when the driver operates the "parking assist start switch" prepared in the HMI 12 when the host vehicle 1 is located at the position 33 near the target parking position 30.

[0040] The parking assist device 10 extracts a feature point around the host vehicle 1 from a surrounding image obtained by photographing the surroundings of the host vehicle 1 using the camera. Hereinafter, the feature point around the host vehicle 1 extracted when the parking assist is implemented will be referred to as a "surrounding feature point". In the example shown in FIG. 6, the surrounding feature point is a triangular icon. Figure 2B The triangular icon in FIG. 6 indicates the surrounding feature point.

[0041] Note that in a case where the illuminance around the host vehicle 1 is less than a prescribed threshold value when the parking assist is implemented, the surrounding feature point is extracted from a photographed image obtained by photographing using the camera in a state where infrared light is irradiated from the infrared projector 19 to the surroundings of the host vehicle 1.

[0042] The parking assist device 10 collates (matches) the learned feature point stored in the storage device 21 and the surrounding feature point and associates the same feature points with each other.

[0043] The parking assist device 10 calculates the relative position of the host vehicle 1 with respect to the target parking position 30 based on the relative positional relationship between the surrounding feature points detected at the time of implementing the parking assist and the learned feature points associated with the surrounding feature points, and the relative positional relationship between the learned feature points and the target parking position 30. For example, the parking assist device 10 can calculate the position of the target parking position 30 in a coordinate system (hereinafter referred to as "vehicle coordinate system") with the current position of the host vehicle 1 as a reference. For example, in the case where the coordinates of the learned feature points and the target parking position 30 in a map coordinate system are stored in the storage device 21, the coordinates of the target parking position 30 in the vehicle coordinate system can be converted from the coordinates of the target parking position 30 in the map coordinate system based on the positions of the surrounding feature points detected at the time of implementing the parking assist and the positions of the learned feature points in the map coordinate system. Alternatively, the current position of the host vehicle 1 in the map coordinate system can be calculated based on the positions of the surrounding feature points detected at the time of implementing the parking assist and the positions of the learned feature points in the map coordinate system, and the relative position of the host vehicle 1 with respect to the target parking position 30 can be calculated from the difference between the coordinates of the host vehicle 1 and the coordinates of the target parking position 30 in the map coordinate system. The parking assist device 10 calculates a target travel trajectory from the current position of the host vehicle 1 to the target parking position 30 based on the relative position of the host vehicle 1 with respect to the target parking position 30. For example, in the case where the position of the host vehicle 1 at the time of starting the parking assist is the turnaround position 31, a trajectory 32 from the turnaround position 31 to the target parking position 30 is calculated. Alternatively, for example, in the case where the position of the host vehicle 1 at the time of starting the parking assist is a position 33 near the target parking position 30, a trajectory 34 from the position 33 to the turnaround position 31 and a trajectory 32 from the turnaround position 31 to the target parking position 30 are calculated. The parking assist device 10 implements parking assist control of the host vehicle 1 based on the calculated target travel trajectory.

[0044] The functional structure of the controller 16 is as shown in FIG. 4. Figure 3 The human-machine interface control section (HMI control section) 40 outputs a map generation instruction to the map generation section 45 to store the learned feature points in the storage device 21 when the parking position learning switch is operated. The HMI control section 40 determines whether or not the driver has performed a shift operation for the turnaround, and whether or not the parking assist start switch has been operated, and outputs the determination results to the parking assist control section 41. The parking assist control section 41 determines whether or not the host vehicle 1 is located near the target parking position 30 stored in the storage device 21. For example, it is determined whether or not the distance between the host vehicle 1 and the target parking position 30 is equal to or less than a predetermined distance. When the host vehicle 1 is located near the target parking position 30 and the parking assist start switch is detected to have been operated or the shift operation for the turnaround is detected to have been performed, the parking assist control section 41 sets the target parking position 30 stored in the storage device 21 as the target parking position 30 of the control, and starts the parking assist control.

[0045] When the parking assist control is started, the parking assist control section 41 outputs a parking position calculation instruction to calculate the position of the target parking position 30 in the vehicle coordinate system to the collation section 47. In addition, a travel trajectory calculation instruction to calculate a target travel trajectory and a target vehicle speed change pattern in which the host vehicle 1 travels along the target travel trajectory is output to the target trajectory generation section 48. The target trajectory generation section 48 calculates the target travel trajectory and the target vehicle speed change pattern from the current position of the host vehicle 1 to the target parking position 30, and outputs them to the parking assist control section 41. The calculation of the target travel trajectory can be performed by a known method employed in an automatic parking device. For example, it can be calculated by connecting from the current position of the host vehicle 1 to the target parking position 30 via the turnaround position 31 with a clothoid curve. In addition, for example, the target vehicle speed change pattern can be a vehicle speed change pattern in which, after accelerating from the current position of the host vehicle 1 to a predetermined set speed, the vehicle is decelerated near the turnaround position 31, parked at the turnaround position 31, then accelerated from the turnaround position 31 to the set speed, and decelerated near the target parking position 30, and then parked at the target parking position 30.

[0046] The parking assist control section 41 outputs information of the target travel trajectory and the current position of the host vehicle 1 to the HMI control section 40. In the case where the target travel trajectory includes a turnaround, information of the turnaround position is output to the HMI control section 40. The HMI control section 40 displays the target travel trajectory, the current position of the host vehicle 1, and the turnaround position on the HMI 12.

[0047] In addition, the parking assist control section 41 outputs a steering control instruction to perform steering control so that the host vehicle 1 travels along the calculated target travel trajectory to the steering control section 49. In addition, a vehicle speed control instruction to control the vehicle speed of the host vehicle 1 in accordance with the calculated target vehicle speed change pattern is output to the vehicle speed control section 50. The image conversion section 42 converts the photographed image of the camera into an overhead image (panoramic monitoring image) viewed from a virtual viewpoint directly above the host vehicle 1. The image conversion section 42 converts the photographed image into an overhead image at a predetermined interval (for example, every time the host vehicle 1 travels a predetermined distance (for example, 50 cm) or a predetermined time (for example, one second)), accumulates the converted overhead images along the travel route of the host vehicle 1, and thereby generates the surrounding image shown in FIG. 6. Figure 2A and Figure 2B The self-position calculation section 43 calculates the current position of the host vehicle 1 on the map coordinate system by a dead reckoning method based on the vehicle information output from the vehicle sensor 15.

[0048] In a case where the learned feature points are stored in the storage 21, the feature point detection section 44 detects the learned feature points and their image feature amounts from the surrounding image output from the image conversion section 42. The surrounding feature points and their image feature amounts are detected when the parking assistance control is implemented. Hereinafter, there is a case where the learned feature points and the surrounding feature points are collectively referred to as "feature points". The feature point detection section 44 detects the feature points and their image feature amounts from the surrounding image output from the image conversion section 42. The detection of the feature points or the calculation of the image feature amounts can be performed, for example, by using a method such as SIFT, SURF, ORB, BRIAK, KAZE, AKAZE, or the like.

[0049] The surrounding image of the host vehicle 1 can contain a vegetation area (i.e., an area where plants are cultivated). In the vegetation area, a difference in brightness occurs between a portion where a plant is irradiated with light and a shadow portion where light is not irradiated, and a feature point of a blurred contour can be extracted in a small size. Since the shadow of a plant leaf varies due to a light source (e.g., the sun), the calculation accuracy of the target parking position can be reduced.

[0050] In addition, in an image captured without irradiating infrared rays in an environment (e.g., daytime) where the illuminance around the host vehicle 1 is equal to or higher than a prescribed threshold, there is a characteristic that the brightness of a vegetation area is lower than that of asphalt, whereas in an image captured with irradiating infrared rays, there is a characteristic that the brightness of the vegetation area is increased. Therefore, when either one of the learned feature points and the surrounding feature points is extracted from an image captured with irradiating infrared rays and the other one is extracted from an image captured without irradiating infrared rays, the calculation accuracy of the target parking position can be reduced.

[0051] Therefore, the parking assistance device 10 of the embodiment, in a case where the learned feature points or the surrounding feature points are extracted from an image captured without irradiating infrared rays around the host vehicle 1 in an environment where the illuminance around the host vehicle 1 is equal to or higher than a prescribed threshold, converts the surrounding image into a monochrome multi-gray scale image in such a manner that the brightness value of a region where the proportion of green components is large in the surrounding image captured around the host vehicle 1 is increased more than that of a region where the proportion of green components is small, and extracts the feature points from the converted multi-gray scale image.

[0052] For example, the surrounding image can be converted into a multi-gray scale image in such a manner that the brightness value of a region where the proportion of green components is large is saturated (i.e., becomes an upper limit value of the image brightness).

[0053] Thus, since the vegetation region of the color surrounding image is a region in which the proportion of green components is large, the luminance value of the region in which the proportion of green components is large is increased, whereby the luminance difference between the portion in which the plant is illuminated with light and the shadow portion in which the plant is not illuminated with light can be reduced. As a result, it is difficult to extract a fuzzy contour feature point in the vegetation region, so the calculation accuracy of the target parking position can be improved. In addition, the luminance of the vegetation region can be made higher than that of the asphalt region in the image captured without irradiation of infrared rays. Thus, the luminance characteristics of the vegetation region and the asphalt region are similar between the image captured without irradiation of infrared rays and the image captured with irradiation of infrared rays, so the calculation accuracy of the target parking position can be improved.

[0054] Next, the method of detecting a feature point by the feature point detection section 44 will be described in detail. In an environment in which the illuminance around the host vehicle 1 is equal to or higher than a predetermined threshold value, the feature point detection section 44 extracts a feature point from the surrounding image captured without irradiation of infrared rays.

[0055] The feature point detection section 44 converts the surrounding image into a multi-gradation image in such a manner that the luminance value of a region in which the proportion of green components is large in the color surrounding image is increased more than that of a region in which the proportion of green components is small, in order to reduce the luminance difference between the portion in which the plant is illuminated with light and the shadow portion in which the plant is not illuminated with light. This multi-gradation image is an example of the "first multi-gradation image" described in the claims.

[0056] As the first method, the surrounding image can be converted into a multi-gradation image in such a manner that the luminance value of a region in which the proportion of green components is large is increased. For example, the feature point detection section 44 converts the surrounding image into a multi-gradation image in such a manner that the luminance value of a region in which the proportion of G components (green components) Cg is larger than a predetermined value is saturated (i.e., becomes an upper limit value of image luminance) based on the ratio of at least two color components containing the G components Cg among the R components (red components) Cr, the G components (green components) Cg, and the B components (blue components) Cb of the color surrounding image. Note that, instead of being saturated, the luminance value of a region in which the proportion of Cg is larger than a predetermined value can be increased based on the ratio of two color components.

[0057] As the second method, (1) a gray scale image of the surrounding image (corresponding to a second multi-gradation image) is generated; (2) the surrounding image is converted into a first intermediate image in such a manner that the luminance value of the vegetation region is reduced; (3) a second intermediate image is generated based on the luminance difference between the gray scale image and the first intermediate image; and (4) the luminance of the second intermediate image is inverted. Through the above processes (1) to (4), a colorless multi-gradation image in which the luminance value of a region in which the proportion of green components is large is increased more than that of a region in which the proportion of green components is small is obtained.

[0058] First, the feature point detection section 44 generates a gray scale image of the surrounding image by performing a general gray scale conversion on the color image, i.e., the surrounding image.

[0059] Figure 4 is a diagram showing one example of a gray scale image. For example, the feature point detection section 44 can determine the luminance value of the gray scale image by weight-compositing the R component Cr, the G component Cg, and the B component Cb of the color surrounding image with a prescribed weight coefficient. The gray scale image is one example of the "second multi-gray scale image" described in the claims. The region Rv surrounded by the broken line represents a vegetation region. In an image captured without irradiating infrared rays toward the surroundings of the host vehicle 1, the vegetation region Rv has the characteristic that the luminance is lower than that of the surrounding asphalt region.

[0060] Next, the feature point detection section 44 converts the color surrounding image into a colorless multi-gray scale image, i.e., the first intermediate image. At this time, the feature point detection section 44 converts the surrounding image into the first intermediate image in such a manner that the luminance value of the vegetation region in which the proportion of the G component Cg is large is decreased compared to regions in which the proportion of the G component Cg is small,

[0061] Figure 5 is a diagram showing one example of the first intermediate image. For example, the feature point detection section 44 can generate the first intermediate image based on the ratio of at least two color components including the G component Cg among the R component Cr, the G component Cg, and the B component Cb of the surrounding image. For example, in the case where the maximum value of each of the R component Cr, the G component Cg, and the B component Cb is 255, the feature point detection section 44 can determine the luminance value Ck of each pixel of the first intermediate image based on the following equations (1), (2).

[0062] [Equation 1]

[0063]

[0064] Note that in the above equation, the B component Cb can be replaced with the R component Cr.

[0065] Next, the feature point detection section 44 generates a second intermediate image based on the luminance difference between the gray scale image of Figure 4 and the first intermediate image of Figure 5 For example, the feature point detection section 44 generates the second intermediate image as a difference image obtained by subtracting the luminance value of each pixel of the gray scale image from the luminance value of each pixel of the first intermediate image. Figure 6 is a diagram showing one example of the second intermediate image. By subtracting the luminance value of each pixel of the gray scale image from the luminance value of each pixel of the first intermediate image, the luminance difference of regions other than the vegetation region Rv (e.g., the asphalt region) can be decreased in the second intermediate image.

[0066] Next, the feature point detection section 44 generates a monochrome multi-gradation image in which the luminance value of a region in which the proportion of green component is large is increased more than that of a region in which the proportion of green component is small, by inverting the luminance of the second intermediate image. Figure 7 is a diagram showing one example of a multi-gradation image obtained by the second method. In the multi-gradation image of Figure 7 , the vegetation region Rv is roughly uniformly brightened (the luminance is roughly uniformly increased), and the region other than the vegetation region Rv (for example, the asphalt region) is roughly uniformly darkened (the luminance is roughly uniformly decreased).

[0067] As the third method, a multi-gradation image obtained by the second method (the luminance-inverted image of the second intermediate image) is taken as a third intermediate image, and a multi-gradation image is generated by weightedly averaging the pixel values of the third intermediate image and the pixel values of the gray scale image.

[0068] Figure 8 is a diagram showing one example of a multi-gradation image obtained by the third method. By increasing the pixel values of the third intermediate image, the luminance value of the vegetation region Rv is increased more than that of the other regions.

[0069] On the other hand, in an environment in which the illuminance around the host vehicle 1 is less than a prescribed threshold value, the feature point detection section 44 extracts feature points from the surrounding image taken when infrared rays are being irradiated with the infrared ray projector 19. In this case, the feature point detection section 44 generates a gray scale image of the surrounding image by performing ordinary gray scale conversion on the color image, that is, the surrounding image. The feature point detection section 44 extracts feature points from the gray scale image. Figure 9A is a diagram showing one example of a gray scale image of the surrounding image taken when infrared rays are being irradiated, and an enlarged image of the boundary portion between the vegetation region Rv and the asphalt region. In the image taken when infrared rays are being irradiated, the luminance of the vegetation region Rv is higher than that of the asphalt region.

[0070] Figure 9B is a diagram showing a gray scale image (the gray scale image of Figure 4 ) of the surrounding image taken without irradiation of infrared rays, and a partial enlarged image. In the image taken without irradiation of infrared rays, unlike the image taken when infrared rays are being irradiated, the luminance of the vegetation region Rv is lower than that of the asphalt region.

[0071] Figure 9C is a diagram showing a multi-gradation image (the multi-gradation image of Figure 8 ) generated in such a manner that the luminance value of the vegetation region Rv is increased more than that of the other regions, and a partial enlarged image. In the multi-gradation image of Figure 9C , the luminance of the vegetation region Rv is increased more than that of the asphalt region, and it is possible to make the luminance characteristics of the vegetation region Rv and the asphalt region similar to those of the gray scale image (the gray scale image of Figure 9AIts brightness characteristics are similar to those of ).

[0072] Reference Figure 3 While storing the learned feature points in storage device 21, the driver operates the parking position learning switch and manually parks the vehicle 1 in the target parking position. Map generation unit 45 receives map generation instructions from HMI control unit 40. Map generation unit 45 stores feature point information, including feature points output from feature point detection unit 44, the current position of the vehicle 1 synchronized with it, and the feature values ​​of the feature points, as learned feature points in storage device 21, generating map data 46. Alternatively, the position of the feature points in the map coordinate system can be calculated based on the current position of the vehicle 1 and stored as feature point information.

[0073] Furthermore, when the driver inputs information that the current position of the vehicle 1 is the target parking position 30 into the parking assistance device 10, the map generation unit 45 receives the current position of the vehicle 1 in the map coordinate system from the positioning device 11 or its own position calculation unit 43, and stores it as the target parking position 30 in the map data 46. That is, the relative positional relationship between the target parking position 30 and multiple feature points is stored as map data 46.

[0074] Subsequently, when the parking assist control unit 41 activates the parking assist control, the comparison unit 47 receives the parking position calculation command from the parking assist control unit 41.

[0075] The comparison unit 47 matches the feature point information stored as learned feature points in the map data 46 with the feature point information of surrounding feature points output from the feature point detection unit 44 when parking assistance is implemented, so that the feature point information of the same feature points is associated with each other. Based on the relative positional relationship between the surrounding feature points and the vehicle 1, and the relative positional relationship between the learned feature points associated with the surrounding feature points and the target parking position 30, the comparison unit 47 calculates the current relative position of the vehicle 1 relative to the target parking position 30. For example, the surrounding feature points are denoted as (x... i y i ), will be compared with surrounding feature points (x) i y i The learned feature points associated with each other are denoted as (x) mi y mi (i = 1 to N). The comparison section 47, based on the least squares method, calculates the affine transformation matrix M using the following formula. affine .

[0076] [Number 2]

[0077]

[0078] in,

[0079]

[0080] The conversion part 47 converts the position (targetx, targety) of the target parking position 30 on the map coordinate system stored in the map data 46 to the position (targetx, targety) of the vehicle coordinate system, using the following equation. m m

[0081] [Num 3]

[0082]

[0083] When the target trajectory generation part 48 receives the travel trajectory calculation instruction from the parking assist control part 41, it calculates the target travel trajectory from the current position of the host vehicle 1 on the vehicle coordinate system to the target parking position 30, and the target vehicle speed change map along which the host vehicle 1 travels. When the steering control part 49 receives the steering control instruction from the parking assist control part 41, it controls the steering actuator 18a so that the host vehicle 1 travels along the target travel trajectory. When the vehicle speed control part 50 receives the vehicle speed control instruction from the parking assist control part 41, it controls the accelerator actuator 18b and the brake actuator 18c so that the vehicle speed of the host vehicle 1 changes in accordance with the target vehicle speed change map.

[0084] When the parking assist control part 41 completes the parking assist control as the host vehicle 1 reaches the target parking position 30, the parking assist control part 41 operates the parking brake 17 and switches the gear to the parking gear (P range).

[0085] (Action)

[0086] Figure 10A is an explanatory diagram of one example of the process of storing the learned feature points in an environment in which the illuminance around the host vehicle 1 is above a prescribed threshold value.

[0087] In step S1, the image conversion part 42 converts the captured image of the camera to a bird's-eye view image observed from a virtual viewpoint directly above the host vehicle 1, and acquires the surrounding image. In step S2, the feature point detection part 44 performs the multi-gray scale image generation process.

[0088] Figure 10B is an explanatory diagram of a first example of the multi-gray scale image generation process. In step S10, the feature point detection part 44 converts the surrounding image to a multi-gray scale image in such a way that the luminance value of a region in which the proportion of green components is large is increased.

[0089] Figure 10C is an explanatory diagram of a second example of the multi-gray scale image generation process.

[0090] ​​In step S20, the feature point detection section 44 converts the surrounding image into a normal gray scale image. In step S21, the feature point detection section 44 converts the surrounding image into a first intermediate image. In step S22, the feature point detection section 44 generates a second intermediate image. In step S23, the feature point detection section 44 generates a multi-gray scale image by inverting the brightness of the second intermediate image.

[0091] Figure 10D is a diagram for explaining a third example of the multi-gray scale image generation processing. The processing of steps S30 to S32 is the same as that of steps S20 to S22 of Figure 10C . In step S33, the feature point detection section 44 generates a third intermediate image by inverting the brightness of the second intermediate image. In step S34, the feature point detection section 44 generates a multi-gray scale image by weightedly averaging the third intermediate image and the gray scale image.

[0092] Referring to Figure 10A , in step S3, the feature point detection section 44 extracts a feature point from the multi-gray scale image. In step S4, the feature point detection section 44 stores the extracted feature point as a learned feature point in the storage device 21.

[0093] Figure 10E is a diagram for explaining one example of the processing of storing a learned feature point in an environment in which the illuminance around the host vehicle 1 is below a prescribed threshold value.

[0094] In step S40, the controller 16 causes the infrared projector 19 to radiate infrared light toward the surroundings of the host vehicle 1. The processing of step S41 is the same as that of step S1 of Figure 10A . In step S42, the feature point detection section 44 converts the surrounding image into a normal gray scale image. In step S43, the feature point detection section 44 extracts a feature point from the gray scale image. The processing of step S44 is the same as that of step S4 of Figure 10A .

[0095] Figure 11A is a diagram for explaining one example of the processing when implementing a parking assistance in an environment in which the illuminance around the host vehicle 1 is above a prescribed threshold value. The processing of steps S50 and S51 is the same as that of steps S1 and S2 of Figure 10AThe steps S1 and S2 are the same as those of the first embodiment. In step S52, the feature point detection section 44 extracts the surrounding feature points from the multi-gradation image. In step S53, the parking assist control section 41 determines whether the distance between the host vehicle 1 and the target parking position 30 is equal to or less than a prescribed distance. In the case where it is equal to or less than the prescribed distance (step S53: Y), the process proceeds to step S54. In the case where it is not equal to or less than the prescribed distance (step S53: N), the process returns to step S50. In step S54, the parking assist control section 41 determines whether a shift operation for a U-turn is detected. In the case where the shift operation is detected (step S54: Y), the process proceeds to step S56. In the case where the shift operation is not detected (step S54: N), the process proceeds to step S55. In step S55, the parking assist control section 41 determines whether the parking assist start SW is operated by the driver. In the case where the parking assist start SW is operated (step S55: Y), the process proceeds to step S56. In the case where the parking assist start SW is not operated (step S55: N), the process returns to step S50.

[0096] In step S56, the collating section 47 reads the learned feature points from the storage device 21. In step S57, the collating section 47 matches the surrounding feature points with the learned feature points. In step S58, the collating section 47 calculates the target parking position 30 on the basis of the matched feature points. In step S59, the target trajectory generation section 48 calculates the target travel trajectory and the target vehicle speed change map. In step S60, the steering control section 49 and the vehicle speed control section 50 control the steering actuator 18a, the accelerator actuator 18b, and the brake actuator 18c on the basis of the target travel trajectory and the target vehicle speed change map. In step S61, the parking assist control section 41 operates the parking brake 17 and switches the gear to the P range when the parking assist control is completed.

[0097] Figure 11B is a diagram for explaining one example of the process when the parking assist is implemented in an environment where the illuminance around the host vehicle 1 is not equal to or more than a prescribed threshold value.

[0098] The process of steps S70 to S72 is the same as that of steps S40 to S42 of the first embodiment. Figure 10E In step S73, the feature point detection section 44 extracts the surrounding feature points from the gradation image. The process of steps S74 to S82 is the same as that of steps S53 to S61 of the first embodiment. Figure 11A

[0099] (EFFECTS OF THE EMBODIMENTS)

[0100] ​(1) In the parking assistance method, when extracting at least one of the learned feature point and the surrounding feature point, i.e., the object feature point, the surrounding image, which is a color image obtained by capturing the surroundings of the host vehicle 1, is converted into a first multi-tone image in such a manner that the luminance difference between a region in which the proportion of green color components is small and a region in which the proportion of green color components is large is increased, and the object feature point is extracted from the first multi-tone image. Thus, since it is difficult to extract a blurred feature point in a vegetation region, the calculation accuracy of the target parking position can be improved.

[0101] (2) Either one of the learned feature point and the surrounding feature point can be extracted from an image obtained by capturing when infrared rays are radiated to the surroundings of the host vehicle 1, and the other one of the learned feature point and the surrounding feature point can be extracted from the first multi-tone image. Thus, even if either one of the learned feature point and the surrounding feature point is extracted from an image captured when infrared rays are radiated, and the other one is extracted from an image captured when infrared rays are not radiated, the calculation accuracy of the target parking position can be improved.

[0102] (3) As a region in which the proportion of green color components in the surrounding image is large, the surrounding image can be converted into a first multi-tone image in such a manner that the luminance value of a vegetation region within the surrounding image is increased. Thus, since it is difficult to extract a blurred feature point in a vegetation region, the calculation accuracy of the target parking position can be improved.

[0103] (4) The first multi-tone image can be generated based on the ratio of at least two color components including green color components among the respective color components of the color image, i.e., the surrounding image.

[0104] Thus, the first multi-tone image can be generated in such a manner that the luminance value of a region in which the proportion of green color components is large is increased compared to a region in which the proportion of green color components is small.

[0105] (5) The surrounding image can be converted into a second multi-tone image, a colorless multi-tone image, i.e., a first intermediate image, can be generated based on the ratio of at least two color components including green color components among the respective color components of the color image, i.e., the surrounding image, and a second intermediate image can be generated based on the luminance difference between the first intermediate image and the second multi-tone image, and the first multi-tone image can be generated by inverting the luminance of the second intermediate image. Thus, the second intermediate image can be generated in such a manner that the luminance difference of a region in which the proportion of green color components is small is reduced.

[0106] (6) A third intermediate image can be generated by inverting the luminance of the second intermediate image, and the first multi-tone image can be generated by weightedly averaging the pixel values of the third intermediate image and the second multi-tone image. Thus, the first multi-tone image can be generated in such a manner that the luminance value of a region in which the proportion of green color components is large is increased compared to a region in which the proportion of green color components is small.

[0107] Reference Signs List

[0108] 1 own vehicle; 10 driving assistance device; 16 controller.

Claims

1. A parking assistance method that assists parking of a host vehicle to a target parking position, characterized by comprising: extracting, in advance, a feature point around the target parking position from an image obtained by capturing surroundings of the host vehicle, as a learned feature point, and storing in a storage device; capturing the surroundings of the host vehicle while moving the host vehicle to the target parking position, to obtain an image; extracting a feature point around the host vehicle from the image of the surroundings of the host vehicle, as a surrounding feature point; calculating a relative position of the host vehicle with respect to the target parking position, based on a relative positional relationship between the learned feature point and the target parking position, and a relative positional relationship between the surrounding feature point and the host vehicle; calculating a target travel trajectory from a current position of the host vehicle to the target parking position, based on the calculated relative position of the host vehicle with respect to the target parking position, to assist movement of the host vehicle along the target travel trajectory; when extracting at least one of the learned feature point and the surrounding feature point, which is an object feature point, from an image captured without irradiating infrared light to the surroundings of the host vehicle, converting a color image, which is a surrounding image obtained by capturing the surroundings of the host vehicle, into a first multi-tone image, in such a manner that a luminance value of a region in which a proportion of green color components is large in the surrounding image is increased more than a region in which the proportion of green color components is small, extracting the object feature point from the first multi-tone image.

2. The parking assistance method according to claim 1, characterized in that: any one of the learned feature point and the surrounding feature point is extracted from an image obtained by capturing while irradiating infrared light to the surroundings of the host vehicle, and the other one of the learned feature point and the surrounding feature point is extracted from the first multi-tone image.

3. The parking assistance method according to claim 1, characterized in that: the surrounding image is converted into the first multi-tone image in such a manner that a luminance value of a region in which a proportion of green color components is large in the surrounding image is increased more than a region in which the proportion of green color components is small, as a region in which vegetation is present in the surrounding image.

4. The parking assistance method according to any one of claims 1 to 3, characterized in that: the first multi-tone image is generated based on a ratio of at least two color components that contain green color components among color components of a color image, which is the surrounding image.

5. The parking assistance method according to any one of claims 1 to 3, characterized in that: the surrounding image is subjected to grayscale conversion to generate a second multi-tone image, a colorless multi-tone image, which is a first intermediate image, is generated based on a ratio of at least two color components that contain green color components among color components of a color image, which is the surrounding image, a second intermediate image is generated based on a luminance difference between the first intermediate image and the second multi-tone image, the first multi-tone image is generated by inverting the luminance of the second intermediate image.

6. The parking assistance method according to any one of claims 1 to 3, characterized in that: performing a grayscale conversion on the surrounding image to generate a second multi-grayscale image, generating a colorless multi-grayscale image, i.e., a first intermediate image, based on a ratio of at least two color components including a green color component among color components of the color image, i.e., the surrounding image, generating a second intermediate image based on a luminance difference between the first intermediate image and the second multi-grayscale image, generating a third intermediate image by inverting the luminance of the second intermediate image, generating the first multi-grayscale image by performing a weighted average of pixel values of the third intermediate image and the second multi-grayscale image.

7. A parking assist device characterized by comprising: having: a photographing device that photographs a surrounding of a host vehicle; a storage device; a controller that extracts a feature point around a target parking position as a learned feature point from an image obtained by photographing the surrounding of the host vehicle using the photographing device and stores the learned feature point in the storage device, acquires an image by photographing the surrounding of the host vehicle while moving the host vehicle to the target parking position, and extracts a feature point around the host vehicle as a surrounding feature point from the image of the surrounding of the host vehicle, calculates a relative position of the host vehicle with respect to the target parking position based on a relative positional relationship between the learned feature point and the target parking position and a relative positional relationship between the surrounding feature point and the host vehicle, and calculates a target travel trajectory from a current position of the host vehicle to the target parking position based on the calculated relative position of the host vehicle with respect to the target parking position, and assists the host vehicle in moving along the target travel trajectory; the controller, when extracting at least one of the learned feature point and the surrounding feature point, i.e., an object feature point, from an image photographed without irradiating infrared light to the surrounding of the host vehicle, converts the surrounding image, i.e., a color image obtained by photographing the surrounding of the host vehicle, into a first multi-grayscale image in such a manner that a luminance value of a region in which a proportion of a green color component is large in the surrounding image is increased more than a region in which the proportion of the green color component is small, extracts the object feature point from the first multi-grayscale image.

Images