Parking assistance method and parking assistance device

Abstract

In a parking assistance method, a surrounding image (S1), which is a color image obtained by capturing an image of the surroundings of a vehicle, is acquired, the surrounding image is converted into a first multi-gradation image (S2 to S6) so that the brightness difference between a region having a small proportion of green components and a region having a large proportion of green components in the surrounding image is increased, and the first multi-gradation image is converted into a second multi-gradation image (S2 to S6) so that the brightness difference between the first multi-gradation image and the second multi-gradation image is increased. Feature points are extracted from the first multi-gradation image (S7), and the relative position of the vehicle with respect to the target parking position is calculated on the basis of the extracted feature points (S32).

Description

Technical Field

[0001] The invention relates to a parking assistance method and a parking assistance device. Background Art

[0002] Patent document 1 describes a driving control device that extracts and stores feature points from an image previously taken of the surroundings of a target parking position, calculates the relative position of the target parking position with respect to the vehicle based on the stored positions of the object markers and the positions of the object markers extracted from an image taken of the surroundings of the vehicle during automatic parking, and automatically moves the vehicle to the target parking position based on the calculated relative position.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: (Japanese) Patent Application Publication No. 2017-138664 Summary of the invention

[0006] Technical problem to be solved by the invention

[0007] When a vegetation area is included in an image captured around the target parking position or an image captured around the vehicle, the calculation accuracy of the target parking position may be reduced because feature points of blurred contours are extracted from the vegetation area.

[0008] An object of the present invention is to improve the calculation accuracy of a target parking position when a photographic image includes a vegetation area in a parking assistance for assisting a vehicle to park at a target parking position based on feature points extracted from a photographic image.

[0009] Technical solutions for solving technical problems

[0010] According to one embodiment of the present invention, a parking assistance method for assisting a vehicle to park at a target parking position is provided. The method comprises: extracting feature points around the target parking position from an image obtained by photographing the surroundings of the vehicle in advance as learned feature points and storing them in a storage device; photographing the surroundings of the vehicle when the vehicle moves to the target parking position to obtain an image; extracting feature points around the vehicle from the image around the vehicle as surrounding feature points; calculating the relative position of the vehicle relative to the target parking position based on the relative position relationship between the learned feature points and the target parking position, and the relative position relationship between the surrounding feature points and the vehicle; calculating the target driving trajectory from the current position of the vehicle to the target parking position based on the calculated relative position of the vehicle relative to the target parking position, and assisting the vehicle to move along the target driving trajectory. When extracting at least one of the learned feature points and the surrounding feature points, namely the object feature points, the surrounding image is converted into a first multi-grayscale image in such a way that the brightness difference between the area with a small proportion of green component and the area with a large proportion of green component in the color image obtained by photographing the surroundings of the vehicle, namely the surrounding image, is increased, and the object feature points are extracted from the first multi-grayscale image.

[0011] Effects of the Invention

[0012] According to the present invention, in parking assistance for assisting a vehicle to park at a target parking position based on feature points extracted from a photographic image, it is possible to improve the calculation accuracy of the target parking position when a vegetation area is included in the photographic image. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a diagram showing an example of the schematic configuration of a parking assist device.

[0014] Figure 2A This is an explanatory diagram of an example of a process of storing learned feature points.

[0015] Figure 2B It is an explanatory diagram of an example of processing when performing parking assistance.

[0016] Figure 3 yes Figure 1 A block diagram of an example of the functional structure of a controller.

[0017] Figure 4 This is a diagram showing an example of a grayscale image generated from a surrounding image when the illumination around the host vehicle is equal to or greater than a predetermined threshold value.

[0018] Figure 5 This is a diagram showing an example of a first intermediate image generated from a surrounding image when the illumination around the host vehicle is equal to or greater than a predetermined threshold value.

[0019] Figure 6 This is a diagram showing an example of the second intermediate image.

[0020] Figure 7 FIG. 1 is a diagram showing an example of a multi-grayscale image obtained using the second method.

[0021] Figure 8 This is a diagram showing an example of a multi-grayscale image obtained using the third method.

[0022] Fig.9A The diagram shows an example of a grayscale image generated from a surrounding image when the illumination around the vehicle is less than a predetermined threshold value and a partially enlarged image.

[0023] Fig. 9B Yes means Figure 4 Grayscale image of the image and partially enlarged image.

[0024] Fig. 9C Yes means Figure 8 Multi-grayscale images and partially enlarged images.

[0025] Fig. 10A This is an explanatory diagram of a first example of the process of storing learned feature points.

[0026] Fig. 10B This is an explanatory diagram of the first example of multi-tone image generation processing.

[0027] Fig. 10C It is an explanatory diagram of the second example of the multi-tone image generation process.

[0028] Fig. 10D It is an explanatory diagram of the third example of the multi-tone image generation process.

[0029] Fig.10E It is an explanatory diagram of a second example of the process of storing learned feature points.

[0030] Fig.11A It is an explanatory diagram of a first example of the process when performing parking assistance.

[0031] Fig. 11B It is an explanatory diagram of a second example of the process when performing parking assistance. DETAILED DESCRIPTION

[0032] (constitute)

[0033] Reference Figure 1. The vehicle 1 has a parking assistance device 10 for assisting the vehicle 1 in parking at a target parking position. The parking assistance device 10 assists the vehicle 1 in traveling along a target driving trajectory from the current position to the target parking position. For example, automatic driving of the vehicle 1 can be controlled (i.e., all or part of the steering angle, driving force, and braking force of the vehicle are controlled, and all or part of the control of the vehicle 1 traveling along the target driving trajectory is automatically implemented) so that the vehicle 1 travels along the target driving trajectory to the target parking position. The vehicle 1 can also be assisted in parking by displaying the target driving trajectory and the current position of the vehicle 1 on a display device that can be visually confirmed by the occupants of the vehicle 1.

[0034] The positioning device 11 measures the current position of the vehicle 1. The positioning device 11 has, for example, a global navigation satellite system (GNSS) receiver. The human-machine interface (HMI) 12 is an interface device between the parking assistance device 10 and the occupant, and has: a display device, a speaker or a buzzer, and an operating member. The shift switch (shift SW) 13 is a switch used by the driver or the parking assistance device 10 to switch the gear position. The external sensor 14 detects objects within a predetermined distance range from the vehicle 1. The external sensor 14 detects the surrounding environment of the vehicle 1, such as the relative position of the object existing around the vehicle 1 and the vehicle 1, the distance between the vehicle 1 and the object, and the direction in which the object exists. The external sensor 14 may include, for example, a camera that captures the surrounding environment of the vehicle 1. In order to respond to the parking assistance of the vehicle 1 in an environment (for example, at night) where the illumination around the vehicle 1 is less than a predetermined threshold, a day and night camera that can take pictures in both the visible light region and the infrared region can be used as the camera of the external sensor 14. It is also possible to have a daytime camera that can take pictures in the visible light region, and a camera that can take pictures in the infrared region. In this embodiment, an example of a case where a day and night camera is used is described. Below, the camera of the external sensor 14 is simply referred to as a "camera". The external sensor 14 may also include ranging devices such as a laser rangefinder, a radar, and LiDAR (Light Detection and Ranging). The vehicle sensor 15 detects various information (vehicle information) of the vehicle 1. The vehicle sensor 15 may include, for example: a vehicle speed sensor, a wheel speed sensor, a three-axis acceleration sensor (G sensor), a steering angle sensor, a steering angle sensor, a gyro sensor, and a yaw rate sensor.

[0035] The controller 16 is an electronic control unit that performs parking assist control of the vehicle 1. The controller 16 includes a processor 20, and peripheral accessories such as a storage device 21. The storage device 21 can have a semiconductor storage device, a magnetic storage device, and an optical storage device. The function of the controller 16 is realized, for example, by the processor 20 executing a computer program stored in the storage device 21. The steering actuator 18a controls the steering direction and steering amount of the steering mechanism according to the control signal of the controller 16. The accelerator actuator 18b controls the accelerator opening of the drive device (engine, drive motor) according to the control signal of the controller 16. The brake actuator 18c operates the brake device according to the control signal of the controller 16.

[0036] When assisting the parking of the vehicle 1 in an environment where the illumination around the vehicle 1 is less than a predetermined threshold, the infrared projector 19 irradiates infrared rays around the vehicle 1 according to the control signal of the controller 16. For example, the infrared projector may be an infrared light emitting diode, which is arranged on the left and right sides of the vehicle 1, and emits infrared rays obliquely downward to the left and right sides of the vehicle 1, and irradiates infrared rays to the road surface around the left and right sides of the vehicle 1.

[0037] Next, the parking assist control performed by the parking assist device 10 will be described. Figure 2A When the parking assistance device 10 is used for parking assistance, feature points are extracted from an image of the area around the target parking position 30 where the vehicle 1 is to be parked, and are pre-stored in the storage device 21. Hereinafter, the feature points stored in the storage device 21 are referred to as "learned feature points". Figure 2A The circular icon in the middle represents a learned feature point. For example, the parking assistance device 10 extracts feature points around the target parking position 30 from the surrounding image obtained by photographing the surroundings of the vehicle 1 using a camera when the vehicle 1 is located near the target parking position 30 (for example, when parking at the target parking position 30 by manual driving). For example, edge points where the brightness of adjacent pixels such as edges or corners of objects such as road markings, road boundaries, obstacles, etc. on the photographed image obtained by photographing with a camera changes by more than a specified amount, or points with features in shape are detected as feature points. When the learned feature points are stored in the storage device 21, for example, the driver operates the "parking position learning switch" prepared as an operating member of the HMI 12. It should be noted that when the illumination around the vehicle 1 is less than a specified threshold, the learned feature points are extracted from the photographed image obtained by photographing the vehicle 1 in a state where infrared rays are irradiated from the infrared projector 19 to the surroundings of the vehicle 1.

[0038] The parking assistance device 10 stores the relative positional relationship between the learned feature points and the target parking position 30. For example, the driver can use the HMI 12 to input information that the vehicle 1 is located at the target parking position 30. The parking assistance device 10 can also calculate the relative positional relationship between the learned feature points and the target parking position 30 based on the position of the learned feature points detected when the vehicle 1 is located at the target parking position 30. For example, the parking assistance device 10 can also store the learned feature points and the coordinates of the target parking position 30 on a coordinate system (hereinafter referred to as the "map coordinate system") with a fixed location as a reference point. In this case, the current position on the map coordinate system measured by the positioning device 11 when the vehicle 1 is located at the target parking position 30 can also be stored as the target parking position 30. In addition, instead of the map coordinate system, the relative position of the target parking position 30 relative to each learned feature point can also be stored.

[0039] Figure 2B This is an explanatory diagram of an example of processing when implementing parking assistance. The parking assistance device 10 implements parking assistance for the vehicle 1 when the vehicle 1 is located near the target parking position 30. For example, the parking assistance device 10 determines whether the driver has performed a shift operation for switching the vehicle 1 forward and reverse when the vehicle 1 is located near the target parking position 30. Mark 21 is the turning position. The parking assistance device 10 can determine that a shift operation for turning back has been performed when the gear position is switched from the forward gear (D gear) to the reverse gear (R gear), or from the R gear to the D gear, and start the parking assistance. The parking assistance can also be started when the driver operates the "parking assistance start switch" prepared in the HMI 12 when the vehicle 1 is located at the position 33 near the target parking position 30.

[0040] The parking assistance device 10 extracts feature points around the host vehicle 1 from a surrounding image obtained by photographing the surroundings of the host vehicle 1 using a camera. Hereinafter, the feature points around the host vehicle 1 extracted when performing parking assistance are referred to as "surrounding feature points". Figure 2B The middle triangle icons represent surrounding feature points.

[0041] It should be noted that when the illumination around the host vehicle 1 is less than a predetermined threshold during parking assistance, surrounding feature points are extracted from a photographed image obtained by taking a picture with the infrared projector 19 irradiating infrared light around the host vehicle 1 with the camera.

[0042] The parking assist device 10 compares and checks (matches) the learned feature points stored in the storage device 21 with the surrounding feature points, and associates the same feature points with each other.

[0043] The parking assist device 10 calculates the relative position of the host vehicle 1 with respect to the target parking position 30 based on the relative positional relationship between the surrounding feature points detected when the parking assist is implemented and the host vehicle 1, and the relative positional relationship between the learned feature points associated with the surrounding feature points and the target parking position 30. For example, the parking assist device 10 may calculate the position of the target parking position 30 on a coordinate system (hereinafter referred to as a "vehicle coordinate system") based on the current position of the host vehicle 1. For example, when the coordinates of the learned feature points and the target parking position 30 in the map coordinate system are stored in the storage device 21, the coordinates of the target parking position 30 on the map coordinate system may be converted into coordinates on the vehicle coordinate system based on the positions of the surrounding feature points detected when the parking assist is implemented and the positions of the learned feature points in the map coordinate system. Alternatively, the current position of the host vehicle 1 on the map coordinate system may be obtained based on the positions of the surrounding feature points detected when the parking assistance is implemented and the positions of the learned feature points in the map coordinate system, and the relative position of the host vehicle 1 to the target parking position 30 may be calculated based on the difference between the coordinates of the host vehicle 1 and the coordinates of the target parking position 30 in the map coordinate system. The parking assistance device 10 calculates the target driving trajectory from the current position of the host vehicle 1 to the target parking position 30 based on the relative position of the host vehicle 1 to the target parking position 30. For example, when the position of the host vehicle 1 is the turnaround position 31 at the time of starting the parking assistance, the trajectory 32 from the turnaround position 31 to the target parking position 30 is calculated. In addition, when the position of the host vehicle 1 is the position 33 near the target parking position 30 at the time of starting the parking assistance, for example, the trajectory 34 from the position 33 to the turnaround position 31 and the trajectory 32 from the turnaround position 31 to the target parking position 30 are calculated. The parking assist device 10 performs parking assist control of the host vehicle 1 based on the calculated target driving trajectory.

[0044] The functional structure of the controller 16 is as follows: Figure 3 As shown. When the parking position learning switch is operated, the human-machine interface control unit (HMI control unit) 40 outputs a map generation instruction to store the learned feature points in the storage device 21 to the map generation unit 45. The HMI control unit 40 determines whether the driver has performed a shift operation for turning back and whether the parking assist start switch has been operated, and outputs the determination result to the parking assist control unit 41. The parking assist control unit 41 determines whether the vehicle 1 is located near the stored target parking position 30. For example, it determines whether the distance between the vehicle 1 and the target parking position 30 is less than a specified distance. When the vehicle 1 is located near the target parking position 30 and the parking assist start switch is detected to be operated or the shift operation for turning back is detected, the parking assist control unit 41 sets the stored target parking position 30 as the target parking position 30 of this control and starts the parking assist control.

[0045] When the parking assist control is started, the parking assist control unit 41 outputs a parking position calculation instruction for calculating the position of the target parking position 30 in the vehicle coordinate system to the control unit 47. In addition, a driving trajectory calculation instruction for calculating the target driving trajectory and the target vehicle speed change diagram of the vehicle 1 traveling along the target driving trajectory is output to the target trajectory generation unit 48. The target trajectory generation unit 48 calculates the target driving trajectory from the current position of the vehicle 1 to the target parking position 30, and the target vehicle speed change diagram, and outputs them to the parking assist control unit 41. The calculation of the target driving trajectory can be applied to a well-known method used in an automatic parking device. For example, it can be calculated by connecting the current position of the vehicle 1, via the return position 31, to the target parking position 30 with a clothoid curve. In addition, for example, the target vehicle speed change diagram can be the following vehicle speed change diagram, that is, after accelerating from the current position of the vehicle 1 to a predetermined set speed, decelerating near the turning position 31 and stopping at the turning position 31, then accelerating from the turning position 31 to the set speed and decelerating near the target parking position 30, and then stopping at the target parking position 30.

[0046] The parking assist control unit 41 outputs information on the target driving trajectory and the current position of the host vehicle 1 to the HMI control unit 40. When the target driving trajectory includes a turnaround, information on the turnaround position is output to the HMI control unit 40. The HMI control unit 40 displays the target driving trajectory, the current position of the host vehicle 1, and the turnaround position on the HMI 12.

[0047] In addition, the parking assist control unit 41 outputs a steering control instruction for steering control to the steering control unit 49 so that the vehicle 1 travels along the calculated target driving trajectory. In addition, a vehicle speed control instruction for controlling the speed of the vehicle 1 according to the calculated target vehicle speed change diagram is output to the vehicle speed control unit 50. The image conversion unit 42 converts the captured image of the camera into a bird's-eye view image (panoramic monitoring image) observed from a virtual perspective directly above the vehicle 1. The image conversion unit 42 converts the captured image into a bird's-eye view image at a specified interval (for example, every time the vehicle 1 travels a specified distance (for example, 50 cm) or a specified time (for example, one second)), accumulates the converted bird's-eye view images along the driving route of the vehicle 1, and thereby generates a Figure 2A and Figure 2B The own position calculation unit 43 calculates the current position of the own vehicle 1 on the map coordinate system using the dead reckoning method based on the vehicle information output from the vehicle sensor 15 .

[0048] When learned feature points are stored in the storage device 21, the feature point detection unit 44 detects the learned feature points and their image feature quantities based on the surrounding image output from the image conversion unit 42. The surrounding feature points and their image feature quantities are detected when the parking assist control is implemented. In the following, there is a case where the learned feature points and the surrounding feature points are collectively referred to as "feature points". The feature point detection unit 44 detects the feature points and their image feature quantities based on the surrounding image output from the image conversion unit 42. The detection of feature points or the calculation of image feature quantities can use methods such as SIFT, SURF, ORB, BRIAK, KAZE, AKAZE, etc.

[0049] The surrounding image of the vehicle 1 may include a vegetation area (i.e., an area where plants are grown). In the vegetation area, there is a brightness difference between the part of the plant that is illuminated by light and the shadow part that is not illuminated by light, and feature points with blurred outlines may be extracted in a small size. Since the shadow of the plant leaves changes due to the light source (such as the sun), the calculation accuracy of the target parking position may be reduced.

[0050] In addition, in an image captured without infrared irradiation in an environment (e.g., during the day) where the illumination around the vehicle 1 is above a predetermined threshold, the brightness of the vegetation area is generally lower than that of the asphalt, while in an image captured with infrared irradiation, the brightness of the vegetation area is higher. Therefore, when either the learned feature points or the surrounding feature points are extracted from the image captured with infrared irradiation and the other is extracted from the image captured without infrared irradiation, the calculation accuracy of the target parking position may be reduced.

[0051] Therefore, when the parking assistance device 10 of the embodiment extracts learned feature points or surrounding feature points from an image captured without irradiating infrared rays around the vehicle 1 in an environment where the illumination around the vehicle 1 is above a prescribed threshold, the surrounding image is converted into a colorless multi-grayscale image in such a manner that the brightness value of an area with a large proportion of green components in the color surrounding image obtained by capturing the surroundings of the vehicle 1 is increased compared to an area with a small proportion of green components, and feature points are extracted from the converted multi-grayscale image.

[0052] For example, the surrounding image may be converted into a multi-grayscale image so that the brightness value of a region having a large proportion of green components is saturated (ie, becomes the upper limit value of the image brightness).

[0053] In this way, since the vegetation area of ​​the colored surrounding image is an area with a large proportion of green components, the brightness value of the area with a large proportion of green components is increased, thereby reducing the brightness difference between the part of the plant that is irradiated with light and the shadow part that is not irradiated with light. As a result, it is difficult to extract fuzzy contour feature points in the vegetation area, so the calculation accuracy of the target parking position can be improved. In addition, the brightness of the vegetation area can be made higher than that of the asphalt area in the image taken without irradiation with infrared rays. As a result, the brightness characteristics of the vegetation area and the asphalt area are similar between the image taken without irradiation with infrared rays and the image taken with irradiation with infrared rays, so the calculation accuracy of the target parking position can be improved.

[0054] Next, a detailed description will be given of a method for detecting feature points by the feature point detection unit 44. In an environment where the illumination around the host vehicle 1 is equal to or higher than a predetermined threshold, the feature point detection unit 44 extracts feature points from a surrounding image captured without infrared radiation.

[0055] In order to reduce the brightness difference between the part of the plant irradiated with light and the shadow part not irradiated with light, the feature point detection unit 44 converts the surrounding image into a multi-grayscale image in such a way that the brightness value of the area with a large proportion of green components in the color surrounding image is increased compared with the area with a small proportion of green components. This multi-grayscale image is an example of the "first multi-grayscale image" described in the claims.

[0056] As a first method, the surrounding image may be converted into a multi-grayscale image in such a manner that the brightness value of the area having a large ratio of the green component is increased. For example, the feature point detection unit 44 converts the surrounding image into a multi-grayscale image in such a manner that the brightness value of the area having a large ratio of the G component (green component) Cg greater than a prescribed value is saturated (i.e., becomes the upper limit of the image brightness) based on the ratio of at least two color components including the G component Cg among the R component (red component) Cr, the G component (green component) Cg, and the B component (blue component) Cb of the color surrounding image. It should be noted that the brightness value of the area having a large ratio of Cg greater than the prescribed value may be increased based on the ratio of the two color components instead of being the upper limit of the image brightness.

[0057] As a second method, (1) a grayscale image of the surrounding image (equivalent to a second multi-grayscale image) is generated; (2) the surrounding image is converted into a first intermediate image in such a manner that the brightness value of the vegetation area is reduced; (3) a second intermediate image is generated based on the brightness difference between the grayscale image and the first intermediate image; (4) the brightness of the second intermediate image is inverted. Through the above processes (1) to (4), a colorless multi-grayscale image is obtained in which the brightness value of the area with a large proportion of green components is larger than that of the area with a small proportion of green components.

[0058] First, the feature point detection unit 44 generates a grayscale image of the surrounding image by performing a normal grayscale conversion on the surrounding image which is a color image.

[0059] Figure 4 is a diagram showing an example of a grayscale image. For example, the feature point detection unit 44 can determine the brightness value of the grayscale image by weighting the R component Cr, the G component Cg, and the B component Cb of the color surrounding image with a predetermined weighting coefficient. The grayscale image is an example of the "second multi-grayscale image" described in the claims. The area Rv surrounded by the dotted line represents a vegetation area. In the image captured without irradiating the surroundings of the vehicle 1 with infrared rays, the brightness of the vegetation area Rv is lower than the brightness of the surrounding asphalt area.

[0060] Next, the feature point detection unit 44 converts the colored surrounding image into a colorless multi-grayscale image, that is, a first intermediate image. At this time, the feature point detection unit 44 converts the surrounding image into the first intermediate image in such a way that the brightness value of the vegetation area with a large proportion of the G component Cg in the surrounding image is smaller than that of the area with a small proportion of the G component Cg.

[0061] Figure 5 is a diagram showing an example of a first intermediate image. For example, the feature point detection unit 44 may generate the first intermediate image based on the ratio of at least two color components including the G component Cg among the R component Cr, the G component Cg, and the B component Cb of the surrounding image. For example, when the maximum value of each of the R component Cr, the G component Cg, and the B component Cb is 255, the feature point detection unit 44 may determine the brightness value Ck of each pixel in the first intermediate image based on the following equations (1) and (2).

[0062] [Number 1]

[0063]

[0064] In addition, in the above formula, the B component Cb may be replaced by the R component Cr.

[0065] Next, the feature point detection unit 44 performs Figure 4 The grayscale image and Figure 5 For example, the feature point detection unit 44 generates a difference image obtained by subtracting the brightness value of each pixel of the grayscale image from the brightness value of each pixel of the first intermediate image as the second intermediate image. Figure 6 : is a diagram showing an example of the second intermediate image. By subtracting the brightness value of each pixel of the grayscale image from the brightness value of each pixel of the first intermediate image, the brightness difference of the area other than the vegetation area Rv (eg, the asphalt area) can be reduced in the second intermediate image.

[0066] Next, the feature point detection unit 44 inverts the brightness of the second intermediate image to generate an achromatic multi-tone image in which the brightness value of a region with a large green component ratio is larger than that of a region with a small green component ratio. Figure 7 is a diagram showing an example of a multi-grayscale image obtained using the second method. Figure 7 In the image, the vegetation region Rv becomes substantially uniformly brighter (the brightness increases substantially uniformly), and the region other than the vegetation region Rv (eg, the asphalt region) becomes substantially uniformly darker (the brightness decreases substantially uniformly).

[0067] As a third method, the multi-grayscale image (brightness-inverted image of the second intermediate image) obtained by the second method is used as a third intermediate image, and the pixel values ​​of the third intermediate image and the grayscale image are weighted averaged to generate a multi-grayscale image.

[0068] Figure 8 : is a diagram showing an example of a multi-grayscale image obtained by the third method. By increasing the pixel value of the third intermediate image, the brightness value of the vegetation area Rv is increased compared to the brightness values ​​of other areas.

[0069] On the other hand, in an environment where the illumination around the vehicle 1 is less than a predetermined threshold, the feature point detection unit 44 extracts feature points from the surrounding image captured when infrared rays are irradiated by the infrared projector 19. In this case, the feature point detection unit 44 generates a grayscale image of the surrounding image by performing a normal grayscale conversion on the color image, i.e., the surrounding image. The feature point detection unit 44 extracts feature points from the grayscale image. Fig.9A An example of a grayscale image of a surrounding image captured by irradiating infrared rays and an enlarged image of a boundary portion between a vegetation region Rv and an asphalt region are shown. In the image captured by irradiating infrared rays, the brightness of the vegetation region Rv is higher than that of the asphalt region.

[0070] Fig. 9B is a grayscale image representing the surrounding image captured without irradiation with infrared rays ( Figure 4 In the image captured without infrared irradiation, unlike the image captured with infrared irradiation, the brightness of the vegetation area Rv is lower than that of the asphalt area.

[0071] Fig. 9C is a multi-grayscale image generated in such a way that the brightness value of the vegetation area Rv is larger than the brightness values ​​of other areas ( Figure 8 The following figure shows a multi-grayscale image of the image) and a partially enlarged image. Fig. 9C In the multi-grayscale image, the brightness of the vegetation area Rv is higher than that of the asphalt area, so that the brightness characteristics of the vegetation area Rv and the asphalt area can be made similar to the grayscale image when irradiated with infrared rays ( Fig.9A) have similar brightness characteristics.

[0072] Reference Figure 3 . When the learned feature points are stored in the storage device 21, the driver operates the parking position learning switch and parks the vehicle 1 at the target parking position by manual driving. The map generation unit 45 receives a map generation instruction from the HMI control unit 40. The map generation unit 45 stores the feature point information including the feature point output from the feature point detection unit 44, the current position of the vehicle 1 synchronized therewith, and the feature amount of the feature point as the learned feature point in the storage device 21, and generates map data 46. The position of the feature point in the map coordinate system may also be calculated based on the current position of the vehicle 1 and stored as the feature point information.

[0073] In addition, when the driver inputs information that the current position of the host vehicle 1 is the target parking position 30 to the parking assistance device 10, the map generation unit 45 receives the current position of the host vehicle 1 on the map coordinate system from the positioning device 11 or the own position calculation unit 43, and stores it in the map data 46 as the target parking position 30. That is, the relative positional relationship between the target parking position 30 and the plurality of feature points is stored as the map data 46.

[0074] Thereafter, when the parking assist control unit 41 starts the parking assist control, the comparison unit 47 receives a parking position calculation instruction from the parking assist control unit 41 .

[0075] The comparison unit 47 matches the feature point information stored as the learned feature point in the map data 46 with the feature point information of the surrounding feature points output from the feature point detection unit 44 when the parking assistance is implemented, and associates the feature point information of the same feature points with each other. The comparison unit 47 calculates the current relative position of the host vehicle 1 relative to the target parking position 30 based on the relative position relationship between the surrounding feature points and the host vehicle 1, and the relative position relationship between the learned feature points associated with the surrounding feature points and the target parking position 30. For example, the surrounding feature points are recorded as (x i ,y i ), will be combined with the surrounding feature points (x i ,y i ) are respectively associated with the learned feature points denoted as (x mi ,y mi )(i=1 to N). The comparison unit 47 calculates the affine transformation matrix M using the following equation based on the least squares method: affine .

[0076] [Number 2]

[0077]

[0078] The comparison unit 47 calculates the position (target x m , targety m ) is converted to the position (targetx, targety) in the vehicle coordinate system.

[0079] [Number 3]

[0080]

[0081] When the target trajectory generating unit 48 receives a driving trajectory calculation instruction from the parking assist control unit 41, it calculates a target driving trajectory from the current position of the host vehicle 1 on the vehicle coordinate system to the target parking position 30, and a target vehicle speed change diagram for the host vehicle 1 to travel along the target driving trajectory. When the steering control unit 49 receives a steering control instruction from the parking assist control unit 41, it controls the steering actuator 18a so that the host vehicle 1 travels along the target driving trajectory. When the vehicle speed control unit 50 receives a vehicle speed control instruction from the parking assist control unit 41, it controls the accelerator actuator 18b and the brake actuator 18c so that the vehicle speed of the host vehicle 1 changes according to the target vehicle speed change diagram.

[0082] When the vehicle 1 reaches the target parking position 30 and the parking assist control is completed, the parking assist control unit 41 operates the parking brake 17 to switch the shift position to the parking position (P position).

[0083] (action)

[0084] Fig. 10A This is an explanatory diagram of an example of a process of storing learned feature points in an environment where the illumination around the host vehicle 1 is equal to or higher than a predetermined threshold value.

[0085] In step S1, the image conversion unit 42 converts the camera image into a bird's-eye view image observed from a virtual perspective directly above the host vehicle 1, thereby acquiring a surrounding image. In step S2, the feature point detection unit 44 performs a multi-grayscale image generation process.

[0086] Fig. 10B 1 is an explanatory diagram of a first example of the multi-tone image generation process. In step S10 , the feature point detection unit 44 converts the surrounding image into a multi-tone image so that the luminance value of the region having a large proportion of the green component increases.

[0087] Fig. 10C It is an explanatory diagram of the second example of the multi-tone image generation process.

[0088] In step S20, the feature point detection unit 44 converts the surrounding image into a normal grayscale image. In step S21, the feature point detection unit 44 converts the surrounding image into a first intermediate image. In step S22, the feature point detection unit 44 generates a second intermediate image. In step S23, the feature point detection unit 44 generates a multi-grayscale image by inverting the brightness of the second intermediate image.

[0089] Fig. 10D FIG. 2 is an explanatory diagram of a third example of the multi-grayscale image generation process. Fig. 10C The steps S20 to S22 are the same. In step S33, the feature point detection unit 44 generates a third intermediate image by inverting the brightness of the second intermediate image. In step S34, the feature point detection unit 44 generates a multi-grayscale image by weighted averaging the third intermediate image and the grayscale image.

[0090] Reference Fig. 10A In step S3 , the feature point detection unit 44 extracts feature points from the multi-grayscale image. In step S4 , the feature point detection unit 44 stores the extracted feature points in the storage device 21 as learned feature points.

[0091] Fig.10E This is an explanatory diagram of an example of a process of storing learned feature points in an environment where the illumination around the host vehicle 1 is less than a predetermined threshold.

[0092] In step S40, the controller 16 uses the infrared projector 19 to irradiate infrared rays around the vehicle 1. Fig. 10A The processing of step S1 is the same as that of step S42. In step S43, the feature point detection unit 44 extracts feature points from the grayscale image. The processing of step S44 is the same as that of step S1. Fig. 10A The processing of step S4 is the same as that of step S4.

[0093] Fig.11A 1 is an explanatory diagram of an example of processing when parking assistance is performed in an environment where the illumination around the host vehicle 1 is greater than a predetermined threshold value. The processing of steps S50 and S51 is similar to that of Fig. 10AThe steps S1 and S2 of are the same. In step S52, the feature point detection unit 44 extracts surrounding feature points from the multi-grayscale image. In step S53, the parking assist control unit 41 determines whether the distance between the host vehicle 1 and the target parking position 30 is less than the prescribed distance. If it is less than the prescribed distance (step S53: Y), the process proceeds to step S54. If it is not less than the prescribed distance (step S53: N), the process returns to step S50. In step S54, the parking assist control unit 41 determines whether a shift operation for turning back is detected. If a shift operation is detected (step S54: Y), the process proceeds to step S56. If a shift operation is not detected (step S54: N), the process proceeds to step S55. In step S55, the parking assist control unit 41 determines whether the parking assist start SW is operated by the driver. If the parking assist start SW is operated (step S55: Y), the process proceeds to step S56. When the parking assist start SW is not operated (step S55 : N), the process returns to step S50 .

[0094] In step S56, the control unit 47 reads the learned feature points from the storage device 21. In step S57, the control unit 47 matches the surrounding feature points with the learned feature points. In step S58, the control unit 47 calculates the target parking position 30 based on the matched feature points. In step S59, the target trajectory generation unit 48 calculates the target driving trajectory and the target vehicle speed change map. In step S60, the steering control unit 49 and the vehicle speed control unit 50 control the steering actuator 18a, the accelerator actuator 18b, and the brake actuator 18c based on the target driving trajectory and the target vehicle speed change map. In step S61, when the parking assist control is completed, the parking assist control unit 41 operates the parking brake 17 and switches the gear to the P gear.

[0095] Fig. 11B This is an explanatory diagram of an example of processing when parking assistance is performed in an environment where the illumination around the host vehicle 1 is less than a predetermined threshold.

[0096] The processing of steps S70 to S72 is Fig.10E In step S73, the feature point detection unit 44 extracts surrounding feature points from the grayscale image. The processing of steps S74 to S82 is the same as Fig.11A The processing of steps S53 to S61 is the same.

[0097] (Effects of implementation methods)

[0098] (1) In a parking assist method, when extracting at least one of a learned feature point and a surrounding feature point, namely, a target feature point, the surrounding image is converted into a first multi-grayscale image in such a manner that the brightness difference between an area with a small green component ratio and an area with a large green component ratio in a color image obtained by photographing the surroundings of the host vehicle 1 is increased, and the target feature point is extracted from the first multi-grayscale image. As a result, it is difficult to extract blurred feature points in a vegetation area, thereby improving the calculation accuracy of a target parking position.

[0099] (2) Either one of the learned feature points and the surrounding feature points can be extracted from an image captured by irradiating infrared rays around the host vehicle 1, and the other one of the learned feature points and the surrounding feature points can be extracted from the first multi-grayscale image. Thus, even if either one of the learned feature points and the surrounding feature points is extracted from an image captured by irradiating infrared rays, and the other one is extracted from an image captured without irradiating infrared rays, the calculation accuracy of the target parking position can be improved.

[0100] (3) As the area with a large proportion of green components in the surrounding image, the surrounding image can be converted into a first multi-grayscale image in such a way that the brightness value of the vegetation area in the surrounding image is increased. As a result, it is difficult to extract blurred feature points in the vegetation area, thereby improving the calculation accuracy of the target parking position.

[0101] (4) The first multi-grayscale image may be generated based on the ratio of at least two color components including a green component among the color components of the color image, ie, the surrounding image.

[0102] Thus, the first multi-grayscale image can be generated in such a manner that the luminance value of the region with a large proportion of the green component is larger than that of the region with a small proportion of the green component.

[0103] (5) The surrounding image may be gray-converted to generate a second multi-grayscale image, and a colorless multi-grayscale image, i.e., a first intermediate image, may be generated based on the ratio of at least two color components including a green component among the color components of the color image, i.e., the surrounding image. The second intermediate image may be generated based on the brightness difference between the first intermediate image and the second multi-grayscale image, and the first multi-grayscale image may be generated by inverting the brightness of the second intermediate image. Thus, the second intermediate image may be generated in such a way that the brightness difference of the region with a small ratio of the green component is reduced.

[0104] (6) The third intermediate image can be generated by inverting the brightness of the second intermediate image, and the first multi-grayscale image can be generated by weighted averaging the pixel values ​​of the third intermediate image and the second multi-grayscale image. Thus, the first multi-grayscale image can be generated in such a way that the brightness value of the area with a large proportion of green components is larger than that of the area with a small proportion of green components.

[0105] Description of Reference Numerals

[0106] 1 Vehicle; 10 Driving assistance device; 16 Controller.

Claims

1. A parking assistance method for assisting a vehicle in parking at a target parking position, the parking assistance method comprising: Extracting feature points around the target parking position from an image obtained by photographing the surroundings of the host vehicle in advance as learned feature points, and storing them in a storage device; When the host vehicle is moved to the target parking position, photographing the surroundings of the host vehicle to acquire an image; Extracting feature points around the host vehicle from the image around the host vehicle as surrounding feature points; Calculating the relative position of the host vehicle relative to the target parking position based on the relative position relationship between the learned feature points and the target parking position and the relative position relationship between the surrounding feature points and the host vehicle; Calculating a target driving trajectory from the current position of the host vehicle to the target parking position based on the calculated relative position of the host vehicle relative to the target parking position, and assisting the movement of the host vehicle along the target driving trajectory; When extracting at least one of the learned feature points and the surrounding feature points, namely the object feature point, The surrounding image is converted into a first multi-grayscale image in such a manner that the brightness difference between an area with a small proportion of green components and an area with a large proportion of green components in the surrounding image, which is a color image obtained by photographing the surroundings of the vehicle, is increased. The object feature points are extracted from the first multi-grayscale image.

2. The parking assistance method according to claim 1, characterized in that: Any one of the learned feature points and the surrounding feature points is extracted from an image captured when infrared rays are irradiated around the vehicle, and any other one of the learned feature points and the surrounding feature points is extracted from the first multi-grayscale image.

3. The parking assistance method according to claim 1, characterized in that: The surrounding image is converted into the first multi-grayscale image in such a manner that the brightness value of a vegetation area in the surrounding image is increased as an area having a large proportion of a green component in the surrounding image.

4. The parking assistance method according to any one of claims 1 to 3, characterized in that: The first multi-grayscale image is generated based on the ratio of at least two color components including a green component among the color components of the color image, ie, the surrounding image.

5. The parking assistance method according to any one of claims 1 to 3, characterized in that: Performing grayscale conversion on the surrounding image to generate a second multi-grayscale image, A colorless multi-grayscale image, namely a first intermediate image, is generated based on the ratio of at least two color components including a green component among the color components of the color image, namely the surrounding image. generating a second intermediate image based on a brightness difference between the first intermediate image and the second multi-grayscale image, The first multi-grayscale image is generated by inverting the brightness of the second intermediate image.

6. The parking assistance method according to any one of claims 1 to 3, characterized in that: Performing grayscale conversion on the surrounding image to generate a second multi-grayscale image, A colorless multi-grayscale image, namely a first intermediate image, is generated based on the ratio of at least two color components including a green component among the color components of the color image, namely the surrounding image. generating a second intermediate image based on a brightness difference between the first intermediate image and the second multi-grayscale image, By inverting the brightness of the second intermediate image, a third intermediate image is generated, The first multi-grayscale image is generated by performing weighted averaging on the pixel values ​​of the third intermediate image and the second multi-grayscale image.

7. A parking assist device, characterized in that: have: A camera device for photographing the surroundings of the vehicle; Storage device; a controller that extracts feature points around a target parking position from an image obtained by photographing the surroundings of the host vehicle using the photographing device in advance as learned feature points and stores the extracted feature points in the storage device; photographs the surroundings of the host vehicle to acquire an image when the host vehicle is moved to the target parking position; extracts feature points around the host vehicle from the image around the host vehicle as surrounding feature points; calculates a relative position of the host vehicle with respect to the target parking position based on a relative positional relationship between the learned feature points and the target parking position and a relative positional relationship between the surrounding feature points and the host vehicle; and calculates a target driving trajectory from the current position of the host vehicle to the target parking position based on the calculated relative position of the host vehicle with respect to the target parking position, thereby assisting the host vehicle in moving along the target driving trajectory; When the controller extracts at least one of the learned feature points and the surrounding feature points, namely, the object feature point, The surrounding image is converted into a first multi-grayscale image in such a manner that the brightness difference between an area with a small proportion of green components and an area with a large proportion of green components in the surrounding image, which is a color image obtained by photographing the surroundings of the vehicle, is increased. The object feature points are extracted from the first multi-grayscale image.

Images