Parking assistance method and parking assistance device

Abstract

This parking assist method includes: acquiring a peripheral image, which is a color image acquired by photographing the periphery of an own vehicle (S1); converting the peripheral image into a first multi-level image so as to increase a difference in brightness in the peripheral image between a region having a low rate of the green component and a region having a high rate of the green component (S2 to S6); extracting a feature point from the first multi-level image (S7); and calculating a relative position of the own vehicle to a target parking position on the basis of the extracted feature point (S32).

Description

[Technical Field]

[0001] The present invention relates to a parking assistance method and a parking assistance device. [Background technology]

[0002] Patent Document 1 describes a driving control device that extracts and stores feature points from images taken in the past around a target parking position, calculates the relative position of the target parking position with respect to the vehicle based on the stored target positions and the target positions extracted from images taken around the vehicle during automatic parking, and automatically moves the vehicle to the target parking position based on the calculated relative position. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2017-138664 Summary of the Invention [Problem to be solved by the invention]

[0004] If an image taken around the target parking position or an image taken around the vehicle contains a vegetation area, there is a risk that the accuracy of calculating the target parking position will decrease due to the extraction of feature points with vague contours in the vegetation area. The present invention aims to improve the accuracy of calculating a target parking position in a parking assistance system that assists in parking a vehicle at a target parking position based on feature points extracted from a captured image when the captured image includes a vegetated area. [Means for solving the problem]

[0005] According to one aspect of the present invention, there is provided a parking assistance method for assisting a host vehicle in parking a target parking position. The method includes the steps of: extracting, in advance, feature points around the target parking position from an image obtained by photographing the surroundings of the host vehicle as learned feature points and storing the learned feature points in a storage device; photographing the surroundings of the host vehicle to obtain an image when moving the host vehicle to the target parking position; extracting the feature points around the host vehicle as surrounding feature points from the image of the surroundings of the host vehicle; calculating a relative position of the host vehicle with respect to the target parking position based on the relative positional relationship between the learned feature points and the target parking position and the relative positional relationship between the surrounding feature points and the host vehicle; calculating a target driving trajectory from the current position of the host vehicle to the target parking position based on the calculated relative position of the host vehicle with respect to the target parking position, and supporting movement of the host vehicle along the target driving trajectory; and when extracting target feature points that are at least one of the learned feature points and the surrounding feature points, the surrounding image is converted into a first multi-tone image so that the brightness difference between an area with a small proportion of green components and an area with a large proportion of green components in the surrounding image, which is a color image obtained by photographing the surroundings of the host vehicle, is increased, and the target feature points are extracted from the first multi-tone image. [Effects of the Invention]

[0006] According to the present invention, in a parking assistance system that assists in parking a vehicle at a target parking position based on feature points extracted from a captured image, the accuracy of calculating the target parking position can be improved when the captured image includes a vegetated area. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram illustrating an example of a schematic configuration of a parking assistance device. [Figure 2A] FIG. 10 is an explanatory diagram of an example of a process for storing learned feature points. [Figure 2B] FIG. 10 is an explanatory diagram of an example of processing when parking assistance is performed. [Figure 3] 2 is a block diagram illustrating an example of a functional configuration of a controller in FIG. 1. FIG. [Figure 4]10 is a diagram showing an example of a grayscale image generated from a surrounding image when the illuminance around the host vehicle is equal to or greater than a predetermined threshold. FIG. [Figure 5] 10 is a diagram showing an example of a first intermediate image generated from a surrounding image when the illuminance around the host vehicle is equal to or greater than a predetermined threshold value. FIG. [Figure 6] FIG. 10 is a diagram showing an example of a second intermediate image. [Figure 7] FIG. 10 is a diagram showing an example of a multi-tone image obtained by the second method. [Figure 8] FIG. 10 is a diagram showing an example of a multi-tone image obtained by the third method. [Figure 9A] 10A and 10B are diagrams showing an example of a grayscale image generated from a surrounding image when the illuminance around the host vehicle is less than a predetermined threshold, and a partially enlarged image thereof; [Figure 9B] 5A and 5B are diagrams showing a grayscale image and a partially enlarged image of FIG. 4. [Figure 9C] 9A and 9B are diagrams showing the multi-tone image and a partially enlarged image of FIG. 8. [Figure 10A] FIG. 10 is an explanatory diagram of a first example of a process for storing learned feature points. [Figure 10B] FIG. 10 is an explanatory diagram of a first example of multi-tone image generation processing. [Figure 10C] FIG. 10 is an explanatory diagram of a second example of multi-tone image generation processing. [Figure 10D] FIG. 10 is an explanatory diagram of a third example of multi-tone image generation processing. [Figure 10E] FIG. 10 is an explanatory diagram of a second example of a process for storing learned feature points. [Figure 11A] FIG. 4 is an explanatory diagram of a first example of processing when parking assistance is performed. [Figure 11B] FIG. 10 is an explanatory diagram of a second example of processing when parking assistance is performed. DETAILED DESCRIPTION OF THE INVENTION

[0008] (composition) Referring to FIG. 1, the host vehicle 1 is equipped with a parking assistance device 10 that assists in parking the host vehicle 1 at a target parking position. The parking assistance device 10 assists the host vehicle 1 in traveling along a target traveling trajectory from the current position of the host vehicle 1 to the target parking position. For example, the host vehicle 1 may be automatically driven to travel along the target traveling trajectory of the host vehicle 1 to the target parking position (i.e., control to automatically perform all or part of traveling along the target traveling trajectory of the host vehicle 1 by controlling all or part of the steering angle, driving force, and braking force of the host vehicle). Parking of the host vehicle 1 may be assisted by displaying the target traveling trajectory and the current position of the host vehicle 1 on a display device that is visible to the occupants of the host vehicle 1.

[0009] The positioning device 11 measures the current position of the host vehicle 1. The positioning device 11 includes, for example, a Global Navigation System (GNSS) receiver. The human-machine interface (HMI) 12 is an interface device between the parking assistance device 10 and the occupant, and includes a display device, a speaker, a buzzer, and an operator. The shift switch (shift SW) 13 is a switch used by the driver or the parking assistance device 10 to switch the shift position. The external sensor 14 detects objects within a predetermined distance range from the host vehicle 1. The external sensor 14 detects the environment surrounding the host vehicle 1, such as the relative position between the host vehicle 1 and an object present around the host vehicle 1, the distance between the host vehicle 1 and the object, and the direction in which the object is located. The external sensor 14 may include, for example, a camera that captures images of the environment surrounding the host vehicle 1. To support parking assistance for the host vehicle 1 in an environment where the illuminance around the host vehicle 1 is below a predetermined threshold (for example, at night), the camera included in the external sensor 14 may be a day / night camera capable of capturing images in both the visible light and infrared light ranges. A daytime camera capable of capturing images in the visible light range and a camera capable of capturing images in the infrared range may be separately provided. In this embodiment, an example in which a day and night camera is used will be described. Hereinafter, the camera of the external environment sensor 14 will be simply referred to as a "camera." The external environment sensor 14 may include a distance measuring device such as a laser range finder, radar, or LiDAR. The vehicle sensor 15 detects various information (vehicle information) about the host vehicle 1. The vehicle sensor 15 may include, for example, a vehicle speed sensor, a wheel speed sensor, a three-axis acceleration sensor (G sensor), a steering angle sensor, a turning angle sensor, a gyro sensor, and a yaw rate sensor.

[0010] The controller 16 is an electronic control unit that performs parking assistance control of the host vehicle 1. The controller 16 includes a processor 20 and peripheral components such as a storage device 21. The storage device 21 may include a semiconductor storage device, a magnetic storage device, an optical storage device, etc. The functions of the controller 16 are realized, for example, by the processor 20 executing a computer program stored in the storage device 21. The steering actuator 18a controls the steering direction and steering amount of the steering mechanism in response to a control signal from the controller 16. The accelerator actuator 18b controls the accelerator opening of the drive device (engine, drive motor) in response to a control signal from the controller 16. The brake actuator 18c activates a braking device in response to a control signal from the controller 16.

[0011] When assisting parking of the host vehicle 1 in an environment where the illuminance around the host vehicle 1 is below a predetermined threshold, the infrared floodlight 19 irradiates the surroundings of the host vehicle 1 with infrared rays in response to a control signal from the controller 16. For example, the infrared floodlights may be infrared light-emitting diodes that are provided on the left and right sides of the host vehicle 1 and emit infrared rays diagonally downward to the left and right sides of the host vehicle 1, respectively, and irradiate the infrared rays onto the road surface around the left and right sides of the host vehicle 1.

[0012] Next, parking assistance control by the parking assistance device 10 will be described. See FIG. 2A. When using parking assistance by the parking assistance device 10, feature points are extracted from an image captured of the surroundings of the target parking position 30, where the host vehicle 1 is to be parked, and stored in advance in the storage device 21. Hereinafter, feature points stored in the storage device 21 will be referred to as "learned feature points." In FIG. 2A, circles represent learned feature points. For example, the parking assistance device 10 extracts feature points around the target parking position 30 from a surrounding image obtained by capturing an image of the surroundings of the host vehicle 1 with a camera when the host vehicle 1 is located near the target parking position 30 (e.g., when manually driving the vehicle to park at the target parking position 30). For example, edge points or points with characteristic shapes, where the brightness of adjacent pixels changes by more than a predetermined amount, such as edges or corners of targets such as road markings, road boundaries, and obstacles, are detected as feature points in the captured image captured by the camera. When storing the learned feature points in the storage device 21, for example, the driver operates a "parking position learning switch" provided as an operator of the HMI 12. When the illuminance around the vehicle 1 is below a predetermined threshold, the learned feature points are extracted from a captured image obtained by capturing an image with the infrared projector 19 irradiating infrared rays around the vehicle 1.

[0013] The parking assistance device 10 stores the relative positional relationship between the learned feature points and the target parking position 30. For example, the driver may input via the HMI 12 that the host vehicle 1 is located at the target parking position 30. The parking assistance device 10 may calculate the relative positional relationship between the learned feature points and the target parking position 30 based on the positions of the learned feature points detected when the host vehicle 1 is located at the target parking position 30. For example, the parking assistance device 10 may store the coordinates of the learned feature points and the target parking position 30 on a coordinate system (hereinafter referred to as a "map coordinate system") that uses a fixed point as a reference point. In this case, the current position on the map coordinate system measured by the positioning device 11 when the host vehicle 1 is located at the target parking position 30 may be stored as the target parking position 30. Alternatively, instead of the map coordinate system, the relative positions of the target parking position 30 with respect to each learned feature point may be stored.

[0014] 2B is an explanatory diagram of an example of processing when parking assistance is performed. The parking assistance device 10 performs parking assistance for the host vehicle 1 when the host vehicle 1 is located near the target parking position 30. For example, the parking assistance device 10 determines whether the driver has performed a shift operation to switch between forward and reverse movement of the host vehicle 1 when the host vehicle 1 is located near the target parking position 30. Reference symbol 31 is the steering position. When the shift position is switched from the drive range (D range) to the reverse range (R range) or from the R range to the D range, the parking assistance device 10 may determine that a shift operation for steering has been performed and start parking assistance. When the host vehicle 1 is at a position 33 near the target parking position 30, parking assistance may be started when the driver operates a "parking assistance activation switch" provided in the HMI 12.

[0015] The parking assistance device 10 extracts feature points around the vehicle 1 from a surrounding image obtained by capturing an image of the surroundings of the vehicle 1 with a camera. Hereinafter, the feature points around the vehicle 1 extracted when parking assistance is performed will be referred to as "surrounding feature points." In Figure 2B, triangular plots represent surrounding feature points. In addition, if the illuminance around the vehicle 1 is less than a predetermined threshold when parking assistance is being performed, surrounding feature points are extracted from the captured image obtained by the camera while the infrared floodlight 19 is irradiating infrared rays around the vehicle 1. The parking assistance device 10 compares the learned feature points stored in the storage device 21 with the surrounding feature points to verify (match) them, and associates identical feature points with each other.

[0016] The parking assistance device 10 calculates the relative position of the vehicle 1 with respect to the target parking position 30 based on the relative positional relationship between the vehicle 1 and the surrounding feature points detected when parking assistance is performed, and the relative positional relationship between the learned feature points associated with the surrounding feature points and the target parking position 30. For example, the parking assistance device 10 may calculate the position of the target parking position 30 on a coordinate system (hereinafter referred to as the "vehicle coordinate system") based on the current position of the vehicle 1. For example, if the coordinates of the learned feature points and the target parking position 30 in the map coordinate system are stored in the storage device 21, the coordinates of the target parking position 30 on the map coordinate system may be converted to coordinates on the vehicle coordinate system based on the positions of the surrounding feature points detected when parking assistance is performed and the positions of the learned feature points in the map coordinate system. Alternatively, the current position of the host vehicle 1 on the map coordinate system may be determined based on the positions of the surrounding feature points detected when parking assistance is performed and the positions of the learned feature points in the map coordinate system, and the relative position of the host vehicle 1 with respect to the target parking position 30 may be calculated from the difference between the coordinates of the host vehicle 1 and the coordinates of the target parking position 30 in the map coordinate system. The parking assistance device 10 calculates a target driving trajectory from the current position of the host vehicle 1 to the target parking position 30 based on the relative position of the host vehicle 1 with respect to the target parking position 30. For example, if the position of the host vehicle 1 at the time when parking assistance is started is a turning position 31, a trajectory 32 from the turning position 31 to the target parking position 30 is calculated. Alternatively, for example, if the position of the host vehicle 1 at the time when parking assistance is started is a position 33 near the target parking position 30, a trajectory 34 from the position 33 to the turning position 31 and a trajectory 32 from the turning position 31 to the target parking position 30 are calculated. The parking assistance device 10 performs parking assistance control of the host vehicle 1 based on the calculated target driving trajectory.

[0017] The functional configuration of the controller 16 is shown in Figure 3. When the parking position learning switch is operated, the human-machine interface control unit (HMI control unit) 40 outputs a map generation command to the map generation unit 45 to store learned feature points in the storage device 21. The HMI control unit 40 determines whether the driver has performed a shift operation to turn the car or whether the parking assist activation switch has been operated, and outputs the determination result to the parking assist control unit 41. The parking assist control unit 41 determines whether the host vehicle 1 is located near the stored target parking position 30. For example, it determines whether the distance between the host vehicle 1 and the target parking position 30 is equal to or less than a predetermined distance. When the host vehicle 1 is located near the target parking position 30 and the parking assist activation switch is operated or a shift operation to turn the car is detected, the parking assist control unit 41 sets the stored target parking position 30 as the target parking position 30 for this control and starts parking assist control.

[0018] When parking assist control is started, the parking assist control unit 41 outputs a parking position calculation command to the matching unit 47 to calculate the position of the target parking position 30 in the vehicle coordinate system. The parking assist control unit 41 also outputs a driving trajectory calculation command to the target trajectory generation unit 48 to calculate a target driving trajectory and a target vehicle speed profile along which the host vehicle 1 will travel on the target driving trajectory. The target trajectory generation unit 48 calculates the target driving trajectory and target vehicle speed profile from the current position of the host vehicle 1 to the target parking position 30 and outputs them to the parking assist control unit 41. A well-known method adopted in automatic parking devices can be applied to calculate the target driving trajectory. For example, the target driving trajectory can be calculated by connecting the current position of the host vehicle 1, via the turning position 31, to the target parking position 30 with a clothoid curve. For example, the target vehicle speed profile may be a vehicle speed profile in which the vehicle 1 accelerates from its current position to a predetermined set speed, then decelerates just before the turning position 31 and stops at the turning position 31, accelerates from the turning position 31 to the set speed, decelerates just before the target parking position 30, and stops at the target parking position 30.

[0019] The parking assist control unit 41 outputs information on the target driving trajectory and the current position of the host vehicle 1 to the HMI control unit 40. If the target driving trajectory includes a turn, it outputs information on the turn position to the HMI control unit 40. The HMI control unit 40 displays the target driving trajectory, the current position of the host vehicle 1, and the turn position on the HMI 12. The parking assist control unit 41 also outputs a steering control command to the steering control unit 49 to control the steering of the host vehicle 1 so that the host vehicle 1 travels along the calculated target travel trajectory. The parking assist control unit 41 also outputs a vehicle speed control command to the vehicle speed control unit 50 to control the vehicle speed of the host vehicle 1 according to the calculated target vehicle speed profile. The image conversion unit 42 converts the captured image from the camera into an overhead image (around view monitor image) viewed from a virtual viewpoint directly above the host vehicle 1. The image conversion unit 42 converts the captured image into an overhead image at predetermined intervals (for example, every time the host vehicle 1 travels a predetermined distance (for example, 50 cm) or a predetermined time (for example, 1 second)), and accumulates the converted overhead images along the travel path of the host vehicle 1 to generate a surrounding image such as that shown in FIGS. 2A and 2B. The host position calculation unit 43 calculates the current position of the host vehicle 1 on a map coordinate system by dead reckoning based on vehicle information output from the vehicle sensor 15.

[0020] When storing learned feature points in the storage device 21, the feature point detection unit 44 detects the learned feature points and their image feature amounts from the surrounding image output from the image conversion unit 42. When performing parking assistance control, the surrounding feature points and their image feature amounts are detected. Hereinafter, the learned feature points and surrounding feature points may be collectively referred to as "feature points." The feature point detection unit 44 detects the feature points and their image feature amounts from the surrounding image output from the image conversion unit 42. Methods such as SIFT, SURF, ORB, BRIAK, KAZE, and AKAZE can be used to detect feature points and calculate image feature amounts.

[0021] The image of the surroundings of the vehicle 1 may include a vegetation area (i.e., an area where plants are growing). In a vegetation area, a difference in brightness occurs between the areas where light hits the plants and the shaded areas where light does not hit, and minute feature points with vague contours may be extracted. Because the shadows of plant leaves change depending on the light source (e.g., the sun), there is a risk that the accuracy of calculating the target parking position may decrease. Furthermore, in an image captured without irradiating infrared rays in an environment where the illuminance around the vehicle 1 is equal to or greater than a predetermined threshold (e.g., daytime), the brightness of the vegetation area is generally lower than that of the asphalt, whereas in an image captured with infrared irradiation, the brightness of the vegetation area is higher. For this reason, if either the learned feature points or the surrounding feature points are extracted from the image captured with infrared irradiation and the other is extracted from the image captured without infrared irradiation, there is a risk that the calculation accuracy of the target parking position will decrease.

[0022] Therefore, when extracting learned feature points and surrounding feature points from an image captured without irradiating infrared light around the vehicle 1 in an environment where the illuminance around the vehicle 1 is equal to or higher than a predetermined threshold, the parking assistance device 10 of the embodiment converts the surrounding image into a colorless multi-tone image so that the brightness value of areas with a large proportion of green components in the color surrounding image obtained by capturing the area around the vehicle 1 is increased compared to areas with a small proportion of green components, and extracts feature points from the converted multi-tone image. For example, the surrounding image may be converted into a multi-tone image so that the luminance value of an area with a large proportion of green components is saturated (that is, so that the luminance value reaches the upper limit of the image luminance). In this way, since the vegetation area of ​​the color surrounding image is an area with a high proportion of green components, increasing the brightness value of the area with a high proportion of green components can reduce the brightness difference between the areas where light hits the plants and the shadow areas where light does not hit. As a result, it becomes difficult to extract vague contour feature points in the vegetation area, improving the accuracy of calculating the target parking position. Also, the brightness of the vegetation area can be made higher than that of the asphalt area in an image captured without infrared irradiation. As a result, the brightness characteristics of the vegetation area and the asphalt area are similar between the image captured without infrared irradiation and the image captured with infrared irradiation, improving the accuracy of calculating the target parking position.

[0023] The following describes in detail how the feature point detection unit 44 detects feature points. In an environment where the illuminance around the vehicle 1 is equal to or greater than a predetermined threshold, the feature point detection unit 44 extracts feature points from an image of the surroundings captured without irradiating it with infrared light. To reduce the difference in brightness between the lighted and shaded areas of the plant, the feature point detection unit 44 converts the surrounding image into a multi-tone image so that the brightness values ​​of areas with a large proportion of green components in the color surrounding image are increased compared to areas with a small proportion of green components in the surrounding image. This multi-tone image is an example of the "first multi-tone image" described in the claims.

[0024] As a first method, the surrounding image may be converted into a multi-tone image so that the luminance value of an area having a large proportion of green components is increased. For example, the feature point detection unit 44 converts the surrounding image into a multi-tone image based on the ratio of at least two color components, including the G component Cg, among the R component (red component) Cr, the G component (green component) Cg, and the B component (blue component) Cb of the color surrounding image, so that the luminance value of an area having a proportion of the G component (green component) Cg greater than a predetermined value becomes saturated (i.e., reaches the upper limit of image luminance). Note that the luminance value of an area having a proportion of Cg greater than a predetermined value may be increased based on the ratio of the two color components rather than the upper limit of image luminance.

[0025] The second method involves (1) generating a grayscale image (corresponding to the second multi-tone image) of the surrounding image, (2) converting the surrounding image into a first intermediate image so that the brightness values ​​of the vegetation areas are reduced, (3) generating a second intermediate image based on the brightness difference between the grayscale image and the first intermediate image, and (4) inverting the brightness of the second intermediate image. These steps (1) to (4) result in a colorless multi-tone image in which the brightness values ​​of areas with a large proportion of green components are increased compared to areas with a small proportion of green components. First, the feature point detection unit 44 generates a grayscale image of the surrounding image by performing a normal grayscale conversion on the surrounding image, which is a color image. FIG. 4 is a diagram showing an example of a grayscale image. For example, the feature point detection unit 44 may determine the brightness value of the grayscale image by weighting and combining the R component Cr, the G component Cg, and the B component Cb of the color surrounding image using a predetermined weighting coefficient. The grayscale image is an example of the "second multi-tone image" described in the claims. The area Rv surrounded by a dashed line indicates a vegetation area. In an image captured without irradiating the surroundings of the vehicle 1 with infrared light, the brightness of the vegetation area Rv is characteristically lower than the brightness of the surrounding asphalt area. Next, the feature point detection unit 44 converts the color surrounding image into a first intermediate image, which is a colorless multi-tone image, such that the brightness value of vegetation areas with a high proportion of the G component Cg is reduced compared to areas of the surrounding image with a low proportion of the G component Cg.

[0026] 5 is a diagram showing an example of the first intermediate image. For example, the feature point detection unit 44 may generate the first intermediate image based on a ratio of at least two color components, including the G component Cg, among the R component Cr, the G component Cg, and the B component Cb of the surrounding image. For example, if the maximum value of each of the R component Cr, the G component Cg, and the B component Cb is 255, the feature point detection unit 44 may determine the luminance value Ck of each pixel of the first intermediate image based on the following equations (1) and (2).

number

[0027] Next, the feature point detection unit 44 generates a second intermediate image based on the brightness difference between the grayscale image of Fig. 4 and the first intermediate image of Fig. 5. For example, the feature point detection unit 44 generates a difference image as the second intermediate image by subtracting the brightness value of each pixel of the grayscale image from the brightness value of each pixel of the first intermediate image. Fig. 6 is a diagram showing an example of the second intermediate image. By subtracting the brightness value of each pixel of the grayscale image from the brightness value of each pixel of the first intermediate image, the brightness difference in areas other than the vegetation area Rv (for example, asphalt areas) is reduced in the second intermediate image.

[0028] Next, the feature point detection unit 44 inverts the brightness of the second intermediate image to generate a colorless multi-tone image in which the brightness values ​​of areas with a large proportion of green components are increased compared to areas with a small proportion of green components. Fig. 7 is a diagram showing an example of a multi-tone image obtained by the second method. In the image of Fig. 7, the vegetation region Rv is almost uniformly bright (the brightness is almost uniformly increased), and areas other than the vegetation region Rv (for example, the asphalt region) are almost uniformly dark (the brightness is almost uniformly decreased).

[0029] In the third method, the multi-tone image obtained by the second method (a brightness-inverted image of the second intermediate image) is used as the third intermediate image, and a multi-tone image is generated by taking a weighted average of the pixel values ​​of the third intermediate image and the pixel values ​​of the grayscale image. 8 is a diagram showing an example of a multi-tone image obtained by the third method. By adding pixel values ​​of the third intermediate image, the brightness value of the vegetation region Rv increases compared to the brightness values ​​of other regions.

[0030] On the other hand, in an environment where the illuminance around the vehicle 1 is below a predetermined threshold, the feature point detection unit 44 extracts feature points from the surrounding image captured while irradiating infrared light from the infrared projector 19. In this case, the feature point detection unit 44 generates a grayscale image of the surrounding image by performing a normal grayscale conversion on the surrounding image, which is a color image. The feature point detection unit 44 extracts feature points from the grayscale image. FIG. 9A shows an example of a grayscale image of the surrounding image captured while irradiating infrared light, and an enlarged image of the boundary between the vegetation area Rv and the asphalt area. In the image captured while irradiating infrared light, the vegetation area Rv has a higher brightness than the asphalt area.

[0031] 9B shows a grayscale image (grayscale image in FIG. 4) of the surrounding image captured without infrared irradiation and a partially enlarged image. In the image captured without infrared irradiation, the brightness of the vegetation area Rv is lower than that of the asphalt area, unlike the image captured with infrared irradiation. 9C shows a multi-tone image (the multi-tone image of FIG. 8) generated so that the brightness value of the vegetation region Rv is higher than that of the other regions, along with a partially enlarged image. In the multi-tone image of FIG. 9C, the brightness of the vegetation region Rv is higher than that of the asphalt region, and the brightness characteristics of the vegetation region Rv and the asphalt region can be made similar to the brightness characteristics in the grayscale image (FIG. 9A) when infrared light is irradiated.

[0032] See Fig. 3. When storing the learned feature points in the storage device 21, the driver operates the parking position learning switch and manually parks the vehicle 1 at the target parking position. The map generation unit 45 receives a map generation command from the HMI control unit 40. The map generation unit 45 stores feature point information including the feature points output from the feature point detection unit 44, the current position of the vehicle 1 synchronized therewith, and the feature amounts of the feature points as learned feature points in the storage device 21, and generates map data 46. The position of the feature points in the map coordinate system may be calculated based on the current position of the vehicle 1 and stored as feature point information. Furthermore, when the driver inputs to the parking assistance device 10 that the current position of the vehicle 1 is the target parking position 30, the map generation unit 45 receives the current position of the vehicle 1 on the map coordinate system from the positioning device 11 or the self-position calculation unit 43, and stores it in the map data 46 as the target parking position 30. In other words, the relative positional relationship between the target parking position 30 and the multiple characteristic points is stored as the map data 46.

[0033] Thereafter, when the parking assist control unit 41 starts parking assist control, the matching unit 47 receives a parking position calculation command from the parking assist control unit 41. The matching unit 47 matches the feature point information stored as learned feature points in the map data 46 with the feature point information of the surrounding feature points output from the feature point detection unit 44 when parking assistance is performed, and associates the feature point information of the same feature points with each other. The matching unit 47 calculates the current relative position of the vehicle 1 with respect to the target parking position 30 based on the relative positional relationship between the surrounding feature points and the vehicle 1 and the relative positional relationship between the learned feature points associated with the surrounding feature points and the target parking position 30. For example, if the surrounding feature points are (x i ,y i ) and the surrounding feature points (x i ,y i ) are the trained feature points associated with each of (x mi ,y mi ) (i=1 to N). The matching unit 47 calculates the affine transformation matrix M affine Calculate.

[0034]

number

[0035] The abutment 47 is the position of the target parking position 30 on the map coordinate system stored in the map data 46 (targetx m ,targety m ) is converted into a position (targetx, targety) in the vehicle coordinate system.

number

[0036] When the target trajectory generation unit 48 receives a driving trajectory calculation command from the parking assist control unit 41, it calculates a target driving trajectory from the current position of the host vehicle 1 on the vehicle coordinate system to the target parking position 30, and a target vehicle speed profile for the host vehicle 1 to travel on the target driving trajectory. When the steering control unit 49 receives a steering control command from the parking assist control unit 41, it controls the steering actuator 18a so that the host vehicle 1 travels along the target driving trajectory. When the vehicle speed control unit 50 receives a vehicle speed control command from the parking assist control unit 41, it controls the accelerator actuator 18b and the brake actuator 18c so that the vehicle speed of the host vehicle 1 changes in accordance with the target vehicle speed profile. When the host vehicle 1 reaches the target parking position 30 and the parking assist control is completed, the parking assist control unit 41 activates the parking brake 17 and switches the shift position to the parking range (P range).

[0037] (operation) FIG. 10A is an explanatory diagram of an example of a process for storing learned feature points in an environment where the illuminance around the vehicle 1 is equal to or greater than a predetermined threshold. In step S1, the image conversion unit 42 acquires a surrounding image by converting the image captured by the camera into an overhead image seen from a virtual viewpoint directly above the vehicle 1. In step S2, the feature point detection unit 44 executes a multi-tone image generation process. 10B is an explanatory diagram of a first example of the multi-tone image generation process. In step S10, the feature point detection unit 44 converts the surrounding image into a multi-tone image so that the brightness value of an area with a high proportion of green components is increased.

[0038] FIG. 10C is an explanatory diagram of a second example of the multi-tone image generation process. In step S20, the feature point detection unit 44 converts the surrounding image into a normal grayscale image. In step S21, the feature point detection unit 44 converts the surrounding image into a first intermediate image. In step S22, the feature point detection unit 44 generates a second intermediate image. In step S23, the feature point detection unit 44 generates a multi-tone image by inverting the luminance of the second intermediate image. Fig. 10D is an explanatory diagram of a third example of the multi-tone image generation process. The processes of steps S30 to S32 are the same as steps S20 to S22 in Fig. 10C. In step S33, the feature point detection unit 44 generates a third intermediate image by inverting the luminance of the second intermediate image. In step S34, the feature point detection unit 44 generates a multi-tone image by taking a weighted average of the third intermediate image and the grayscale image. 10A, in step S3, the feature point detection unit 44 extracts feature points from the multi-tone image, and in step S4, the feature point detection unit 44 stores the extracted feature points in the storage device 21 as learned feature points.

[0039] FIG. 10E is an explanatory diagram of an example of a process for storing learned feature points in an environment where the illuminance around the vehicle 1 is less than a predetermined threshold. In step S40, the controller 16 causes the infrared projector 19 to emit infrared rays to the surroundings of the vehicle 1. The processing in step S41 is the same as the processing in step S1 of FIG. 10A. In step S42, the feature point detection unit 44 converts the surrounding image into a normal grayscale image. In step S43, the feature point detection unit 44 extracts feature points from the grayscale image. The processing in step S44 is the same as the processing in step S4 of FIG. 10A.

[0040] FIG. 11A is an explanatory diagram of an example of processing when parking assistance is performed in an environment where the illuminance around the host vehicle 1 is equal to or greater than a predetermined threshold. The processing in steps S50 and S51 is the same as steps S1 and S2 in FIG. 10A. In step S52, the feature point detection unit 44 extracts surrounding feature points from the multi-tone image. In step S53, the parking assistance control unit 41 determines whether the distance between the host vehicle 1 and the target parking position 30 is equal to or less than a predetermined distance. If it is equal to or less than the predetermined distance (step S53: Y), the processing proceeds to step S54. If it is not equal to or less than the predetermined distance (step S53: N), the processing returns to step S50. In step S54, the parking assistance control unit 41 determines whether a shift operation for turning around has been detected. If a shift operation has been detected (step S54: Y), the processing proceeds to step S56. If a shift operation has not been detected (step S54: N), the processing proceeds to step S55. In step S55, the parking assist control unit 41 determines whether the parking assist activation switch has been operated by the driver. If the parking assist activation switch has been operated (step S55: Y), the process proceeds to step S56. If the parking assist activation switch has not been operated (step S55: N), the process returns to step S50.

[0041] In step S56, the matching unit 47 reads the learned feature points from the storage device 21. In step S57, the matching unit 47 matches the surrounding feature points with the learned feature points. In step S58, the matching unit 47 calculates the target parking position 30 based on the matched feature points. In step S59, the target trajectory generation unit 48 calculates the target driving trajectory and the target vehicle speed profile. In step S60, the steering control unit 49 and the vehicle speed control unit 50 control the steering actuator 18a, the accelerator actuator 18b, and the brake actuator 18c based on the target driving trajectory and the target vehicle speed profile. In step S61, when the parking assist control is completed, the parking assist control unit 41 activates the parking brake 17 and switches the shift position to P range.

[0042] FIG. 11B is an explanatory diagram of an example of processing when parking assistance is performed in an environment where the illuminance around the vehicle 1 is less than a predetermined threshold. The processes of steps S70 to S72 are the same as steps S40 to S42 in Fig. 10E. In step S73, the feature point detection unit 44 extracts surrounding feature points from the grayscale image. The processes of steps S74 to S82 are the same as the processes of steps S53 to S61 in Fig. 11A.

[0043] (Effects of the embodiment) (1) In the parking assistance method, when extracting target feature points, which are at least one of learned feature points and surrounding feature points, the surrounding image, which is a color image obtained by photographing the surroundings of the vehicle 1, is converted into a first multi-tone image so as to increase the difference in brightness between areas with a small proportion of green components and areas with a large proportion of green components, and the target feature points are extracted from the first multi-tone image. This makes it difficult to extract ambiguous feature points in vegetation areas, thereby improving the calculation accuracy of the target parking position.

[0044] (2) Either the learned feature points or the surrounding feature points may be extracted from an image captured when infrared rays are irradiated around the vehicle 1, and the other of the learned feature points or the surrounding feature points may be extracted from the first multi-tone image. This allows for improved calculation accuracy of the target parking position even if either the learned feature points or the surrounding feature points are extracted from an image captured with infrared rays irradiated and the other is extracted from an image captured without infrared rays irradiated.

[0045] (3) The surrounding image may be converted into a first multi-tone image so that the brightness value of the vegetation area in the surrounding image, which is an area with a large proportion of green components, is increased. This makes it difficult to extract ambiguous feature points in the vegetation area, thereby improving the accuracy of calculating the target parking position. (4) The first multi-tone image may be generated based on a ratio of at least two color components including a green color component among the color components of the surrounding image, which is a color image. This allows the first multi-tone image to be generated so that the luminance value in an area with a large proportion of green components is higher than in an area with a small proportion of green components.

[0046] (5) The second multi-tone image may be generated by grayscale conversion of the surrounding image, a first intermediate image that is a colorless multi-tone image based on a ratio of at least two color components including a green component among the color components of the surrounding image that is a color image, a second intermediate image that is a colorless multi-tone image based on a luminance difference between the first intermediate image and the second multi-tone image, and the first multi-tone image may be generated by inverting the luminance of the second intermediate image. This allows the second intermediate image to be generated so that the luminance difference in areas with a small proportion of the green component is reduced. (6) A third intermediate image may be generated by inverting the luminance of the second intermediate image, and a first multi-tone image may be generated by weighted averaging the pixel values ​​of the third intermediate image and the second multi-tone image, thereby generating a first multi-tone image such that the luminance value of an area with a large proportion of green components is higher than that of an area with a small proportion of green components. [Explanation of symbols]

[0047] 1... host vehicle, 10... driving assistance device, 16... controller

Claims

1. A parking assistance method for assisting a vehicle in parking at a target parking position, comprising: extracting feature points around the target parking position from an image obtained by photographing the surroundings of the host vehicle as learned feature points in advance and storing the learned feature points in a storage device; capturing an image of the surroundings of the vehicle when moving the vehicle to the target parking position; extracting feature points around the host vehicle as surrounding feature points from an image of the surroundings of the host vehicle; Calculating a relative position of the host vehicle with respect to the target parking position based on a relative positional relationship between the learned feature points and the target parking position and a relative positional relationship between the surrounding feature points and the host vehicle; calculating a target driving trajectory from the current position of the host vehicle to the target parking position based on the calculated relative position of the host vehicle with respect to the target parking position, and supporting movement of the host vehicle along the target driving trajectory; When extracting a target feature point which is at least one of the learned feature points and the surrounding feature points, converting the surrounding image, which is a color image obtained by photographing the surroundings of the host vehicle, into a first multi-tone image so that a luminance difference between an area having a small proportion of green components and an area having a large proportion of green components in the surrounding image is increased; A parking assistance method comprising extracting the target feature points from the first multi-tone image.

2. 2. The parking assistance method according to claim 1, wherein either the learned feature points or the surrounding feature points are extracted from an image captured when infrared rays are irradiated around the host vehicle, and the other of the learned feature points and the surrounding feature points is extracted from the first multi-tone image.

3. The parking assistance method according to claim 1, characterized in that the surrounding image is converted into the first multi-tone image so that the brightness value of a vegetation area in the surrounding image, which is an area with a high proportion of green components, is increased.

4. The parking assistance method according to any one of claims 1 to 3, characterized in that the first multi-tone image is generated based on a ratio of at least two color components including a green component among the color components of the surrounding image, which is a color image.

5. grayscale converting the ambient image to generate a second multi-tone image; generating a first intermediate image, which is a colorless multi-tone image, based on a ratio of at least two color components including a green component among the color components of the surrounding image, which is a color image; generating a second intermediate image based on a luminance difference between the first intermediate image and the second multi-tone image; generating the first multi-tone image by inverting the luminance of the second intermediate image; 4. The parking assistance method according to claim 1, wherein the vehicle is driven in a direction parallel to the road surface.

6. grayscale converting the ambient image to generate a second multi-tone image; generating a first intermediate image, which is a colorless multi-tone image, based on a ratio of at least two color components including a green component among the color components of the surrounding image, which is a color image; generating a second intermediate image based on a luminance difference between the first intermediate image and the second multi-tone image; generating a third intermediate image by inverting the luminance of the second intermediate image; 4. The parking assistance method according to claim 1, wherein the first multi-tone image is generated by calculating a weighted average of pixel values ​​of the third intermediate image and the second multi-tone image.

7. an imaging device that captures images of the surroundings of the vehicle; A storage device; a controller that extracts, in advance, feature points around a target parking position from an image obtained by photographing the surroundings of the host vehicle with the imaging device and stores the extracted feature points in the storage device as learned feature points; photographs the surroundings of the host vehicle to obtain an image when the host vehicle is moved to the target parking position, extracts feature points around the host vehicle from the image of the surroundings of the host vehicle as surrounding feature points; calculates a relative position of the host vehicle with respect to the target parking position based on a relative positional relationship between the learned feature points and the target parking position and a relative positional relationship between the surrounding feature points and the host vehicle; calculates a target driving trajectory from the current position of the host vehicle to the target parking position based on the calculated relative position of the host vehicle with respect to the target parking position; and assists the movement of the host vehicle along the target driving trajectory; When extracting a target feature point which is at least one of the learned feature points and the surrounding feature points, the controller: converting the surrounding image, which is a color image obtained by photographing the surroundings of the host vehicle, into a first multi-tone image so that a luminance difference between an area having a small proportion of green components and an area having a large proportion of green components in the surrounding image is increased; A parking assistance device characterized in that the target feature points are extracted from the first multi-tone image.

Images