Parking assistance device and parking assistance method

Abstract

To provide a parking support device, parking support method, and parking support program capable of selecting a feature point so as to improve accuracy in automatic parking.SOLUTION: A parking support device includes: an image acquisition unit acquiring camera images from cameras capturing respective different directions around a vehicle; a feature point detection unit extracting feature points from the camera images; a feature point selection unit evaluating the feature points and selecting feature points to be registered when performing learning travel where the vehicle is manually parked and a parking route and a parking position of the vehicle are registered in a map; and a vehicle control unit causing the vehicle to park on the basis of the map when the vehicle is automatically parked. The feature point selection unit varies priority for registration of the feature points or the number of the feature points for registration according to a position on the parking route, or selects, at the position on the parking route, the feature point to be registered on the map on the basis of a position of the camera or a relative position of the feature point with respect to an optical axis direction of the camera.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present disclosure relates to a parking assistance device. and Parking assistance By law Regarding. [Background technology]

[0002] Patent Document 1 discloses a method for registering only characteristic points around the parking position on a map, and not registering characteristic points near the parking start position on a map. Patent Document 2 discloses a method for registering more characteristic points on a map the closer the parking position is.

[0003] The technologies in Patent Documents 1 and 2 focus on the fact that the closer you are to the parking position, the more accurate the position estimation needs to be, and aim to reduce the map capacity and the amount of calculation required when automatically parking by eliminating or reducing the number of feature points registered on the map for positions farther from the parking position. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2022-114526 [Patent Document 2] Japanese Patent Application Publication No. 2018-75866 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technologies of Patent Documents 1 and 2 do not take into consideration the fact that the obtained accuracy varies depending on the angle of the camera with respect to the optical axis direction, that the difference in elevation between the feature point and the camera affects the position accuracy, etc., and therefore the position accuracy may decrease even at the parking position.Furthermore, they do not consider the need for position accuracy even at positions away from the parking position, such as when turning around at a position to avoid an obstacle when backing into a parking space, or when passing by an obstacle on the way to the turning around position.

[0006] Furthermore, the method does not take into consideration the fact that the required accuracy differs between the longitudinal positional accuracy of the vehicle and the lateral positional accuracy, and that the direction in which a feature point contributes to positional accuracy differs depending on the orientation of the feature point relative to the vehicle, and therefore it may not be possible to properly register a feature point in a position requiring accuracy, which is advantageous for obtaining positional accuracy in the direction requiring accuracy, on a map.

[0007] An object of the present disclosure is to provide a parking assistance device, a parking assistance method, and a parking assistance program that are capable of selecting feature points to improve the accuracy of automatic parking. [Means for solving the problem]

[0008] In order to solve the above problems, one aspect of a parking assistance device according to the present disclosure includes an image acquisition unit that acquires camera images from cameras that capture different directions around the vehicle, a feature point detection unit that extracts feature points from the camera images, a feature point selection unit that evaluates the feature points and selects the feature points to be registered on the map during a learning drive in which the vehicle is manually parked and the parking route and parking position are registered on a map, and a vehicle control unit that parks the vehicle based on the map during automatic parking, wherein the feature point selection unit varies the priority for registering feature points or the number of feature points to be registered depending on the position on the parking route, or selects the feature points to be registered on the map based on the position of the camera or the relative position of the feature points with respect to the optical axis direction of the camera at the position on the parking route. The feature point selection unit evaluates the feature points based on the distance between the position on the parking route and the feature points, and when the position on the parking route is a position with a low priority for registering the feature points or a position with a small number of registrations, the feature point selection unit preferentially registers the feature points with a longer distance on the map compared to when the position on the parking route is a position with a high priority for registering the feature points or a position with a large number of registrations. do.

[0009] Furthermore, one aspect of a parking assistance method according to the present disclosure includes the steps of acquiring camera images from cameras each capturing different directions around a vehicle, extracting feature points from the camera images, manually parking the vehicle and during a learning drive in which the parking route and parking position are registered on a map, evaluating the feature points and selecting the feature points to be registered on the map, and during automatic parking, parking the vehicle based on the map, wherein the step of selecting the feature points varies the priority of registering the feature points or the number of feature points to be registered depending on the position on the parking route, or selects the feature points to be registered on the map based on the position of the camera or the relative position of the feature points with respect to the optical axis direction of the camera at the position on the parking route. The step of selecting the feature points evaluates the feature points based on the distance between the position on the parking route and the feature points, and when the position on the parking route is a position where the priority for registering the feature points is low or the number of registered feature points is small, the feature points with a longer distance are registered on the map with higher priority than when the position is a position where the priority for registering the feature points is high or the number of registered feature points is large. do. [Effects of the Invention]

[0011] According to the present disclosure, feature points can be selected to improve the accuracy of automatic parking. [Brief explanation of the drawings]

[0012] [Figure 1] 3A and 3B are diagrams illustrating an example of feature point detection performed by the parking assistance device according to the present embodiment. [Figure 2] FIG. 1 is a diagram illustrating automatic parking using self-position estimation according to the present embodiment. [Figure 3] 1 is a diagram showing a vehicle to which the parking assistance device according to the present embodiment can be applied. [Figure 4] 1 is a block diagram showing a parking assistance device according to an embodiment of the present invention; [Figure 5] 1 is a diagram illustrating a hardware configuration of a parking assistance device according to an embodiment of the present invention. [Figure 6] FIG. 2 is a diagram illustrating a positional relationship between a vehicle and an object. [Figure 7] FIG. 10 is a diagram illustrating the influence of the angle relative to the optical axis direction of the camera on accuracy. [Figure 8] FIG. 10 is a diagram illustrating the effect of the difference in height between the camera and the feature points on accuracy. [Figure 9]FIG. 10 is a diagram illustrating the positions of feature points and the sensitivities in the front-rear and left-right directions of the vehicle. [Figure 10] FIG. 10 is a diagram illustrating the relationship between the distance from the vehicle path to a feature point and sensitivity. [Figure 11] FIG. 10 is a diagram illustrating changes in sensitivity of feature points. [Figure 12] FIG. 10 is a diagram illustrating the positions of feature points and sensitivity in the left-right direction. [Figure 13] FIG. 10 is a diagram showing a case where a feature point is present on the path of a vehicle. [Figure 14] FIG. 10 is a diagram showing changes in sensitivity of feature points when a vehicle moves in the X direction. [Figure 15] FIG. 10 is a diagram illustrating the positions of feature points and the sensitivity of the vehicle body posture. [Figure 16] 10A and 10B are diagrams illustrating a state in which the orientation or position of the camera changes. [Figure 17] FIG. 1 is a diagram illustrating the position of a vehicle requiring high accuracy in self-location estimation. [Figure 18] FIG. 10 is a diagram showing the arrangement of feature points around an end point. [Figure 19] FIG. 10 is a diagram showing the arrangement of feature points around an end point. [Figure 20] FIG. 10 is a diagram illustrating feature points around a straight line section. [Figure 21] FIG. 10 is a diagram showing a posture convergence section in which the occupant aligns the vehicle in the left-right direction by steering. [Figure 22] FIG. 10 is a diagram showing a posture convergence section in which the occupant aligns the vehicle in the left-right direction by steering. [Figure 23] FIG. 10 is a diagram showing a turning section requiring precision in the left-right direction. [Figure 24] FIG. 10 is a diagram illustrating sensitivity in the left and right directions when the vehicle is turning. [Figure 25] FIG. 10 is a diagram illustrating sensitivity in the left and right directions when the vehicle is turning. [Figure 26] FIG. 1 is a diagram showing a situation where a vehicle requires accuracy in the left-right direction. [Figure 27] 10 is a flowchart of map generation according to the present embodiment. [Figure 28] FIG. 10 is a diagram illustrating the adjustment of end points. [Figure 29] FIG. 10 is a diagram illustrating an example of endpoints and score allocation to sections. [Figure 30] FIG. 10 is a diagram illustrating feature point registration at an end point. [Figure 31] FIG. 10 is a diagram showing a range of feature points to be registered in a curved section. [Figure 32] FIG. 10 is a diagram showing a range of feature points to be registered in a straight line section. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that each of the embodiments described below represents a specific example of the present disclosure. Therefore, the components, the arrangement and connection of each component, and each step and the order of each step shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components not recited in independent claims will be described as optional components.

[0014] Furthermore, each drawing is a schematic diagram and is not necessarily a precise illustration. In each drawing, substantially the same components are denoted by the same reference numerals, and redundant explanations will be omitted or simplified.

[0015] 1 is a diagram showing an example of feature point detection performed by the parking assistance device in this embodiment. With learning-based automatic parking, if you manually park and store the starting position and parking route, you can automatically drive and park along the same parking route simply by activating the automatic parking function at the same starting position.

[0016] More specifically, a vehicle 1 equipped with a parking assistance device generates a map including feature points during manual parking (called learning driving), and during automatic parking, the vehicle 1 uses the map to estimate the position and attitude of the vehicle 1 while driving. The feature points registered on the map are points that represent the subject captured in the camera image, and are points that can identify the position of the subject (image). Figure 1 shows how the image of the subject is captured by a right camera 2a (described later) mounted on the right side mirror of the vehicle 1. The image captured by the camera is called a camera image.

[0017] A feature point is a point in an image captured by a camera, the position of which can be identified. For example, as shown in FIG. 1, if there is a linear image 3, its end points may be used as feature points; if there is a circular image 4, its center point may be used as feature points; and if there is a polygonal image 5, its corners may be used as feature points. Since the camera image may be a color image or a black-and-white image, the parking assistance system may use points of change in brightness or points of change in hue as feature points. Since the position of a feature point in a camera image depends on the orientation of the feature point relative to the vehicle 1, the parking assistance system can calculate the orientation of the feature point from the camera image.

[0018] As the vehicle 1 moves forward and the position of the right camera 2a moves from point A to point B, the orientation of the feature points changes. At this time, the position of each feature point can be identified by applying the principles of triangulation to each feature point using line segment AB as the baseline. Many pairs of the feature points' characteristics (color and shape) and their positions (coordinates) are registered on the map.

[0019] FIG. 2 is a diagram showing automatic parking using self-location estimation in this embodiment. Learning-based automatic parking generates a map containing parking route data during learning driving, and during automatic parking, the vehicle drives according to the parking route registered in the map. The parking assistance device may register the parking route as a set of points, such as a start position 11, a parking position 14, and a start point 12 and end point 13 of a curved section, or may register the parking route as a combination of a straight section and a curved section. The curved section may also include information on the steering angle and turning radius. The positions of characteristic points captured by the camera on the parking route are also registered on the map.

[0020] In other words, feature points and parking routes are registered on the map. The feature points registered on the map are called markers to distinguish them from feature points detected from camera images. In other words, the map has markers and parking routes registered.

[0021] If there are many markers and there are markers in all directions of the vehicle, the vehicle's position can be accurately estimated while following a parking route and parking in an accurate position. However, if the number of markers is limited due to memory capacity, the error in the estimated vehicle position will increase, and the parking route or parking position 14 may be shifted or automatic parking may fail.

[0022] This embodiment focuses on the fact that the accuracy of the automatic parking route and parking position 14 changes depending on the selection of feature points to be registered on a map, and discloses a method for selecting feature points to be registered on a map so as to improve the accuracy of automatic parking. Note that, since selecting feature points presupposes that they will be registered on a map, selecting feature points to be registered on a map is sometimes simply referred to as "selecting feature points." For example, the method disclosed in this embodiment can be rephrased as a method for selecting feature points so as to improve the accuracy of position estimation.

[0023] The map generation method in this embodiment prioritizes registering feature points located within a predetermined range based on the camera equipped on vehicle 1 on the map, and the predetermined range is determined based on at least one of the optical axis direction of the camera or the position of the camera.

[0024] A range where the angle with respect to the optical axis of the camera is small is a direction in which the angle of the feature point can be accurately identified, so if the feature points in that direction are registered on a map, it is advantageous for controlling the steering angle. A feature point with a small difference in elevation from the camera is advantageous for controlling the steering angle because the angle of the detected feature point is less affected by changes in the posture of the vehicle body. A feature point with a small distance from the camera is advantageous for accurately estimating the position because the angle of the detected feature point changes greatly when the position of the camera changes.

[0025] For example, by using the camera position and camera optical axis when vehicle 1 is parked during learning driving as references, feature points that are close to the camera, close to the camera optical axis, and have a small difference in elevation from the camera can be selected and registered on the map, allowing the vehicle to accurately park in the same position during automatic parking. Alternatively, a position and direction requiring positional accuracy may be identified, and the positions of feature points advantageous for obtaining accuracy in that direction may be identified at that position, and feature points in more advantageous positions may be preferentially selected, thereby enabling the required accuracy to be obtained with fewer feature points. This method of identifying a position and direction requiring positional accuracy will be explained after the following configuration example.

[0026] 3 is a diagram showing a vehicle to which the parking assistance device of this embodiment can be applied. Cameras 2 are provided at four locations on the front, rear, left and right sides of the body of vehicle 1. Each camera 2 is equipped with a fisheye lens and has a horizontal field of view of 180 degrees or more (see dashed lines).

[0027] Each camera 2 is mounted at a depression angle to capture the road surface, so when the range of the road surface captured is converted into a horizontal field of view, a range of about 240 degrees of the road surface is captured by one camera 2. For example, the front and rear wheels and the sides of the vehicle are captured in the image captured by the side cameras 2a installed on the left and right sides of the vehicle.

[0028] FIG. 4 is a block diagram showing the parking assistance device 100 according to this embodiment. The camera 2 outputs a captured camera image, which is received by an image processing unit 130 of the parking assistance device 100 to generate a display image, etc. The image processing unit 130 may also be called an image acquisition unit. The display image is output from the notification unit 180 to the HMI device 20, and the notification unit 180 superimposes a message on the display image or outputs a voice message in response to an instruction from the state management unit 110. The state management unit 110 accepts a user operation via the operation device 10 and controls the functions of the parking assistance device 100 in response to the user operation. Here, the touch panel of the navigation device 40 is included in the operation device 10.

[0029] The state management unit 110 receives position information from the main body (not shown) of the navigation device 40. When performing a learning drive, the map generation unit 120 generates a map. The map generation unit 120 adds position information acquired by the navigation device 40 via GPS to the map and records the map in the storage unit 170. In this way, when performing automatic parking, the position information of the navigation device 40 can be compared with the position information added to the map, and the map to be used for automatic parking can be selected.

[0030] The image processing unit 130 generates a display image and a detection image. The detection image may be, for example, an image that emphasizes changes in brightness and color, or an image that has undergone processing to extract contours and edges. The feature point detection unit 140 extracts feature points from the detection image. Feature points are not points located inside the image surface or along its edges, but points at corners or edges whose positions can be identified. Therefore, it is sufficient to extract the image's contour and identify those corners or edges. The feature point information detected by the feature point detection unit 140 includes information on the color and shape of the image and information on the feature point's position on the camera image. The map generation unit 120 registers feature point information when generating a map, but after evaluating the feature points, registers only selected feature points on the map. The map generation unit 120 may also be called a feature point selection unit because it selects feature points to be registered on the map.

[0031] If the feature point is a corner, the color information of the feature point may include the color of the acute angle side and the color of the obtuse angle side. If the feature point is the end of a line, the color information of the feature point may include the color of the line and the color of the background. Since feature points are set by identifying the position of the image on the image, sharp images, i.e., images with high contrast, are preferable. Feature points detected from images with low contrast may change position depending on conditions such as the light source, or may even become undetectable, and are therefore undesirable as feature points to be registered on a map. Therefore, they may be excluded at the feature point detection stage.

[0032] For example, in the image of an object with a gently curved surface, the position of the boundary between the illuminated area and the shadow changes depending on the direction of the light ray, so registering feature points on the boundary line on the map may cause position estimation errors. Therefore, it is recommended that the map generation unit 120 recognizes that the boundary between light and shadow has a gradation-like appearance (i.e., low contrast) and does not register such feature points.

[0033] Furthermore, the map generating unit 120 may include the goodness of a feature point as a basic score of the feature point in the feature point information, and may add the goodness to an evaluation value based on the feature point's position when selecting a feature point to register on the map. Alternatively, when there are multiple feature points with the same evaluation based on their positions, the map generating unit 120 may select the feature point with the highest basic score.

[0034] If only feature points in the same direction as vehicle 1 are selected, there may be no difference in the orientation of the feature points as seen from vehicle 1. The principle of triangulation is to identify a position based on the difference in the orientation of feature points, so if the distribution of feature points is uneven, it may become impossible to identify the position of vehicle 1, or the accuracy of position estimation may decrease.

[0035] Therefore, the positions and directions of selected feature points are prevented from concentrating in a specific position or direction. For example, the map generating unit 120 sets an interval threshold that limits the interval between selected feature points, and once a certain feature point is selected, the map generating unit 120 prevents subsequent selection of feature points that are less than the interval threshold from the selected feature point.

[0036] Alternatively, when a certain feature point is selected, the map generating unit 120 performs lateral inhibition, which reduces the evaluation value as the distance from the feature point decreases, making feature points less likely to be selected near the selected feature point. By setting such distance thresholds and lateral inhibition characteristics for each position or section on the parking route, the number and density of selected feature points can be controlled.

[0037] Furthermore, the map generation unit 120 may assign evaluation values to feature points located at positions necessary for achieving positional accuracy, depending on the situation occurring on the parking path, so that feature points at the necessary positions are more likely to be selected. Alternatively, the map generation unit 120 may increase the number and density of feature points registered on the map at necessary positions by changing the interval threshold or lateral inhibition characteristics between positions where positional accuracy is required and positions where it is not. The settings for controlling the number and density of feature points to be selected may be referred to as "scoring." For example, the map generation unit 120 assigns scores according to the position on the parking path, assigning higher scores near parking positions and turning positions and decreasing scores elsewhere, thereby efficiently ensuring the necessary positional accuracy.

[0038] When the learning drive begins, the map generation unit 120 temporarily stores the feature points detected at the start position 11 of the learning drive in the memory unit 170. This storage of the feature points detected at the start position 11 is not a final registration on the map, and may be called temporary registration. The feature points to be registered on the map are determined after the learning drive is completed. Alternatively, unnecessary feature points from the temporarily registered feature points may be deleted after the learning drive is completed. Information on the feature points detected at the start position 11 is necessary to identify the position and posture of the vehicle 1 when automatic parking begins.

[0039] When the vehicle 1 starts its learning journey, the map generation unit 120 tracks the positions of the feature points on the camera images. For example, as shown in FIG. 1, the map generation unit 120 compares feature points captured in a camera image at time A with feature points captured in a camera image at a later time B to detect pairs of feature points whose color, shape, and movement amounts match. Matching of movement amounts means that, assuming that a feature point at time A moves to the position of a feature point at time B due to vehicle movement, the direction and distance of movement of the feature point roughly correspond to the direction and distance of movement of the vehicle. This process of matching feature points captured at different times is called tracking. The change in the positions of paired feature points on the camera images is due to motion parallax caused by the movement of the vehicle 1 between time A and time B. Therefore, the three-dimensional coordinates of the feature points can be identified by applying the principles of triangulation to motion parallax.

[0040] The map generating unit 120 further evaluates the feature points whose three-dimensional coordinates have been identified, selects those that meet predetermined conditions, and registers them on the map.

[0041] The storage unit 170 has a volatile area and a non-volatile area, and the map is stored in the non-volatile area. Storing in a non-volatile area is also called recording or registering.

[0042] When automatic parking begins, feature point detection unit 140 compares information about feature points detected from the camera image by feature point detection unit 140 (such as information about the color and shape of the image and information about the position of feature points on the camera image) with information about feature points read from the map to detect feature points that match the map. If automatic parking start position 11 is close to the position where learning driving began and there is not much difference in the orientation of the vehicle body, the positions of feature points on the camera image when automatic parking begins should not be much different from the positions of feature points on the camera image when learning driving began. This difference in the positions of feature points corresponds to the difference in the position and orientation of the vehicle, so it is possible to identify from the difference in the positions of the feature points how the position and orientation of the vehicle differs between when learning driving began and when automatic parking began.

[0043] Specifically, the position of a feature point on a camera image corresponds to the angle of the feature point (the angle of the line connecting the feature point and the camera relative to the optical axis direction), and indicates the angle at which the feature point is visible from the vehicle 1. When the position estimation unit 150 compares the angles of the feature points on all four sides with the three-dimensional coordinates of the feature points on the map, the position of the vehicle that appears at the angle that matches the feature points on all four sides is limited to a specific range. Therefore, the position estimation unit 150 identifies the position and orientation of the vehicle 1 by finding an optimal solution for the position and orientation of the vehicle 1 that matches the angles of multiple feature points. This process is called self-position estimation. The degree to which the angles of the feature points match the conditions may be referred to as likelihood, and the process of finding the optimal solution may be referred to as maximum likelihood estimation.

[0044] The self-location estimation can be performed using an existing method, and a detailed description thereof will be omitted. However, regardless of the method used, if the coordinates of a feature point registered on a map deviate from the coordinates of the actual feature point, or if the angle of a feature point estimated from the position at which the feature point appears on a camera image deviates from the actual angle, the estimated position will deviate from the actual position. In the self-location estimation by the position estimation unit 150, if a large number of feature points are used, even if there is a deviation in the information of some of the feature points, the deviation in the position estimation can be kept small. However, if there are few feature points, the deviation in the position estimation may become significant.

[0045] During learning driving, the driving control unit 160 communicates with the vehicle control device, acquires information on the number of rotations of the wheels and the steering angle, and calculates the amount of movement and direction of movement of the vehicle for each unit time. For example, a small section is defined as the amount of movement and direction of the vehicle for each unit time expressed as the length and direction of a line segment. In other words, a small section has data on length and direction. A small section may also be expressed as the number of rotations of the wheels or the steering angle. The driving control unit 160 calculates the position, attitude, and route of the vehicle moment by moment by accumulating the data for the small sections.

[0046] The map generation unit 120 may receive route information from the driving control unit 160 in units of data for small sections, or may receive the parking route (a collection of data for small sections) at the time of parking. This parking route will be a broken line made up of small sections, so both ends of each small section will be called break points. The parking route during learning driving may be a collection of data for break points.

[0047] When performing automatic parking, the driving control unit 160 estimates the position, attitude, and route of the vehicle based on information on the wheel rotation speed and steering angle, and outputs instruction values to the vehicle control device 30 every moment so that the vehicle route reproduces the route used during learning. In other words, the parking route used during automatic parking is a reproduction of the parking route used during learning.

[0048] If the vehicle's position or attitude deviates from the position or attitude of the vehicle during the learning drive, the driving control unit 160 first controls the steering angle to change the steering angle so that the path intersects with the path used during the learning drive, and when the vehicle overlaps with the path used during the learning drive, controls the steering angle so that the vehicle's attitude matches the attitude used during the learning drive. In other words, the driving control unit 160 estimates the vehicle's position, attitude, and path, and performs feedback control of the steering angle so that the vehicle's path follows the parking path used during the learning drive. Therefore, the driving control unit 160 may also be called a vehicle control unit. Alternatively, the position estimation unit 150 and the memory unit 170 may be included in the vehicle control unit, focusing on parking the vehicle based on a map.

[0049] When automatically parking, the driving control unit 160 may reproduce the vehicle speed used during learning driving, or may limit the vehicle speed. This is because if the vehicle speed is high, the wheels may slip and the vehicle may deviate from the route. The driving control unit 160 may, for example, maintain the vehicle speed at 5 km / h.

[0050] The position estimation unit 150 estimates the position and attitude of the vehicle separately from the driving control unit 160. For example, when it estimates that a slip has occurred based on the number of rotations of the wheels, the position and attitude data of the vehicle in the driving control unit 160 may be overwritten with the position and attitude data estimated by the position estimation unit 150, thereby correcting the steering angle control thereafter.

[0051] Fig. 5 is a diagram showing the hardware configuration of parking assistance device 100 in this embodiment. The functions of parking assistance device 100 may be implemented in the hardware shown in Fig. 5. Parking assistance device 100 may be a computer including a CPU 101, a ROM 102, a RAM 103, an I / O (input / output interface) 104, and an IMP (Image Processor) 105, with each element connected via a bus.

[0052] Parking assistance device 100 may accommodate multiple elements on a single chip, or one element may be configured with multiple chips. A single bus may be used, or multiple types of buses may be combined. For example, CPU 101, ROM 102, RAM 103, and IMP 105 may be accommodated on a single chip and connected via a parallel bus, while I / O 104 may be configured with multiple chips and connected to the chip containing CPU 101 via a serial bus.

[0053] The CPU 101 controls the entire parking assistance device 100. The functions of each unit of the parking assistance device 100 may be implemented in the form of a program executed by the CPU 101. The ROM 102 and RAM 103 correspond to storage units, with the ROM 102 corresponding to a non-volatile memory. The RAM 103 is used for temporary storage as a work area for the CPU 101. For example, the RAM 103 temporarily stores camera images such as display images and detection images, and information on detected feature points. The IMP 105 is a processor with enhanced processing performance specialized for image processing and parallel processing, and the processing of the image processing unit 130, feature point detection unit 140, position estimation unit 150, etc. may be executed by the IMP 105. In other words, the parking assistance device shown in FIG. 4 may be realized by a combination of hardware such as that shown in FIG. 5 and a program executed on the hardware.

[0054] Next, we will explain the relationship between the direction of feature points and accuracy. Figure 6 is a diagram showing the positional relationship between vehicle 1 and objects. The optical axis of side camera 2a housed in the left side mirror is oriented perpendicular to the longitudinal axis of the vehicle body, and object C close to the optical axis direction and object D located approximately 90 degrees from the optical axis direction are captured in a single side camera image. However, the accuracy of the position of the images captured in the camera image is not uniform. For example, the position of the image of object D captured in the periphery (near the left edge) of the side camera image is less accurate than the position of the image of object C captured in the center. The reason for this inaccuracy will be explained below.

[0055] Figure 7 shows the effect on accuracy of the angle of the camera relative to the optical axis direction. A lens that can capture images in directions perpendicular to the optical axis direction is called a fisheye lens, and is configured so that even light rays incident from directions perpendicular to the optical axis direction hit the image sensor by repeatedly bending light rays with multiple lenses. However, the way images are captured in a fisheye image 201 is very different between the center, which captures objects close to the optical axis direction on the image sensor, and the peripheral area, which captures objects outside the center and far from the optical axis.

[0056] At the center of the sensor, light coming from the optical axis direction is weakly converged and strikes the sensor surface at an angle close to a right angle, whereas at the periphery of the sensor, light rays from oblique to perpendicular directions are strongly converged and strike a narrow area on the sensor surface at an angle. For example, the distance that a light spot moves on the sensor surface as the angle of incidence of a thin light beam changes from 90 degrees to 45 degrees is shorter than the distance that the light spot moves as the angle of incidence changes from 45 degrees to 0 degrees. In other words, light outside the 45-degree angle is converged and strikes a narrow area on the sensor surface, and the convergence is stronger toward the outside, so that at the periphery of the fisheye image 201, the image of the subject appears squashed toward the center.

[0057] Furthermore, light contains wavelengths ranging from red to blue, and because the angle at which it is bent by the lens varies depending on the wavelength, red and blue light rays strike different positions on the image sensor. This is called chromatic aberration, and the effects of chromatic aberration are greatest in the peripheral areas where light rays are bent more sharply. In other words, color shift occurs in the peripheral areas, where the position of the image changes depending on the color. Furthermore, the focal plane where the image is formed is not flat but curved, and since the focal plane is adjusted to match the center of the image sensor, the peripheral areas become out of focus and out of focus. Furthermore, because the image sensor has a three-dimensional structure, light striking the image sensor at an angle in the peripheral areas also causes blurring. Images in the peripheral areas also become blurred due to being stretched during the subsequent distortion correction. In other words, due to various factors, images in the peripheral areas are blurred more than images in the center.

[0058] Because a lens bends light rays more strongly toward the outside, the relationship between the orientation of the subject and the position of the subject's image is a nonlinear function. Therefore, image processing unit 130 corrects the distortion of fisheye image 201 by stretching it outward according to the inverse function of this nonlinear function, generating distortion-corrected image 203. This is called lens distortion correction 202. Specifically, image processing unit 130 stretches fisheye image 201 in the direction away from the image center according to the inverse function of the nonlinear image height characteristic, which indicates the relationship between the distance from the center and the size of the image, to generate distortion-corrected image 203.

[0059] The image height characteristics are expressed by a graph with the image height on the vertical axis and θ degrees on the horizontal axis, where the image height is the distance between the image of light incident on the optical axis at an angle of θ degrees to the optical axis and the image center, when the position of the image of light incident on the optical axis is the image center. The detection image may be one that has undergone image processing on the distortion-corrected image 203, or the distortion-corrected image 203 may be used as is. Since the distance from the image center of the image on the detection image (image height) is directly proportional to the angle θ with respect to the optical axis, in self-localization, the position of the image on the detection image is converted into the orientation of the subject.

[0060] However, in reality, distortion remains in the distortion-corrected image 203 after distortion correction, with the image at the periphery being more distorted than the image at the center. One of the reasons for the remaining distortion is the problem of lens precision (variation). The lenses of vehicle-mounted cameras are constructed by combining lenses manufactured using a die-cutting method, and the precision of lenses manufactured using the die-cutting method is lower than the precision of lenses manufactured using a polishing method. Therefore, the image height characteristics of vehicle-mounted camera lenses vary within the range of shipping standards, and the range of image height variation tends to be wider in lower-cost cameras.

[0061] Furthermore, due to the extremely small size of the lenses used in vehicle-mounted cameras and cost constraints, when multiple lenses are stacked together, optical axis alignment, which involves adjusting the position and angle of the lenses so that their optical axes coincide, is not performed. If the optical axes of the lenses are not aligned, unequal image height occurs, where the image height changes depending on the direction relative to the center of the image. For example, when comparing the distance from the image center of an image of a subject located 90 degrees to the right of the optical axis with that of a subject located 90 degrees to the left, the distance from the image center will be different for the left and right images.

[0062] In principle, if the image height characteristics are measured in multiple directions for each camera and recorded for each camera, and the image height characteristics applied are changed for each camera and direction, distortion can be corrected fairly accurately even if unequal image heights or image height variations occur. However, in reality, due to cost constraints, individual image height characteristics are not measured, and distortion correction is performed using the same standard image height characteristics for all cameras and directions. In other words, distortion correction does not account for unequal image heights or image height variations, so the image position varies in the peripheral areas. Also, image height characteristics vary depending on the wavelength, but in in-vehicle cameras, due to cost constraints, distortion correction is performed using the same standard image height characteristics for all color components. In other words, distortion correction does not correct chromatic aberration, so the image position in the peripheral areas changes depending on the color of the subject.

[0063] In other words, the image position shifts by the amount that the lens's image height characteristics deviate from the standard value, and the image position also shifts depending on the color. Furthermore, blur also makes position identification unstable. For example, if the image is blurred and the contours are gradation-like, the quantified image position may change in relation to the detection threshold. Since image blur and position shifts are greater in the peripheral areas, it can be said that the position of images reflected in the peripheral areas is inaccurate. Since self-localization converts the position of an object in the image, specifically the lateral position of the object's image in the corrected image, into the object's orientation, it can be said that the orientation of objects reflected in the peripheral areas is less accurate than that of objects reflected in the center.

[0064] From the above, it can be said that it is advantageous in terms of accuracy to preferentially register feature points located in the direction of the camera's optical axis on a map. Therefore, when selecting feature points to be registered on a map, the feature point selection unit that selects feature points to be registered on a map evaluates the feature points based on the angle of the feature points with respect to the optical axis of a specified camera, and preferentially registers feature points that have a smaller angle with respect to the optical axis of the camera on the map over feature points that have a larger angle with respect to the optical axis of the camera.

[0065] Next, we will explain how the difference in height between the camera and feature points affects accuracy. Figure 8 is a diagram showing how the difference in height between the camera and feature points affects accuracy. The top and bottom figures are views of the vehicle body as seen from the front, with the top figure showing the vehicle during learning driving, with the vehicle standing upright. The bottom figure shows the vehicle during automatic parking, with one tire of the vehicle leaning in the roll direction after running over a bump 213 on the road surface. The bump 213 may be, for example, a mound of snow or soil, or it may be a situation where the other wheel has fallen into a rut.

[0066] Point E represents the position of the front camera that captures the image ahead of vehicle 1. Directly in front of the vehicle is a pole 211, with point F representing the position of a characteristic point at the base of pole 211 and point G representing the position of a characteristic point at the tip of pole 211. Also, there is a utility pole 212 on the right side of the vehicle body as seen from the driver.

[0067] The position estimation unit 150 converts the positions of the feature points on the camera image, specifically the lateral positions of the feature points on the image, into the orientation of the object. In the upper diagram, feature points F and G are both captured in the center of the image, so the identified orientations are the same. In the lower diagram, feature points F and G are also located directly in front of the camera, but point H on the road surface is captured in the center of the camera image, and feature points F and G are both captured to the right of the center of the image. The displacements of feature points F and G are both caused by the inclination of the vehicle body, but feature point F, which has a large difference in elevation from the camera, has a larger amount of displacement due to the inclination of the vehicle body than feature point G, which has a small difference in elevation from the camera.

[0068] If feature point F is registered on the map in the state shown in the figure above during learning driving, and self-location estimation is performed in the state shown in the figure below during automatic parking, feature point F will appear to the right of the center of the image, so even though pole 211 is directly in front of the camera, it is assumed that the vehicle body is shifted to the left (right in the figure) in the direction of travel, and the steering angle is corrected to the right so that feature point F is displayed in the center of the camera image. This causes vehicle 1 to change course, and vehicle 1 approaches utility pole 212 on the right side. In this way, the roll of the vehicle body can cause errors in self-location estimation, resulting in incorrect steering angle correction.

[0069] Now, consider a case where, instead of feature point F with a large difference in elevation from the camera, feature point G with a small difference in elevation is registered on the map. In this case, the amount of displacement of feature point G due to the tilt of the vehicle body is smaller than the amount of displacement of feature point F, so it can be expected that the amount of change in steering angle due to errors in self-position estimation will also be small. In other words, registering feature point G with a small difference in elevation from the camera is advantageous in terms of accuracy. Therefore, when selecting feature points, the feature point selection unit that selects feature points to be registered on the map evaluates the feature points based on the difference in elevation between a predetermined camera and the feature points, and gives priority to registering feature points with a small difference in elevation from the camera on the map over feature points with a large difference in elevation from the camera.

[0070] The conditions for feature points that are advantageous in terms of the accuracy of position estimation have been explained above based on the difference in elevation from the camera and the positional relationship with the optical axis direction, but the camera position and optical axis direction change as the vehicle 1 moves, so the positions of feature points that contribute to accuracy cannot be identified unless the positions on the route are identified. In other words, it is necessary to identify the positions on the route and select feature points that are advantageous in terms of accuracy at those positions.

[0071] It is also better to vary the registration priority and the number of feature points to be registered depending on the position on the route. Therefore, the feature point selection unit may identify a predetermined point or a predetermined section on the parking route, set the priority for registering feature points or the number to be registered according to the position on the specific parking route, evaluate the feature points at the specific position on the parking route based on the position of the camera or the relative position with respect to the optical axis direction of the camera, and preferentially register the feature points that are advantageous in terms of accuracy at that position. This step may be repeated for each position on the parking route.

[0072] Alternatively, the parking route may be evaluated from a bird's-eye view, and a process for setting the registration priority and the number of feature points to be registered for each position on the route may be performed first, and then a process for selecting feature points to be registered on the map may be performed based on the position of the camera or the relative position of the camera in the direction of the optical axis at the position on the parking route. Alternatively, the effect may be obtained by either a process for varying the registration priority or the number of feature points to be registered depending on the position on the parking route, or a process for selecting feature points to be registered on the map based on the position of the camera or the relative position of the feature points in the direction of the optical axis at the position on the parking route.

[0073] Furthermore, depending on the position on the route, there are directions that require precision and directions that do not, so the cameras that should capture feature points that contribute to precision differ. Therefore, when identifying a position on the route and evaluating feature points based on the camera position or the relative position with respect to the camera's optical axis direction, it is advisable to identify the camera that should be used as the evaluation standard at that position and select feature points that contribute to precision based on the position and optical axis direction of that specific camera. For example, at a position where precision in the left-right direction is required but precision in the front-to-back direction is not required, feature points captured in the optical axis directions of the front and rear cameras are registered first, and a larger number of feature points are registered near the front and rear cameras. In this case, the registration of feature points captured by the left and right cameras may be postponed, or the number of feature points to be registered may be reduced.

[0074] Alternatively, a higher score may be assigned to positions where precision is highly required, and many feature points may be registered, while a lower score may be assigned to positions where precision is less required. Alternatively, the stage for evaluating feature points and the stage for selecting feature points may be clearly separated, with selection being performed after all evaluations have been completed. For example, an evaluation step may be performed for all positions, in which feature points are evaluated at a certain position and high evaluation points are assigned to feature points that are advantageous in terms of achieving precision, and then feature points may be selected in descending order of evaluation points. In this case, similar effects can be expected by identifying cameras that contribute to precision at positions where precision is highly required, and assigning high evaluation points to feature points that contribute to precision.

[0075] Below, we will summarize the information that must be identified and the matters that must be considered when estimating the vehicle's position for automatic parking. The information that must be identified when estimating the vehicle's position for automatic parking is the vehicle's longitudinal position, lateral position, and orientation (posture). When applying this to an automatic parking path, the longitudinal direction of vehicle 1 corresponds to the tangent direction of the parking path, the lateral direction of vehicle 1 corresponds to the normal direction of the parking path, and the orientation (posture) of vehicle 1 corresponds to the inclination of the tangent or normal line of the parking path. In automatic parking, the vehicle is steered according to its lateral position, and its speed is controlled and gears are changed according to its longitudinal position.

[0076] Since feature points contribute to position estimation by being captured by the camera, the direction of the camera's optical axis is the criterion for evaluating the contribution of feature points to position estimation. The tangential direction of the parking path corresponds to the optical axis directions of the front and rear cameras 2 of the vehicle 1, and the normal direction of the parking path corresponds to the optical axis directions of the left and right cameras 2a of the vehicle 1.

[0077] A parking path can be divided into sections such as straight sections and curved sections, and the direction and degree of estimation accuracy required vary depending on the characteristics of the section. The estimated position is obtained by applying triangulation to the orientation of feature points, but depending on the positional relationship between the section and feature points, accuracy may or may not be improved. Here, feature points that contribute greatly to accuracy are called "sensitive" feature points, and feature points that contribute little to accuracy are called "low-sensitivity" feature points.

[0078] The relationship between the position of feature points and the accuracy obtained thereby will be summarized below. Figure 9 is a diagram showing the position of feature points and their sensitivity in the front-rear, left-right directions of vehicle 1. Consider a case where feature points I, J, K, and L are located on the front, rear, left, and right sides of vehicle 1, and when vehicle 1 moves forward, its movement is detected based on changes in the orientation of the feature points (motion parallax). The orientation of the feature points is a vehicle-based orientation, with 0 degrees in front of vehicle 1 in the forward direction. Cameras 2 on the front, rear, left, and right sides of vehicle 1 capture images of the feature points, and the orientation of the feature points is detected as their lateral positions on the camera images. For example, feature point I is at 0 degrees, and its orientation does not change even when vehicle 1 moves forward. In other words, the motion parallax of feature point I is zero. Since self-localization estimates the amount of movement based on motion parallax, the amount of movement cannot be estimated using feature point I. This can also be said to be insensitive to movement in the front-rear direction.

[0079] Feature point J at the rear of vehicle 1 is sensitive to longitudinal movement because its orientation changes with longitudinal movement of vehicle 1. Feature points KL on the left and right sides of vehicle 1 have greater changes in orientation with longitudinal movement of vehicle 1, so they can be said to have greater sensitivity than feature points I and J in the longitudinal direction of vehicle 1.

[0080] Comparing the sensitivity of feature point K and feature point L, the closer feature point K has a larger change in orientation and therefore a higher sensitivity than the farther feature point L. In general, closer feature points have a larger motion parallax than farther feature points, so they can be said to have a higher sensitivity.

[0081] Therefore, in situations where high accuracy in detecting the position in the forward and backward directions is required, the map generation unit 120 should select feature points located in the left and right directions, and it is even better to select feature points located in the left and right directions that are closer than those that are farther away.

[0082] FIG. 10 is a diagram showing the relationship between the distance from the path of vehicle 1 to a feature point and the sensitivity. FIG. 11 is a diagram showing changes in the sensitivity of a feature point as vehicle 1 moves. The sensitivity on the vertical axis may be interpreted as motion parallax. FIG. 11 corresponds to FIG. 10 and is a graph showing changes in the sensitivity of each feature point as vehicle 1 passes beside the feature points shown in FIG. 10. For example, when vehicle 1 passes directly beside feature points K and L, the sensitivity (motion parallax) between feature points K and L is maximized. At this time, the sensitivity of feature point K is greater than the sensitivity of feature point L, but as vehicle 1 moves away, the sensitivity of feature point K decreases and becomes less sensitive than feature point L. In other words, the closer a feature point is to the path of vehicle 1, the narrower the range of high sensitivity.

[0083] If the vehicle 1 requires positional accuracy in the longitudinal direction not only directly beside the feature point K but also before and after it, it is sufficient to register feature points K' and K'' before and after the feature point K. In other words, a narrow range of high sensitivity can be compensated for by increasing the number of feature points. For example, when setting an interval threshold that limits the interval between selected feature points in order to avoid bias in the feature points registered on a map, the interval threshold can be reduced for sections requiring accuracy in the longitudinal direction, thereby narrowing the interval between the selected feature points. In other words, by identifying sections requiring accuracy during learning driving and densely registering a large number of feature points close to the route of that section, it is possible to obtain the required accuracy in sections requiring accuracy during automatic parking.

[0084] Alternatively, the map generation unit 120 may register both feature points K that are close to the route and feature points L that are far from the route on the map, so that when the vehicle 1 moves away from feature point K, the required accuracy can be maintained using feature point L. Alternatively, in sections where low accuracy in the longitudinal direction is acceptable, the required accuracy can be maintained with fewer feature points by selecting only feature points L that are far from the route of the vehicle 1. Note that the sensitivity (motion parallax) of feature points decreases as the distance increases, so even when selecting feature points that are far away, it is advisable to exclude those that are, for example, 10 meters or more from selection.

[0085] A location where accuracy is maintained with a small number of feature points can be described as a location with a low priority and a small number of registrations. For a location with a high priority and a large number of registrations, it is better to register nearby feature points, so the distance of the prioritized feature points can be changed depending on the location on the parking path. In other words, the feature point selection unit evaluates feature points based on the distance between the feature point and the location on the parking path. If the location on the parking path has a low priority for registering feature points or a small number of registrations, feature points with a longer distance are registered on the map with higher priority than feature points with a high priority for registering feature points or a large number of registrations. For example, since the number of registrations is reduced at locations where accuracy in the forward and backward directions is not required, feature points captured by the left and right cameras that are approximately 5 meters away from the route can be registered with higher priority, and feature points within 2 meters of the route can be not registered.

[0086] 12 is a diagram showing the position of feature points and their sensitivity in the left-right direction. Considering a case where the position of vehicle 1 changes in the left-right direction, the orientation of left and right feature points KL does not change, whereas the orientation of front and rear feature points I and J changes. In other words, with regard to the left-right position, the front and rear feature points I and J have a larger motion parallax than the left and right feature points K and L, and therefore can be said to have a higher sensitivity. Therefore, in situations where high accuracy in detecting the left-right position is required, the map generation unit 120 should preferentially select feature points that have a small angle with respect to the front-to-rear direction of the vehicle.

[0087] FIG. 13 is a diagram showing a case where there is a feature point on the vehicle's path. FIG. 14 is a diagram showing changes in the sensitivity of the feature point when the vehicle moves in the X direction. FIG. 14 corresponds to FIG. 13 and shows the time changes in the sensitivity of feature points I and J in the left-right direction when vehicle 1 passes from left to right as shown in FIG. 13. Here, feature points I and J are assumed to be on the road surface, and vehicle 1 passes over the feature points. The fact that closer feature points have higher sensitivity than more distant feature points is the same in the case of left-right sensitivity as in the case of forward-backward position changes. However, as shown in FIG. 13, when a feature point is below vehicle 1, the feature point is in the blind spot of camera 2 and cannot be detected, so the sensitivity becomes zero.

[0088] In other words, nearby feature points have good sensitivity, but if the feature point is in a blind spot, the sensitivity will be lost. Therefore, it is possible to select the feature point closest to the front camera from among the feature points that can be detected at a position requiring precision in the left-right direction (for example, parking position P).

[0089] FIG. 15 is a diagram showing the positions of feature points and the sensitivity of the vehicle body posture. When the orientation of the vehicle 1 changes, the orientations of the front, rear, left, and right feature points I, J, K, and L all change, so there is no difference in sensitivity due to the orientation. The fact that nearby feature points have higher sensitivity than distant feature points is the same as the sensitivity in the front-rear and left-right directions. Therefore, in situations where high accuracy in detecting the orientation of the vehicle is required, the map generation unit 120 may select nearby feature points with priority over distant feature points.

[0090] The difference in sensitivity of feature points depending on whether they are near or far when the direction of the vehicle 1 changes is a secondary effect caused by motion parallax, which occurs when the position of the camera 2 changes when the direction of the vehicle 1 changes. Figure 16 is a diagram showing a state in which the direction or position of the camera 2 changes.

[0091] For example, as shown in the figure above, when the position of camera 2 remains the same and only the orientation of camera 2 changes, the image of a nearby object and the image of a distant object move the same amount on the screen. In other words, when only the orientation of camera 2 changes, there is no motion parallax, so there is no difference in the sensitivity of feature points due to distance. As shown in the figure below, when the orientation of vehicle 1 changes, and the orientation of camera 2 changes at the same time, the image of a nearby object moves more than the image of a distant object due to motion parallax. In other words, Figure 16 shows that the image of a distant object moves the same amount as or less than the image of the nearby object. Patent Document 2 states that "the farther an object is from the vehicle, the more it will move in response to a change in attitude," which is the exact opposite of the phenomenon shown in the figure, but this is thought to be a misunderstanding of the secondary effects of motion parallax.

[0092] Next, the positions of vehicle 1 that require high accuracy in self-location estimation will be described. Fig. 17 is a diagram showing the positions of vehicle 1 that require high accuracy in self-location estimation. M is the parking start position, N is the turning start position, Q is the turning position, R is the turning end position, and S is the parking position. If the parking path is divided into straight sections and curved sections, M to S are all the end points (start or end points) of the sections. Of the end points, points N and R do not change the vehicle's traveling direction (gear position), so they are called passing points, and points M, Q, and S are called stopping points, because the vehicle stops at these points.

[0093] At the stopping point, the steering angle is changed, the direction of travel is changed, and the vehicle is stopped, so if the longitudinal positional accuracy at the end point is poor, the vehicle may deviate from the parking path or the parking position may be shifted. Therefore, longitudinal positional accuracy is required at the end point. Also, if the lateral position is shifted at the parking position S or the turning position Q, the vehicle may approach an obstacle, so lateral positional accuracy is required at the stopping point. In other words, positional accuracy in both the longitudinal and lateral directions is required at the stopping point.

[0094] Of the passing points N and R, N does not require high horizontal positional accuracy because there are no obstacles on either side, but R is the point where the vehicle starts moving straight toward the parking position, and since left and right positioning is performed here, left and right positional accuracy is also required. At points where positional accuracy is required, a feature point close to the camera or optical axis direction can be selected using the position and optical axis direction of camera 2 at that point as the basis. Furthermore, left and right positioning begins just before R, so left and right positional accuracy is also required at the position just before R.

[0095] 18 and 19 are diagrams showing the arrangement of feature points around the end points. If the left and right positions need to be accurate at the position R in FIG. 17, for example, alignment can be started from point R' just before R, as shown in FIG. 18. In that case, it is sufficient to arrange the feature points so that positional accuracy can be obtained between R' and R.

[0096] 18, when the vehicle 1 is at position R, the map generating unit 120 may place feature points in ranges 221 located in the optical axis directions of the front, rear, left, and right cameras 2 of the vehicle 1. In this way, even at the position of point R' just before R, it can be expected that feature points in the same range 221 will be captured in directions close to the optical axes of the front, rear, left, and right cameras 2. In other words, by placing feature points around the end points, sufficient positional accuracy can be obtained even just before the end points.

[0097] Furthermore, the range of feature points to be registered may be adjusted so that they can be easily captured even in front of the endpoints, as shown in Fig. 19. Range 222 in Fig. 19 is adjusted by shifting the position of range 221 in response to the fact that the head of vehicle 1 is positioned to the left at the position of point R' in front of R, and is adjusted so that feature points are positioned approximately directly in front of camera 2 of vehicle 1 at both R and R'. Feature points located around these endpoints, including those in front of them, are called endpoint feature points.

[0098] In an automatic parking route, the section separated by a pair of endpoints is called a section. Based on their shape, sections can be divided into straight sections where the vehicle travels straight and curved sections where the vehicle turns at a fixed steering angle. For example, in Figure 17, MN and RS are straight sections, and NQ and QR are curved sections.

[0099] Geometrically, a section includes endpoints, but in processing feature points, feature points at endpoints (feature points placed around endpoints) and feature points at sections are treated separately, and feature points at endpoints are not included in feature points at sections. This is because feature points at endpoints must be treated preferentially. Specifically, when focusing on one section, feature points are first placed around the endpoints, and then, if necessary, feature points are placed around the section. When a vehicle is near an endpoint, position accuracy can be obtained using feature points placed around the endpoints, so if position accuracy needs to be obtained in the intermediate portion between endpoints, they are used as feature points for the section. In other words, feature points for sections can be treated as auxiliary points, and do not need to be registered if not necessary.

[0100] The position in the forward / backward direction within a section (the direction along the section if it is a straight section, or the tangent direction if it is a curved section) only needs to be determined when approaching the endpoints, so positional accuracy in the forward / backward direction is not necessary in the intermediate sections between the endpoints. In other words, it is sufficient to obtain positional accuracy in the left / right direction within a section (the direction perpendicular to the section if it is a straight section, or the normal direction if it is a curved section). Within a section, positional accuracy is not obtained solely from feature points placed as feature points of the section, but is obtained using all detected feature points. In other words, to obtain positional accuracy in the left / right direction within a section, feature points at the endpoints can be used, so feature points can be used to supplement feature points at positions in the middle of the section where accuracy is insufficient.

[0101] FIG. 20 is a diagram showing the ranges of feature points at the endpoints R and S of a straight section RS along which a vehicle backs up from R to S, using ellipses. For example, in the straight section from R to S, there is a range 231 of feature points located on the left and right sides of the vehicle at position R, a range 233 of feature points located in front and behind the vehicle, a range 232 of feature points located on the left and right sides of the vehicle at position S, and a range 234 of feature points located in front and behind the vehicle. When the vehicle is positioned near the midpoint between endpoints R and S, feature points in the blind spot below the vehicle cannot be detected, but all other detection points can be detected. Furthermore, the feature points in the ranges 233 and 234 located in front and behind the vehicle 1 are in the direction of the optical axis of the camera in the front-rear direction, so sufficient accuracy can be achieved in estimating the left-right position. Therefore, for example, when the section is a short straight section as shown in FIG. 20, there may be no feature points to register as feature points for the section.

[0102] In this way, if feature points at endpoints are registered with priority, fewer feature points will be registered as feature points for sections, and the majority of the registered feature points will be feature points at endpoints. Alternatively, feature points at endpoints may be registered with priority by allocating (assigning) more feature points to endpoints than to section feature points. This can also be expressed as saying that the feature point selection unit registers feature points according to their positions on the parking route, and that feature points to be registered at endpoints are registered with priority over feature points to be registered in sections, or that more feature points are registered at endpoints than in sections.

[0103] In addition, each section may be divided into a section just before the end and the rest, and the section just before the end may be called the end section, and the rest may be called the intermediate section, and the end section may be given the same priority as the end section when registering feature points. The end point is the end point of the section that has the shorter route length to the parking position, and the route length to the parking position is the distance from that point along the parking route to the parking position.

[0104] This is to address the fact that even if it is determined that the vehicle has deviated from the route near the end point, it is difficult to correct it back onto the route. For example, if it is determined that the vehicle is 20 cm off to the side 1 meter before the parking position, it is difficult to correct, but if it is determined that the vehicle is 2 meters before, it is possible to correct it. In other words, even if it is determined that the vehicle is off at the end point, it is too late, so the system makes it possible to accurately estimate the position from a position before the end point, where it is possible to correct the deviation.

[0105] Therefore, the feature point selection unit evaluates the path length between a position on the parking path and the parking position, and either prioritizes feature points to be registered at positions with shorter path lengths over feature points to be registered at positions with longer path lengths, or registers more feature points at positions with shorter path lengths than at positions with longer path lengths. For example, even within one section, feature points to be registered at the end portion may be prioritized over feature points to be registered at the middle portion, and more feature points to be registered at the end portion may be registered than feature points to be registered at the middle portion.

[0106] The difference from the former, which prioritizes only endpoints, is that the range in which feature points are preferentially allocated to obtain accuracy is expanded to the portion just before the end (end portion), but as mentioned above, sufficient accuracy can often be obtained even at the end portion if many feature points are allocated around the endpoints, so there need not be any substantial difference. In this embodiment, a distinction is made between endpoints and end portions in order to explain the background to the accuracy required at endpoints and end portions. However, if feature points are allocated with priority given to endpoints, and then, when arranging feature points for a section, feature points are allocated with priority given to portions close to the end points, it can be expected that the same result will be obtained as when end portions are prioritized, so either method is acceptable in practice.

[0107] 21 and 22 are diagrams showing the posture convergence section in which the occupant aligns the vehicle in the left-right direction by steering the vehicle 1 before the parking position. Generally, the front wheels of a vehicle 1 are steerable, and the rear wheel axles are fixed and do not rotate left or right. Therefore, while the vehicle can move freely in the front-rear direction, it is difficult to move left or right, and the rear of the vehicle is particularly difficult to move left or right. Because of these characteristics of the vehicle 1, when parking, the driver often first steers the vehicle in the left-right direction before the parking position, and then aligns the vehicle in the front-rear direction to the parking position before stopping.

[0108] In other words, the last section of the parking path is a straight section that ends at the parking position, and before that there is a section where the vehicle is aligned left and right by steering. The vehicle's attitude (direction) changes while the vehicle is aligned left and right, but once the alignment is complete, the steering angle is returned to neutral and the change in attitude converges; this section is called the attitude convergence section. The position of this attitude convergence section depends on the state of the parking space 241. For example, if the parking space 241 is a parking frame painted on the road surface and there are no three-dimensional objects around it, the vehicle can be aligned even within the parking space 241.

[0109] Therefore, as shown in Figure 21, the posture convergence section continues from the outside of the parking space to the inside of the parking space, and the length of the final straight section may be one vehicle length or less. However, as shown in Figure 22, when parking on pallet 242 in a mechanical parking lot, the vehicle can only go straight on pallet 242, so left and right alignment must be completed just before pallet 242. Therefore, the posture convergence section ends just before pallet 242, and the length of the final straight section is one vehicle length or more.

[0110] Since parking spaces 241 are often set at approximately right angles to the road or passageway facing the parking space 241, there is often a curved section where the vehicle body must turn before the straight section. Therefore, when returning the steering angle to neutral at the end of the curved section, the left and right positions are often adjusted by adjusting the speed at which the steering angle is returned. In such cases, as shown in Figure 22, the end of the last curved section, in other words, the terminal end of the last curved section, becomes the attitude convergence section.

[0111] Therefore, a section where the attitude angle of the vehicle 1 has changed and the difference between the attitude angle and the attitude angle of the vehicle at the parking position is equal to or less than a predetermined angle threshold is defined as an attitude convergence section, and the positional accuracy in the left-right direction is improved in the attitude convergence section. For example, if the parking path ends with a straight section and there is a curved section before that, the attitude convergence section where left-right positioning is performed is the end of the curved section. By definition, the attitude convergence section does not include the last straight section. This is because the straight section is a section where the attitude angle of the vehicle is maintained and left-right positioning is completed in the straight section. In other words, positional accuracy in the attitude convergence section may be given more importance than positional accuracy at the parking position, and feature point registration in the attitude convergence section may be given priority over feature point registration at the parking position.

[0112] As described above, the section where the driver has performed left-right positioning may be set as the posture convergence section, or the posture convergence section may be determined geometrically. In the posture convergence section, it is preferable to arrange feature points so that left-right positioning accuracy can be obtained. For example, as shown in FIG. 22 , when parking on a pallet 242, if feature point 244 at the entrance of pallet 242 can be registered on the map, it is preferable for left-right positioning in the posture convergence section. When vehicle 1 enters onto pallet 242, feature point 244 at the entrance of pallet 242 is hidden under the vehicle body and loses its sensitivity. However, because left-right positioning is completed just before pallet 242, there is no problem even if feature point 244 at the entrance loses sensitivity while driving straight on pallet 242.

[0113] Feature points that are highly sensitive to positional changes in the left-right direction are feature points located in the optical axis direction of the front and rear cameras 2 of the vehicle 1, and the optical axis direction of the front and rear cameras 2 is a tangential direction of the parking path from a bird's-eye view. Furthermore, feature points closer to the front and rear cameras 2 of the vehicle 1 have higher sensitivity. Therefore, in an attitude convergence section where the attitude angle of the vehicle 1 is changing and the difference between the attitude angle and the attitude angle of the vehicle 1 at the parking position is equal to or less than a predetermined angle threshold, the map generation unit 120 may preferentially register on the map feature points that are located in the tangential direction of the attitude convergence section and that are close to the attitude convergence section.

[0114] Figure 23 is a diagram showing a turning section that requires precision in the left-right direction. Vehicle 1 has the tendency to move straight when the driver releases the steering wheel, which is called straight-line stability. The vehicle turns when the driver turns the steering wheel to change the direction of the steered wheels. However, if the road surface is wet and the front wheels slip or their grip on the road surface weakens, the straight-line stability may cause the vehicle to deviate to the outside of the path. Furthermore, as shown in Figure 23, if the vehicle is out of position left-right when turning at the entrance to parking space 251, parking may not be possible.

[0115] To prevent such a situation, it is necessary to detect lateral positional deviations in curved sections and control the steering angle so as not to deviate from the route. Since lateral positional deviations are likely to occur when the steering angle is large and the turning radius is small, curved sections where the absolute value of the steering angle is equal to or greater than a predetermined steering angle threshold can be particularly classified as turning sections, and the lateral positional accuracy in turning sections can be improved.

[0116] As described above, since the attitude convergence section may be the end of a turning section, the section corresponding to either the attitude convergence section or the turning section may be set as the section for improving the positional accuracy in the left-right direction. Also, the entire curved section with a large steering angle may be set as the section for improving the positional accuracy in the left-right direction.

[0117] FIG. 24 is a diagram showing sensitivity in the left-right direction when the vehicle 1 turns. Feature points captured in the optical axis direction of the cameras 2 in front and behind the vehicle 1 are sensitive to positional deviations in the left-right direction, but because the orientation (direction of travel) of the vehicle 1 changes in curved sections, the direction of the optical axis of the cameras 2 is not constant. Therefore, the definition of direction in curved sections and the conditions for feature points to be prioritized are summarized below. In curved sections, the vehicle turns with a turning radius that corresponds to the steering angle, so we will summarize the case where the turning radius is constant and the curved section forms an arc that is part of the circumference as a model. In this case, the left-right direction can be referred to as the radial direction, and the front-rear direction as the circumferential direction.

[0118] Figure 24 shows a case where the curved section is an arc acb that runs from point a to point b via point c. If vehicle 1 is at point c and camera 2 observes the directions of points a and b relative to vehicle 1, the difference in direction between points a and b, represented by ∠acb, is constant according to the inscribed angle theorem, no matter where point c is located on arc acb.

[0119] However, if vehicle 1 deviates from the curved section (arc acb) to the outside and is at point d, the difference in direction between points a and b, represented by ∠adb, will be smaller than when vehicle 1 is on the parking path, such that ∠adb<∠acb. This shows that points a and b, or characteristic points on arc acb, have a certain degree of sensitivity in detecting the left-right position.

[0120] FIG. 25 is a diagram showing sensitivity in the left-right direction when the vehicle 1 is turning. In FIG. 25, points on the arc ef are sampled at equal intervals, and an arrow 261 representing the fore-and-aft direction of the vehicle 1 is plotted assuming that the vehicle 1 is located at each of these positions. The fore-and-aft direction of the vehicle 1 is also the tangent direction to the arc ef. As shown in FIG. 25, in the section where the vehicle 1 is turning, the fore-and-aft direction of the vehicle 1 faces outside the arc, so it can be said that feature points with high sensitivity in the left-and-right direction are located outside the arc. Therefore, the map generation unit 120 may preferentially select feature points that are outside the arc and close to the arc as feature points with high sensitivity in the left-and-right direction.

[0121] In contrast, the area near the center of the arc is always located directly to the side regardless of the left-right or front-rear position of the vehicle 1, and the angle seen from the vehicle 1 does not change much, so it can be said that the motion parallax is small and the sensitivity is low (the contribution to position accuracy is small). Therefore, when the route forms an arc, the map generation unit 120 evaluates the feature points based on the positional relationship between the feature points and the route, and it is preferable that feature points in the front-rear direction of the vehicle or on the outside of the arc are more easily registered on the map than feature points on the inside of the arc.

[0122] Since the longitudinal direction of the vehicle 1 is also a tangential direction, the map generating unit 120 may preferentially select feature points that are in the tangential direction and close to the curved section, as in the attitude convergence section. As a result, feature points that are outside the arc and close to the arc are preferentially selected.

[0123] 26 is a diagram showing a situation in which vehicle 1 requires accuracy in the left-right direction. When passing by an obstacle (for example, a utility pole 271) or when turning around in front of a fence, positional accuracy is required to avoid contact. In manual parking, the driver often slows down vehicle 1 in such cases, so the position where vehicle 1 has slowed down or is traveling at a low speed may be estimated to be close to the obstacle.

[0124] The map generator 120 may also determine that an obstacle is approaching based on distance information from the obstacle obtained from the obstacle detection device. That is, the map generator 120 may determine that position accuracy is necessary when the vehicle speed is low or deceleration is occurring, or when the distance to the obstacle is close. The map generator 120 may also determine that position accuracy is particularly necessary when the distance to the obstacle is close and the vehicle speed is low or deceleration is occurring.

[0125] Specifically, the feature point selection unit (map generation unit 120) registers feature points based on vehicle speed information indicating the vehicle speed or distance information indicating the distance to an obstacle, and registers feature points on the map in preference to locations where the vehicle speed is low or where there is deceleration, rather than locations where the vehicle speed is high and there is no deceleration, or in preference to locations where the distance between the vehicle 1 and an obstacle is short, rather than locations where the distance between the vehicle 1 and an obstacle is long.

[0126] The direction requiring precision near an obstacle is not limited to the left-right direction. For example, when turning around in front of an obstacle, positional precision in the front-rear direction is also required to prevent a collision due to overtraveling. Therefore, the range in which feature points are preferentially selected may be different for a passing point where the vehicle passes without stopping and a stopping point where the vehicle stops. At a passing point, feature points close to the positions of the front and rear cameras or the direction of the optical axes of the front and rear cameras may be preferentially selected, and at a stopping point, feature points close to the positions of the front, rear, left, and right cameras or the direction of the optical axes of the front, rear, left, and right cameras may be preferentially selected.

[0127] Alternatively, the priority of registering feature points may be changed depending on the obstacle detection information. For example, if no obstacle is detected ahead at the turning point, there is no risk of collision even if the positional accuracy in the forward and backward directions is low, so the priority of feature points captured by the left and right cameras may be lowered.

[0128] Furthermore, position accuracy should not be evaluated in two stages, "necessary" or "not necessary," but should be evaluated in a continuous manner according to distance, vehicle speed, etc. It is advisable to evaluate the accuracy by combining information on the distance to an obstacle and vehicle speed when the driver passes by or approaches an obstacle during manual parking. For example, the map generation unit 120 may evaluate feature points based on vehicle speed and changes in vehicle speed, and may preferentially register feature points near the camera just before the garage, based on the fact that the vehicle speed is low when entering the garage and that the vehicle speed decreases just before the garage.

[0129] Alternatively, the map generation unit 120 may use vehicle speed as the only condition and make it easier to register feature points located in the front-to-rear direction of the vehicle 1 on the map at locations where the vehicle speed is low after deceleration than at locations where the vehicle speed is high and no deceleration occurs, or may evaluate whether the driver is reducing the vehicle speed as an indicator of the level of positional accuracy required, and register more feature points closer to the vehicle 1 as the vehicle speed is lower. In other words, evaluation may be performed using a combination of multiple conditions, but a combination is not essential.

[0130] 27 is a flowchart of map generation in this embodiment. The process of generating a map through learning driving may be executed by the map generation unit 120 of the parking assistance device 100 in the following steps.

[0131] In the start point process, the map generation unit 120 registers the GPS coordinates of the start position and collects information on detected feature points (step S1). In the tracking process, the map generation unit 120 tracks the feature points, identifies their coordinates, and collects route information such as steering angle and movement amount (step S2). In the end point process, the map generation unit 120 determines that the learning drive is complete (step S3).

[0132] In the analysis process, the map generation unit 120 identifies the endpoints and sections of the parking route and organizes the feature points (step S4). In the point allocation process, the map generation unit 120 determines the number of feature points to be registered for each endpoint and each section (step S5). In the registration process, the map generation unit 120 determines the feature points to be registered for each endpoint and each section (step S6).

[0133] 27 corresponds to a case where the parking route is evaluated from a bird's-eye view, and the process of setting the registration priority and the number of feature points to be registered for each position on the route is first performed, and then the process of selecting the feature points to be registered on the map is performed based on the position of the camera at the position on the parking route or the relative position of the camera with respect to the optical axis direction. In other words, the process of setting the registration priority and the number of feature points to be registered corresponds to step S5, and the process of selecting the feature points to be registered on the map based on the position of the camera at the position on the parking route or the relative position of the camera with respect to the optical axis direction corresponds to step S6. Each step will be described in detail below.

[0134] For the explanation of the parking route, refer to FIG. 2. The starting point of the starting point process in S1 is the starting position 11 of the learning drive. At the starting point, the map to be used during automatic parking is identified from the GPS coordinates, and information is acquired to enable initial self-location estimation. First, the map generation unit 120 acquires the GPS coordinates of the starting position 11 from the navigation device 40 and stores them in a volatile area of the storage unit 170. Also, in the starting point process, the map generation unit 120 collects information on characteristic points detected at the starting position to use for initial self-location estimation for automatic parking. Since the collected information is temporarily stored in the volatile area of the storage unit 170, hereinafter, the map generation unit 120 collecting information may be rephrased as the map generation unit 120 writing information to the volatile area of the storage unit 170.

[0135] The tracking process of S2 is performed while the vehicle 1 travels from the start position 11 to the parking position 14. During this time, the map generation unit 120 tracks feature points, calculates the coordinates of the feature points that have been successfully tracked, and collects feature point information including the coordinates. In addition, during the tracking process, the map generation unit 120 collects route information such as the steering angle, vehicle speed, gear position, moving direction, and moving distance, as well as obstacle detection information.

[0136] The end point of the end point process in S3 is the parking position 14 where the vehicle was parked during the learning drive, and when the gear position becomes parking (P), that point is determined to be the end point. If the map generation unit 120 is performing tracking processing at the time the end point is determined, the tracking processing ends and, on the condition that the gear position is P, the process moves to analysis processing. In other words, the end point process is a step that determines the start of analysis processing.

[0137] In the analysis process of S4, the map generation unit 120 analyzes the parking route and identifies the endpoints and sections. In the next point allocation process, the map generation unit 120 allocates points to each endpoint and section, so the analysis process can be said to be pre-processing for the point allocation process.

[0138] S1 to S3 are processes during learning driving, and S4 and onwards are processes after learning driving. During learning driving, map generation unit 120 prioritizes data collection, and performs other processes after learning driving. For example, during learning driving, map generation unit 120 records the data of small sections or the time series of data of break points received from driving control unit 160 directly in storage unit 170, and after learning driving, analyzes the parking route to identify sections and endpoints.

[0139] For example, the map generation unit 120 receives from the driving control unit 160 a parking route in the form of a broken line made up of many small sections. Therefore, in the analysis process of S4, the map generation unit 120 merges the multiple small sections to reconstruct the parking route into a small number of sections divided by a small number of endpoints. For example, the map generation unit 120 may analyze the time series of data for the small sections, merge a series of small sections where the steering angle does not change before and after to form a section, and use the break points (points of change in steering angle) where the steering angle changes before and after as the endpoints. Alternatively, the parking route may be analyzed from a bird's-eye view and approximated with a small number of straight sections and a small number of curved sections divided by fewer endpoints than the break points.

[0140] The map generation unit 120 registers the parking route simplified by this reconstruction and approximation as the route for automatic parking. In other words, the map generation unit 120 reduces the number of endpoints of the automatic parking route compared to the turning points of the parking route during learning. In automatic parking, the occupants feel unsettled every time the vehicle passes an endpoint (a point where the steering angle changes). Therefore, simplifying the parking route during learning from a bird's eye view and reducing the number of endpoints of the automatic parking route reduces the number of times the occupants feel unsettled, resulting in a better usability.

[0141] It is also advantageous in terms of accuracy for the map generation unit 120 to have fewer end points than break points. Because steering angle change points are points where control is performed to change the steering angle during automatic parking, feature points should be preferentially placed around them to ensure positional accuracy. However, if all break points are made end points, the number of feature points per end point will be reduced. Therefore, it is advisable for the map generation unit 120 to simplify the route to reduce the number of sections and reduce the number of end points to which feature points are assigned. In this way, positions that do not require positional accuracy are excluded from the end points, so more feature points can be placed at positions (end points) that require positional accuracy.

[0142] Furthermore, the map generation unit 120 may treat the break points (change points of steering angle) of the automatic parking route and the endpoints to which feature points are assigned as separate entities, reducing the number of endpoints to which feature points are assigned, but leaving the break points (change points of steering angle) of the parking route unchanged. In other words, the map generation unit 120 may reduce the number of endpoints to which feature points are assigned without changing the registered parking route by making the majority of change points of steering angle break points to which feature points are not assigned. Break points (change points of steering angle) to which feature points are not assigned may be called recessive endpoints, and making break points to which feature points are not assigned may be rephrased as recessive endpoints.

[0143] 28 is a diagram showing the adjustment of endpoints. For example, when sections gh and ij of a parking route passing through points g, h, i, and j are straight sections and section hi is a short curved section, the map generating unit 120 evaluates the relationship between the parking route and the parking position or obstacles. Then, depending on the evaluation result, the map generating unit 120 may determine one or both of points h and i as inferior endpoints and exclude them from the endpoints to which feature points are assigned.

[0144] For example, if point j is the parking position 14 and the section between points h and i, i.e., section hi, is evaluated as an attitude convergence section for adjusting the attitude and lateral position of the vehicle body, the map generation unit 120 may leave both points h and i as endpoints to which feature points are assigned. For example, if an obstacle is detected near point h, the map generation unit 120 may evaluate that the lateral position accuracy of the vehicle body is required near point h, and leave point h as an endpoint to which feature points are assigned, but may make point i a recessive endpoint (break point) and exclude it from the endpoints to which feature points are assigned.

[0145] For example, if point j is not near parking position 14 and no obstacles are detected near points h and i, the map generation unit 120 may evaluate that positional accuracy is not required at points h and i, and may treat points h and i as inferior end points (break points), excluding both from the end points to which feature points are assigned.

[0146] In this case, the map generation unit 120 may treat points h and i as if they did not exist, approximate section gj with a large arc, and register it on the map as a single curved section gj with no intermediate bends, or may leave points h and i as recessive endpoints (bends) and register it on the map as three sections gh, hi, ij where the steering angle changes at intermediate bends. In either case, the number of feature points decreases near points h and i, and the number of feature points that can be assigned to one endpoint can be increased. In summary, the analysis process performed by the map generation unit 120 can be said to be a process that analyzes the parking route during learning driving and reduces the number of endpoints to which feature points are assigned to compared to the bends that are the points where the steering angle changes during learning driving.

[0147] 29 is a diagram showing an example of point allocation to endpoints and sections. In the point allocation process of S5, the map generation unit 120 determines the upper limit (point allocation) of the number of feature points to be registered at endpoints and sections. At that time, the map generation unit 120 evaluates the parking positions of the endpoints and sections, their relationships with obstacles, and the amount of change in steering angle and attitude, and changes the point allocation accordingly.

[0148] For example, the map generation unit 120 assigns a higher score to an end point or section that is closer to the parking position S. Since the user evaluates the quality of parking based on the parking position S, the shorter the route length to the parking position S, the higher the accuracy. The score may be adjusted according to the risk. For example, since the risk is high near an obstacle, it is advisable to assign a higher score to an end point or section that is close to the obstacle. Furthermore, the map generation unit 120 may estimate that a section where the driver traveled at a low speed is riskier than a section where the driver traveled at a faster speed, and assign a higher score to the section.

[0149] The map generation unit 120 evaluates the amount of change in the vehicle's steering angle and attitude angle and reflects it in the allocation of points. This is because the greater the amount of change in the steering angle or attitude angle, the higher the risk of lateral positional deviation. The amount of change in attitude angle is the amount of change in the vehicle's attitude angle (the direction the front of the vehicle is pointing) between the start and end points of the section. Lateral position accuracy is also required in the attitude convergence section, which is a section in which the difference between the vehicle's attitude angle and the vehicle's attitude angle at the parking position is equal to or less than a predetermined angle threshold. Therefore, the feature point selection unit evaluates the amount of change in the vehicle's steering angle and attitude angle, and in either a turning section in which the steering angle is equal to or greater than a predetermined threshold, or a turning section in which the amount of change in attitude angle within the section is equal to or greater than a predetermined threshold, or an attitude convergence section in which the difference between the vehicle's attitude angle and the vehicle's attitude angle at the parking position is equal to or less than a predetermined angle threshold, the feature point selection unit may preferentially register feature points located tangential to the section on the map, or may allocate more points to register more feature points.

[0150] Between endpoints and sections, endpoints are assigned points with priority over sections, and are assigned more points than sections. In the example of Figure 29, endpoints M, N, Q, R, and S are assigned 75% of the points, but since feature points around endpoints can often be detected in the sections before and after them, it is not a problem if the sections are assigned fewer points. Since the points assigned to sections are to enable position estimation even between endpoints, the map generation unit 120 may assign more points to particularly long sections if there are any.

[0151] End point M is the starting point, and since automatic parking cannot begin unless the vehicle's self-position can be estimated at the starting point, a high score is assigned to it. The end point with the next highest score is R. Vehicle 1 has finished adjusting its lateral position and attitude at R, and the end of section QR is also an attitude convergence section, so it is given priority in score allocation over parking position S. In other words, priority is not determined solely by the length of the route to parking position S, but rather the priority and score allocation are determined according to the required positional accuracy. For example, in section RS, the vehicle is close to obstacles on the left and right, and it is difficult to correct its attitude after end point R, so end point R is assigned a high score.

[0152] Q is the turning position, and S is the parking position. Both are stopping points and close to obstacles, so a high score is assigned to them to ensure accurate longitudinal positioning. N is the starting point of a turn, but it is a passing point, and the route to parking position S is long and not close to any obstacles, so a lower score is assigned to it than the other end points.

[0153] Among the sections, the QR section, which includes the posture convergence section, is assigned the highest score. The NQ section, which has the same turning radius as the QR section, is assigned a lower score because it is farther from the parking position, but it is assigned a higher score than the straight section MN and straight section RS. The straight section MN is assigned a lower score because the route length to the parking position S is longer than the straight section RS.

[0154] In the registration process of S6, the map generation unit 120 first determines the order in which feature points are to be determined, and then determines the feature points to be registered for each endpoint and each section. The order in which feature points are determined is such that determination of endpoint feature points takes priority over determination of section feature points. However, priority may also be given to feature points with a shorter route length to the parking position S or feature points with a higher score. For example, the order may be point S, section RS, point R, section QR, point Q, section NQ, point N, section MN, and point M, or the order may be point R, point S, point Q, point M, section QR, section NQ, section RS, point N, and section MN. Furthermore, the map generation unit 120 may process the sections after processing all of the endpoints, such as in the order of point R, point S, point Q, point M, point N, section QR, section NQ, section RS, and section MN.

[0155] In the registration process, the map generation unit 120 imposes a density restriction to prevent feature points from concentrating in a small area. For example, the registration of feature points around a previously registered feature point (marker) may be prohibited, or the evaluation value of the feature point may be deducted (lateral inhibition). In other words, as a side effect of the density restriction, a previously registered feature point prevents the subsequent registration of surrounding feature points. When determining the order in which feature points are to be determined, the reason for registering feature points of high importance at endpoints or intervals first is to prevent feature points with high sensitivity from being unable to be registered due to the density restriction.

[0156] In addition, the map generation unit 120 adjusts the density restriction parameters according to the points allocated so that a sufficient number of feature points can be registered at locations requiring high positional accuracy, adjusting the density restriction threshold and the strength of lateral inhibition so that the density is higher where the points are high. This can also be expressed as follows: the feature point selection unit includes a density restriction unit that restricts the density of feature points to be registered, and the density restriction unit varies the density depending on the location on the parking path, and the density is higher at locations with a high priority for registering feature points or locations where a large number of feature points are registered than at locations with a low priority and where a small number of feature points are registered. In other words, at locations requiring high positional accuracy, the restriction imposed by the density restriction unit is relaxed, and feature points are concentrated in a narrow range.

[0157] 30 is a diagram showing the registration of a feature point at endpoint R, which is the end point of the curved section QR that turns toward the parking position. When registering the feature point at endpoint R, the map generation unit 120 selects a camera to be used as the evaluation standard depending on the direction in which accuracy is required, and evaluates the angle of the feature point with respect to the optical axis of the camera, the difference in elevation between the feature point and the camera, and the distance, using the selected camera as the standard, and preferentially registers feature points that are in advantageous positions for obtaining accuracy.

[0158] For example, since end point R requires positional accuracy in the left-right direction, the map generation unit 120 selects feature points captured in the center of the front and rear cameras. When feature points k, l, and m are captured by the rear camera, k and l are in the direction of the camera's optical axis, but k is closer to the camera, so the map generation unit 120 assigns a higher priority to k. Although l and m are the same distance from the camera, m is at a larger angle from the optical axis and is captured in the peripheral area, so the map generation unit 120 assigns a lower priority to m. If there is a feature point l' that is located at the same position on the plane as feature point l and has a smaller elevation difference from the camera than l, the map generation unit 120 assigns priority to l' over l. If only one of l or l' can be registered due to density restrictions, the map generation unit 120 may register feature point l' with a smaller elevation difference from the camera and exclude feature point l with a larger elevation difference.

[0159] In this way, the map generation unit 120 determines the priority, for example, k > l' > m, and determines the range of feature points to be registered according to the allocation of points. For example, when the number of feature points (allocation of points) to be registered in the field of view of the rear camera is 3, the map generation unit 120 registers all of k, l', and m, but when the allocation of points is low, it registers only k and l', or only k. In practice, the number of feature points to be registered in the field of view of the rear camera at the end point of the posture convergence interval will be 4 or more, so it is also possible to register feature points l, m, and m', which are in positions that are disadvantageous in terms of achieving accuracy. Furthermore, feature points within the field of view (not shown) may be registered up to the number determined by the allocation of points, and in this case, it is only necessary to preferentially register feature points that are advantageous in terms of achieving accuracy.

[0160] For example, the evaluation value of each feature point is calculated based on the angle relative to the optical axis direction of the rear camera and the difference in elevation from the camera, and the feature point with the highest evaluation value is registered first. After one feature point is registered, the evaluation values of feature points near it are deducted due to density restrictions. Next, the feature points are re-sorted using the updated evaluation values, and the feature point with the next highest evaluation value is registered. This process of density restrictions, sorting, and registration, which forms one cycle, can be repeated the number of times determined by the score allocation.

[0161] When selecting a feature point for an end point, the map generation unit 120 may take into consideration the sections before and after it. For example, if feature point n and feature point p are within the field of view of the forward camera and only one can be registered due to density restrictions, the map generation unit 120 may evaluate that feature point n is closer to the optical axis direction of the forward camera while traveling through section QR and preferentially register feature point n. Alternatively, the map generation unit 120 may evaluate the positions of feature points relative to the sections before and after the end point, add points to the feature points, and register feature points with higher evaluation values first. Both feature point n and feature point p are eligible for point addition because they are located outside the arc of curved section QR, but because feature point n is closer to the arc, it is preferable to add more points. Furthermore, because the portion before R corresponds to the posture convergence section, the map generation unit 120 may adjust the density restriction threshold and register both feature point n and feature point p.

[0162] When registering feature points of the endpoints, the map generation unit 120 allocates points to each camera depending on the direction and traveling direction requiring precision, based on the number of feature points to be registered. For example, the map generation unit 120 allocates points to all cameras in a skewed manner depending on the direction and traveling direction requiring precision, such as 40% to the rear camera facing the traveling direction, 30% to the front camera facing the opposite direction, 20% to the right camera facing the outside of the route QR, and 10% to the left camera facing the inside.

[0163] In this way, registering feature points in other directions, in addition to those requiring precision, makes it easier to maintain functionality even when the environment changes. In other words, robustness is improved. For example, if only rear feature points k and l' and front feature point n are registered because positional precision in the left-right direction is required, and left-right feature points m and m' are not registered, automatic parking may not be possible due to changes in the environment. For example, if rear feature points k and l' are located in a garage and the rear camera image is completely black due to the direction of sunlight, and the feature points cannot be detected, the vehicle's position cannot be determined using only the front feature point n, and self-localization cannot be performed, so automatic parking cannot continue. However, if feature points captured by the left and right cameras, such as m and m', are also registered, self-localization can be performed based on the detection points in the three directions (forward, left, and right) even when the rear camera image is completely black.

[0164] Density restrictions, which prevent feature points from concentrating in a small area, also have the same purpose. For example, if only feature points located in a small nearby area are registered and those feature points belong to a vehicle parked nearby, if that vehicle disappears or moves, automatic parking may not be possible or the parking position may be shifted.

[0165] For example, if you select a feature point that is higher than the camera, the probability that it is another vehicle decreases, and if you select a feature point that is far from the camera, the probability that it is not on the road increases. In other words, it is better to distribute the placement of feature points in terms of the height difference and distance from the camera.

[0166] 31 is a diagram showing the range of feature points to be registered in a curved section. At a feature point, the camera position and optical axis direction are fixed, whereas the camera position and optical axis direction change in a curved section. Therefore, the map generation unit 120 may evaluate feature points based on the trajectory of the camera position movement within the section or the range in which the optical axis direction changes.

[0167] For example, if arc qr is the trajectory of the camera in the traveling direction and the optical axis direction changes from the direction qs to the direction rt, then range 281 enclosed by arc qr, arc st, and line segments qs and rt corresponds to the optical axis direction and can be said to be a range close to the camera. Also, range 282 enclosed by arcs st, uv, and line segments su and tv in Figure 31 has the same conditions with respect to the optical axis direction and can be said to be a range farther from the camera. Therefore, feature points in range 281 may be assigned a higher evaluation value than feature points in range 282, and feature points with higher evaluation values may be registered preferentially.

[0168] In this way, the map generating unit 120 may identify a range in a curved section where the conditions in the optical axis direction are the same, and within that range, evaluate feature points based on their distance from the camera position. Alternatively, as a simpler method, the map generating unit 120 may evaluate feature points based on their distance from the camera trajectory.

[0169] FIG. 32 is a diagram showing the range of feature points to be registered in a straight section. In a straight section, the optical axis directions of the front and rear cameras do not change, while the optical axis directions of the left and right cameras move in parallel. Therefore, the relationship between the camera and the feature point is determined by the distance between the camera's trajectory and the feature point. Therefore, the map generation unit 120 may, for example, use the trajectories of the left and right cameras as a reference and divide the space into tree-ring-shaped (or Baumkuchen-shaped) regions 291 at the same distance from the camera's trajectory, and evaluate the feature points. For example, within a range on one tree ring at the same distance from the camera's trajectory, priority may be given to feature points with a small angular difference from the camera's optical axis, and among feature points with the same angular difference from the camera's optical axis, priority may be given to feature points with a short distance from the camera's trajectory. Furthermore, the map generation unit 120 may change the evaluation criteria and, within a range on one tree ring at the same distance from the camera, priority may be given to feature points with a small difference in elevation from the camera. This evaluation method can also be implemented in curved sections.

[0170] Furthermore, when registering feature points, the distance of feature points to be registered with priority can be adjusted according to the score allocation. This is because nearby feature points have a high maximum sensitivity, but the range of high sensitivity is narrow, so the sensitivity drops rapidly as the distance from the feature point increases. Also, nearby feature points may be in the vehicle's blind spot, and when a feature point is in the vehicle's blind spot, its sensitivity becomes zero. Therefore, the feature point selection unit evaluates feature points based on the distance between the feature point and its position on the parking path. If the position on the parking path has a low priority for registering feature points or a small number of feature points, feature points with a longer distance are registered on the map with priority over positions with a high priority for registering feature points or a large number of feature points. A position with a high priority and many feature points to be registered has a high score allocation. In other words, if the score allocation is high, nearby feature points are prioritized, and if the score allocation is low, distant feature points are prioritized. In this way, by prioritizing the registration of distant feature points in positions with a small number of points, position accuracy within the section can be ensured with a small number of feature points.

[0171] Furthermore, the map generation unit 120 may prioritize feature points with shorter route lengths to the parking position, and may register feature points within a single section starting with those closest to the end point. This is to prevent feature points that are located near an end point, which requires greater accuracy, from not being registered due to density limitations. For this reason, for example, a gradient scoring system may be used to add more points to the evaluation values of feature points that are candidates for registration within a single section, the closer they are to the end point, so that feature points closest to the end point are registered first.

[0172] When feature point registration at the endpoints is performed first, feature points are already registered near the start and end points of the section. In particular, in straight sections, positional accuracy in the left-right direction can often be ensured by the feature points registered at the front and rear endpoints. Therefore, when registering feature points at the endpoints first, the map generation unit 120 should register left and right feature points using the left and right cameras as references to register feature points for the section, thereby ensuring positional accuracy in the front-to-back direction. Furthermore, when registering feature points in curved sections, the map generation unit 120 should preferentially register feature points on the outside of the arc over feature points toward the center of the arc.

[0173] In addition, this disclosure also includes forms obtained by applying various modifications to each embodiment that a person skilled in the art would think of, or forms realized by arbitrarily combining the components and functions of each embodiment within the scope of this disclosure. [Industrial Applicability]

[0174] The present disclosure can be used in a parking assistance device and a parking assistance method. [Explanation of symbols]

[0175] 1 vehicle 2 Cameras 10 Operating device 11 Starting position 12 end points 14 Parking Location 20 HMI device 30 Vehicle control device 40 Navigation Devices 100 Parking assistance device 101 CPU 102 ROM 103 RAM 104 I / O 105 IMP 110 Status Management Unit 120 Map Generation Unit 130 Image processing section 140 Feature point detection unit 150 Position estimation part 160 Travel control unit 170 Storage section 180 Information Department 201 fisheye images 202 Distortion correction 203 Image after distortion correction 211 Paul 213 Convex 241 parking spaces 242 palettes 261 Arrow 271 Electric pole

Claims

1. an image acquisition unit that acquires camera images from cameras that respectively capture different directions around the vehicle; a feature point detection unit that extracts feature points from the camera image; a feature point selection unit that evaluates the feature points and selects the feature points to be registered on the map during a learning drive in which the vehicle is manually parked and the parking route and parking position are registered on the map; a vehicle control unit that parks the vehicle based on the map during automatic parking, the feature point selection unit varies the priority of registering feature points or the number of feature points to be registered depending on the position on the parking path, or selects feature points to be registered on the map based on the position of the camera or the relative position of the feature points with respect to the optical axis direction of the camera at the position on the parking path, the feature point selection unit evaluates the feature points based on the distance between the position on the parking path and the feature point, and when the position on the parking path is a position with a low priority for registering the feature point or a position with a small number of registrations, the feature point selection unit preferentially registers the feature point with a longer distance on the map compared to when the position on the parking path is a position with a high priority for registering the feature point or a position with a large number of registrations. Parking assistance device.

2. the feature point selection unit evaluates the feature points based on angles of the feature points with respect to an optical axis of the camera, and registers, on the map, feature points with smaller angles with priority over feature points with larger angles. The parking assistance device according to claim 1.

3. the feature point selection unit evaluates the steering angle and attitude angle of the vehicle, and in a turning section where the steering angle is equal to or greater than a predetermined threshold, or where the amount of change in attitude angle within the section is equal to or greater than a predetermined threshold, or in an attitude convergence section where the difference between the attitude angle of the vehicle and the attitude angle of the vehicle at a parking position is equal to or less than a predetermined angle threshold, preferentially registers on the map feature points in a tangential direction of the section, or increases the number of feature points to be registered; The parking assistance device according to claim 1.

4. the feature point selection unit registers feature points based on vehicle speed information indicating the vehicle speed of the vehicle or distance information indicating the distance between the vehicle and an obstacle, and registers feature points at locations where the vehicle speed is low or deceleration is occurring preferentially over locations where the vehicle speed is high and no deceleration is occurring, or registers feature points at locations where the distance is short preferentially over locations where the distance is long. The parking assistance device according to claim 1.

5. the feature point selection unit evaluates a path length between a position on the parking path and the parking position, and either registers feature points at positions with a shorter path length in preference to feature points at positions with a longer path length, or registers more feature points at positions with a shorter path length than at positions with a longer path length. The parking assistance device according to claim 1.

6. acquiring camera images from cameras each of which views a different direction around the vehicle; extracting feature points from the camera image; a step of evaluating the feature points and selecting feature points to be registered on the map during a learning run in which the vehicle is manually parked and the parking route and parking position are registered on a map; and during automated parking, parking the vehicle based on the map; the step of selecting the feature points varies a priority for registering the feature points or the number of feature points to be registered depending on the position on the parking path, or selects the feature points to be registered on the map based on the position of the camera or the relative position of the feature points with respect to the optical axis direction of the camera at the position on the parking path; The step of selecting the feature points includes evaluating the feature points based on the distance between the position on the parking route and the feature points, and when the position on the parking route is a position with a low priority for registering the feature points or a position with a small number of registrations, registering the feature points with a longer distance on the map with higher priority than when the position is a position with a high priority for registering the feature points or a position with a large number of registrations. Parking assistance methods.

7. the step of selecting the feature points evaluates the feature points based on angles of the feature points with respect to an optical axis of the camera, and registers the feature points with smaller angles on the map with higher priority than the feature points with larger angles. The parking assistance method according to claim 6.

8. The step of selecting the feature points includes evaluating the steering angle and attitude angle of the vehicle, and preferentially registering, or increasing the number of registered feature points, feature points tangential to a turning section where the steering angle is equal to or greater than a predetermined threshold value, or an attitude convergence section where the difference between the attitude angle of the vehicle and the attitude angle of the vehicle at a parking position is equal to or less than a predetermined angle threshold value, on the map. The parking assistance method according to claim 6.

9. The step of selecting the feature points includes registering feature points based on vehicle speed information indicating a vehicle speed of the vehicle or distance information indicating a distance between the vehicle and an obstacle, and registering feature points at locations where the vehicle speed is low or deceleration is occurring preferentially over locations where the vehicle speed is high and deceleration is not occurring, or registering feature points at locations where the distance is short preferentially over locations where the distance is long. The parking assistance method according to claim 6.

10. The step of selecting the feature points includes evaluating a path length between a position on the parking path and the parking position, and registering feature points at positions with a shorter path length in preference to feature points at positions with a longer path length, or registering more feature points at positions with a shorter path length than at positions with a longer path length. The parking assistance method according to claim 6.

Images